How to speed up your Development Test suite

Image by Juan-Calderon.

When you practice Test-Driven Development, most of the time you need to run only a small number of tests to validate your recent code changes. Unfortunately things change once you start refactoring. Refactoring models implicates your entire application. So in order to keep things from going down the drain, you’ll need to run all (or most of) your tests constantly. And that’s where things get tedious.

At Codeship we develop our Ruby on Rails web app test-driven with Rspec. Most of our specs are high level request specs (in a way integration tests) which are slow compared to controller or model specs. One of the reasons for this slowness is that request specs test the whole application stack. Another reason is that we changed the default capybara driver from Rack::Test to Poltergeist.

“Poltergeist? But that’s so much slower!”

Right! But we chose Poltergeist because it tests a web application more like a user experiences it. While Rack::Test only considers the HTML (or whatsoever) response from your application, Poltergeist launches a headlessWebKit browser, provided by PhantomJS.

This way we could not only test our backend application by navigating through the HTML code, but also test our awesome Javascript sugar.

“Ok, so use Poltergeist only when your specs require Javascript!”

That would be enough to test all of the Javascript functionality. Unfortunately Javascript can be a villain, too. If there’s a Javascript error, chances are that even things stop working that normally wouldn’t rely on Javascript. If Poltergeist comes across a Javascript error it will throw an exception immediately and thus fail the spec.

Because of this additional safety we decided on making Poltergeist the default capybara driver.

Development vs. Codeship CI

Using test-driven development will eventually take you to the point of misery, when running all your specs slows down your productivity unbearably. Poltergeist lets us reach this point at record speed. So what to do now? On the one hand we wanted to retain the quality of our web app, on the other hand we wanted to continue developing without feeling the need of poking our eyes out while waiting for the specs to succeed.

Running our spec suite on my computer with this setup took 8:45 minutes.

The solution was Codeship itself: We would speed up the specs in development as much as possible and let the Continous Integration server do all the cumbersome work.

Step one: Use Poltergeist only for Javascript

I know, that’s what you always said. We decided on using Poltergeist for all request specs that required Javascript, and Rack::Test for the rest in development. On the continous integration server we’d still run all specs with Poltergeist.

This little change reduced the execution time to 5:00 minutes.

Step two: Skip slowest specs

There were some specs that took up to 40 seconds to run. These were used to check some quite complicated procedures that rarely changed. We decided to skip all specs for development that took longer than 10 seconds.

RSpec provides tagging to achieve this: We tagged our slow tags with

speed: "slow"

Running our specs with

rspec --tag ~speed:slow

reduced the execution time by half: 2:29 minutes. Yei!

Step three: Skip remote specs

In Codeship we integrate a couple of external services like Github. Of course we also needed to verify that the communication with these services worked. But a developer’s life is hard: Sometimes you are on a train, on a plane or there’s simply no network reception. And all of a sudden many of your specs fail.

Therefore we tried to remove the dependencies on external services for as many specs as possible. We tagged all specs that still required internet access with

remote: true

and skipped them in development. Executing

rspec --tag ~@remote --tag ~speed:slow

now finished in: 1:28 minutes.

By changing our test setup and skipping time-consuming tests in development we were able to cut down test execution time by more than 83%. This way we could stay productive during development and still perform extensive checks on our web application using the Codeship continous integration server.

Codeship – A hosted Continuous Deployment platform for web applications

Here’s a final overview of the results after each optimization step:

number of specs execution time (minutes)
All tests with Poltergeist: 128 8:45
All tests with Rack::Test/Poltergeist: 128 5:00
Without slow specs (> 10s): 107 2:29
Without slow and remote specs: 99 1:27

Further Information

Here are some more blog posts that explain how to speed up your test suite:

Subscribe via Email

Join 16,045 Newsletter Subscribers.

Join the Discussion

Leave us some comments on what you think about this topic or if you like to add something.