Faster test suite boot times with Ruby on Rails

This is a republished guest blog post. The original article is written by Manfred Stienstra. You can find it on the fingertips blog.


Why you need a fast test suite

Developers need to be able to run tests quickly or they will stop running them.

The biggest bane of test driven development, or whatever variant you practice, is long boot times. Even when you just run one test a slow boot will make it a tedious job. There are a number of ways to reduce startup times in a Ruby on Rails project.

Load less dependencies to get a faster test suite boot time

Project dependencies need to be loaded every time you start your test suite, less dependencies means faster startup. Keeping project dependencies to a minimum is always a good idea, not just because of boot time.

Let’s jump into an example. We’re assuming you’re using the following, simplified, Gemfile for your dependencies:

gem "rails"
gem "bcrypt"
gem "foreman"
gem "thin"
gem "newrelic_rpm"
gem "airbrake"

First we stop loading gems which aren’t used in the code.

gem "rails"
gem "bcrypt"
gem "foreman", require: false
gem "thin", require: false
gem "newrelic_rpm"
gem "airbrake"

Then we make sure dependencies are only specified for the environments where they’re used in.

gem "rails"
gem "bcrypt"

group :development do
  gem "foreman", require: false
end

group :production do
  gem "thin", require: false
  gem "newrelic_rpm"
  gem "airbrake"
end

At this point you might start getting undefined constant errors about Airbrake. You can decide to either move Airbrake back into all environments or stub its implementation.

We generally do this by defining the module or class ourselves from a support file in the test directory, like so:

module Airbrake
    extend self
    def notify(exception, opts = {})
    end
  end

Partial example of a stub implementation for Airbrake.

On the one hand you probably never want to send data from your test suite to Airbrake. On the other hand some people are more comfortable running their entire production stack in at least one or two tests.

Reduce time spent setting up tests

Basically you need to do less. There are a few strategies to accomplish this.

One strategy is moving setup code as local to your test as possible so it only runs for the tests where it’s needed. For example, don’t run setup methods specific for functional tests in unit test.

Keep in mind that your test_helper.rb or spec_helper.rb is loaded during every boot. Try to limit slow operations in this file.

Tests that require long setup tasks sometimes need to be run in a variety of contexts: library tests, model tests, and functional tests. Sometimes you can cheat a little bit and test through the entire stack in one massive integration test.

Continuous Deployment for Rails projects to Heroku

Another strategy is to perform long setup tasks only once and find a way to cache the result to speed up next runs.

In one of our applications we process YAML files with questionnaire definitions and create records for them in the database. In hindsight we should have never done this, but that’s a discussion for another day.

In unit and functional tests we use fixtures to test specific situations. For regression tests we need the production definitions. We speed up loading these by dumping the database tables with INTO OUTFILE and load them with LOAD DATA INFILE. We also hook into Rails fixture loading code so it only happens once for the entire suite.

Finally, never, ever, ever, ever, call external services from your tests. Not only is this slow, but it might accidentally erase production data. In most tests we override Net::HTTP#start to make sure it’s never called by accident.

Factories or fixtures

There is a lot to say about database setup, so it deserves its own little chapter.

Let’s get the elephant out of the room first. Factories are slow! They’re slow by design.

Factories can be used to create database records anywhere in your test suite. This makes them pretty flexible and allows you to keep your test data local to your tests. The drawback is that it makes them almost impossible to speed up in any significant way.

We can understand why factories are slow by looking at why ActiveRecord fixtures are fast.

Fixtures are bulk loaded once. A transaction is created in the database when a test starts. After the test the database is rolled back to its initial state. This means the fixtures don’t have to be reloaded for each test.

Most factory implementations start a test with an empty database and run through all the factories needed for a test each time it’s run. After the test the database is reset to its blank state by removing all records. On top of that records are created through ActiveRecord model instances, which creates a lot of objects in the Ruby VM, which need to be garbage collected.

We try to use fixtures as much as possible. We write little scenarios with them to make tests easier to understand. Paul might be a pro account holder who has paid his bills the last two years, but unfortunately missed a payment this month because his credit card expired.

Using small scenarios like this reduce the chance of someone messing up tests by using the wrong fixture or changing fixtures to not fit their intended scenario. You can even document these scenarios shortly in the fixture files or test helper.

Use a fast test framework

Your test framework needs to be loaded and initialized during every boot.

RSpec is notoriously slow. If you’re willing to part with it, you might want to consider minitest/spec. You can use minitest/spec with Rails out of the box since version 4.0. Another alternative is Peck on Rails.

In doubt, measure!

Optimizations don’t really make sense when you can’t measure the speedups. Most test frameworks print their runtimes, but this doesn’t include boot time. The time tool is your friend.

$ time ruby test/unit/post_test.rb
Loaded suite test/unit/post_test
Started
..
Finished in 0.001313 seconds.

2 tests, 6 assertions, 0 failures, 0 errors
ruby test/unit/post_test.rb  0.15s user 0.08s system 96% cpu 0.238 total

In this example test/unit reports 0.001313 seconds, but the real runtime is 0.238.

To measure particular parts of your code you can either use a profiler or the Benchmark class.

Isolate the slow code and keep the benchmark in place to measure if your fix actually helps.


We think Manfred makes some great points here. As mentioned above you should definitely check out the fingertips blog – they have some great stuff on there!

How do you keep your test suite fast? Are there any other things you found work really well for you? Let us know in the comments. We would love to get your opinion!

Subscribe via Email

Be sure to join 13,643 subscribers of our newsletter to receive updates on software development best practices, Continuous Delivery and tips and tricks to start shipping your product faster.

Join the Discussion

Leave us some comments on what you think about this topic or if you like to add something.

  • soulcutter

    RSpec is notoriously slow? In what way specifically?

  • adone

    spork and parallel_tests in conjunction with the guard solve all problems that you described

  • Manfreds

    @soulcutter:disqus RSpec is slow al starting up, probably because it loads a ton of source files.

    Eloy Duran (@alloy) did a quick benchmark the other day: https://gist.github.com/alloy/6249335. It takes RSpec almost 400 milliseconds to run one spec on a fast Mac Pro with SSD.

    • http://rosenfeld.heroku.com/ Rodrigo Rosenfeld Rosas

      wow, it must really slow down your TDD iteration if you have to wait 1s before your test code start running :) A really good reason not to use RSpec (obvious sarcasm here) :)

      • Manfreds

        It all adds up. It could be one of the factors why you’re waiting 5 or 10 seconds for your suite to start. With 400 milliseconds, it’s probably not the biggest time sink, but that depends on the suite.

  • Manfreds

    @adone Well, it doesn’t solve the boot times, you still have to wait for Spork to start at least once. Sometimes Spork will need to the entire app because of dependency problems. That said, Spork is a nice solution and I’ve used it in the past.

    parallel_tests doesn’t really help with the boot times, but it does speed up the total runtime for your suite. I will probably discussing solutions like that in the near future.

    Peck can also run suites concurrently out of the box.

  • soulcutter

    I see the benchmark is against 1.9.3 – ruby 2.0 has better performance around requires, I suspect it levels the playing field somewhat as far as startup time. Still, startup time tends to be a drop in the bucket when you’re faced with long-running tests, but thanks for elaborating. You should put that in the body of your post! I don’t necessarily agree about the notoriously slow comment, but I see more where you’re coming from now.

    I highly recommend Spring or Zeus (but mostly Spring) over Spork FWIW.

  • Manfreds

    Long startup times are mostly annoying when you’re trying to make a specific test or test file green. I find it harder to keep concentrated on the programming task when I need to wait 5 seconds for my test to start.

    On my Macbook Air 2.0 is actually a little bit slower than 1.9: https://gist.github.com/Manfred/6300112

  • joel

    With regards to factories I think some of your assumptions might be a little off. It’s not the factories that are slow but the accessing of the database that’s slow. For unit tests you can, more often than not, use the build or build_stubbed strategies instead of create. Either of those two will not touch the database, saving a lot of time.

    Granted for integration tests you’ll need to use the create strategy for good end-to-end testing, but for unit, controller or view tests you can lean on objects that aren’t touching your database.

    Josh Clayton wrote about this on the Giant Robots blog: http://robots.thoughtbot.com/post/22670085288/use-factory-girls-build-stubbed-for-a-faster-test

    Hope this helps.

    • Manfreds

      To be even more specific, inserts through ActiveRecord used to be very slow. This appears to be less of an issue on Rails 4.0 and Ruby 2.0.

      You can create a lot of strategies to speed up factories, but I believe it will always be slower than batch loading records once. Especially if you use a vendor specific way of doing it. The fact that factories can create records all over tests makes it that you can never batch load them all before running the entire suite.

      I believe setting up test data as close to the test as possible makes them much more readable and maintainable. I am personally not comfortable with adding both a factory library and another gem for optimising it. I think that’s a personal choice the developers on a project need to make.

      • joel

        I’ve never added another gem for factory optimization. I didn’t think there was a need to do so. Good isolated unit tests don’t necessarily need that – see Gary Bernhard’s DAS screencasts for good examples. If I recall correctly he doesn’t use either – factories or fixtures. (I could be wrong).

        To each their own but I prefer factories to fixtures solely for proximity and ease of understanding the unit or system under test. I haven’t used fixtures enough as a result of it’s introducing too much of a “mystery guest” code smell.

        Like I said to each their own but I maintain that factories, when used properly, are more than fine for the job.

  • josh_cheek

    “RSpec is not the reason your Rails test suite is slow” — https://twitter.com/dchelimsky/status/180676997305991168

    • Manfreds

      Well, it depends on how you test. Here is an example where RSpec is slower: https://gist.github.com/Manfred/6300112.

      Note that I’m talking about test suite boot times and not the entire runtime of the suite. On the entire suite, and especially for slow suites a few hundred milliseconds don’t matter that much.

  • http://rosenfeld.heroku.com/ Rodrigo Rosenfeld Rosas

    Factories aren’t slower by design. But they’re indeed way better for expressing your expectations. I usually create common records in a before(:all) of some context using factories. In my set-up (using PostgreSQL, which supports nested transactions – or savepoints) the before-all blocks are also rolled back in the end of the context. So they are expressive (more then fixtures) and not slower. I should notice I’m using Sequel and this gem in my specs:

    https://github.com/rosenfeld/rspec_around_all/blob/config_around/spec/rspec_around_all_spec.rb

    • http://rosenfeld.heroku.com/ Rodrigo Rosenfeld Rosas
    • Manfreds

      Sure, you can introduce strategies to run your factories as little as possible. But even then they will still be slower because of at least two reason.

      Factories generally create records in the database through the database abstraction layer. Rails fixtures generate inserts directly from YAML or CSV items. This means it skips a whole lot of ORM plumbing.

      Unless you batch load all the records created by factories up front you might end up with duplicate efforts. For example: a model and a controller spec which both use the same record will still create it twice in your solution.

      I do agree with you that factories can be more expressive than fixtures and because they’re close to the tests it’s usually easier to read.

      • http://rosenfeld.heroku.com/ Rodrigo Rosenfeld Rosas

        The ORM overhead is not as high as one might expect. Fixtures load faster because of batch insert, not because they bypass your models:

        https://gist.github.com/rosenfeld/6318820

        But you’re right that factories will slow down your full suite in the sense you’ll be recreating the same records in multiple contexts sometimes.

        But on the other hand I find it very hard to read tests relying on fixtures. When you use a factory you know what is the important thing in the test. For instance:

        FactoryGirl.create :field, data_type: ‘text’

        When we do that, we’re stating that we don’t care to what section the field belongs or if it has any parent or what label it has or if it has any children, etc. You just care that a field exists of type ‘text’. This makes it easier to understand the tests instead of trying to understand what parts of the fixtures are required to replicate the use case.

        I’d love to be able to create several savepoints in the database and simply alternate to the wanted context for each examples group without having to recreate the same records over and over for the different contexts. But I have no idea how to achieve that or whether it’s possible…

        The reason why we’ll always have this debate about fixtures vs factories is because none of them are perfect solutions. I’m not trying to convince you to use factories instead because I understand you want to focus so much in your test suite performance that you prefer it over easier understanding of your tests contexts. I don’t. And that’s fine. The important thing is for people to understand the trade-offs before they make their decision when they’re evaluating which approach to adopt.

  • Alexander Balashov

    “We try to use fixtures as much as possible”

    FactoryGirl is extemely slow, that’s true. And factories with their transaction rollback strategy are fast, that’s also true. But factories are not very convenient. But what if we can get the speed of transaction rollback strategy and the convinience of FactoryGirl? I thought that it would be a good solution and created https://github.com/evrone/factory_girl-seeds that solves the problem of slow FactoryGirl.

    1. The biggest problem with FactoryGirl in fact is not slow database. The biggest problem is that every FactoryGirl.create creates associations where each association creates it’s own association and so on. So if just use already created records for associations we get a huge boost. It looks like this

    FactoryGirl.define do

    factory :post do

    title “Demo”

    user { seed(:user) }

    end

    end

    On my test suite it gave me 2x less execution time, about 8m instead of 15m :)

    2. And also you can use these seeds (preloaded records) in your “it” blocks just like fixtures.

    user = FactoryGirl.seed(:user)

    It does not create record in DB, it just loads already created record.

    Read more on https://github.com/evrone/factory_girl-seeds . I hope it helps everyone :heart: :).

  • Alexander Balashov

    https://github.com/jonleighton/spring is really a great option to start test, especially when you need to start just one spec but very fast.