How to get 4x the performance out of Heroku with Unicorn

Vote on Hacker News

heroku_unicorn

Update: After the shitstorm surrounding Heroku’s routing they have updated their docs and now regard Unicorn as their default Rails server. They now provide 2x dynos with twice the RAM so you can run bigger applications with unicorn or run more unicorn workers on one server. Last but not least New Relic updated their gem to make sure that the queuing time shown for a Heroku application is correct. We have left the rest of the blogpost untouched as it still reflects how you can set up Unicorn on your Heroku machines. But take the changes in the recent Heroku blogposts into account.


This is the first of a two part series on how we set up Codeship with Heroku. The second part deals with Assets, Sprites and Amazon Cloudfront. This guide is only relevant and tested with the Heroku Cedar stack.

I have had several debates over the last couple of months whether Heroku is the way to go and especially if it is expensive or not. They provide a great service, but their documentation makes them look pretty bad when it comes to price. $35 for basically one single concurrent request seems very expensive, especially when starting out with a new project (although of course the first dyno is free). Heroku provides plenty of resources, but as it only allows to listen on one port you can run only one thin instance (as recommended by their documentation). What we need here is a webserver that listens on one port, but can work through several concurrent requests. Sounds like a job for Unicorn.

What is Unicorn

Unicorn is a ruby http server that starts one master process listening on one port and forks several worker processes. Every incoming client request is handed to a worker by the master and when finished the master returns the result to the client. Thus it only needs to listen on one port, but can work on several concurrent requests. Defunkt wrote a nice blogpost about unicorn some time ago that goes into more detail how GitHub uses it.

Setup

To start using Unicorn all you have to do is:

1. Create a Procfile

2. Add unicorn config in config/unicorn.rb

3. Set the default Logger in application.rb (and not just production.rb) to STDOUT, otherwise logging doesn’t work. Thx to @krainboltgreene for mentioning that just setting this in production.rb is not enough.

Unicorn config

You can find all config parameters in the unicorn documentation. Let’s go quickly through the configuration we use for worker_processes: Setting the number of worker processes timeout: Time after which a worker is restarted if unresponsive preload_app: Load the application before forking workers. Set to true if you use NewRelic (which you should) or you won’t see any data before_fork/after_fork: Disconnect in before_fork and reconnect in after_fork for your Database, Resque or other services. Without those handlers there will be regular database errors.

New Relic

Go to the NewRelic Addon of your Heroku application and check your dynos and memory consumption in the dyno tab.

Average memory consumption

Check your Memory Consumption in NewRelic and set the worker_processes accordingly. The average consumption is shown on the Dyno tab of your NewRelic dashboard.

menu

One Heroku Dyno has 512Mb of Memory, so make sure your combined workers do not exceed that maximum amount, or your dyno will be shut down.

memory

On the right hand side of that same tab you can see the number of dynos. Make sure it is the same as you set in worker_processes.

dynos

Benchmarks

I ran several tests with ApacheBench to determine how much the performance improved. I ran 1000 Requests with 100 concurrent connections against the landing page of our staging application. The following graph shows the time the requests took combined with 1-4 workers. Going from one process to several increases performance drastically, from then on it is still a boost to your application, but not as drastically. However you have to find the right spot on how many workers you want unicorn to fork depending on your application. Having too many may shut down your dyno due to memory constraints.

Conclusion

So in closing using Unicorn as your Heroku Webserver not only pays off, but should be put into the Heroku documentation at least as advanced information. I actually talked to people and showed them our Unicorn setup, which convinced them that Heroku is not as expensive as it seems and especially when starting your project is a very viable alternative to having your own Server.

With every new dyno you get several more concurrent requests, which is pretty neat. If you have any questions regarding the setup or anything else you can send an email to  flo@codeship.io or tweet to @Codeship.

Codeship – A hosted Continuous Deployment platform for web applications

Thanks

This post is very much built on Michael van Rooijen’s Blogpost. Gists that helped with the setup were by leshill and jamiew


Download Efficiency in Development Workflows: A free eBook for Software Developers. This book will save you a lot of time and make you and your development team happy.

Author: Florian Motlik

At Codeship I am responsible for the general tech vision and making sure that all of our users are happy and keep their build green. I've always been interested in helping people build great software, great products and just in general make something happen.

Posted by: Florian Motlik | Conversation: 12 comments | Category: Most Popular, Uncategorized | Tags: , , ,

  • Woni Telluh

    Your point #4 says: “Set the default Logger in application.rb (and not just production.rb) to STDOUT, otherwise logging doesn’t work. Thx to @krainboltgreene for mentioning that just setting this in production.rb is not enough.”

    But it does not say why “not just production.rb”. Do you know by any chance why setting this in production.rb is insufficient?

    For instance, this article – http://help.papertrailapp.com/kb/configuration/unicorn – which Heroku Support refers people to, does not mention application.rb, rather <environment>.rb.

    Thoughts?

    • https://www.codeship.io/ Florian Motlik

      Sorry for the late reply. The problem was simply that we didn’t get sufficient log output in development and test then

  • Pingback: Procfile, Foreman and Unicorn | Christopher Concepcion

  • Chase Tran

    Excuse my lack of knowledge, how does a thin server work on Heroku compared to unicorn?

    • https://www.codeship.io/ Florian Motlik

      Hi Chase,

      When you run a thin server it can only work on one request at a time. With Unicorn or Puma (read our latest blog post about puma here: http://blog.codeship.io/2013/10/16/unleash-the-puma-on-heroku.html) the server works on several requests at a time.

      Thus you use the resources Heroku gives you a lot better. We’ve seen major speed improvements with switching to Unicorn first and then to Puma

      • Chase Tran

        Even with 3 dynos, the thin server serves one request at a time?

  • Pingback: Rails Tutorial: Setting Up a Server on Heroku for Free!Blog de Code

  • Grace Gimon

    I have a problem setting up this configuration, the worker_processes value is 2 but in New Relic it shows 10 instances running ( See that I have 2 web dynos and 1 worker).

    If you have any idea why would this be happening, it would be great, thanks.

    • https://www.codeship.io/ Florian Motlik

      2 worker_processes x 2 dynos + a Heroku worker should be 5. Could you check in your logs when you restart the dynos it should show you the number of workers it uses.

      • http://kevinhq.com/ Kevin

        I have similar problem.
        I set up this on unicorn.rb :
        worker_processes Integer(ENV["WEB_CONCURRENCY"] || 2)

        with 1 dyno (1GB RAM), no worker.

        but instances running in New relic sometimes shows 5 instances running!
        In average it shows 2 intances running which is correct.

  • Matt

    Major flaw to this setup – Why is the worker killed on deployment? shouldn’t background tasks continue to run on worker till task is finished, then deploy? It seems this would be a common scenario with continuously deploying code, but the long-running worker tasks like imports and jobs that require a few minutes or more are killed.

    • https://www.codeship.io/ Florian Motlik

      Do you mean the Unicorn or a Heroku Worker? Heroku automatically kills workers if they don’t respond after a few seconds to a kill request. That’s a limit of the Heroku platform you have to deal with