Shopify's path to Rails 3

Shopify's path to Rails 3

The TL;DR version

Shopify recently upgraded to Rails 3!

We saw minor improvements in overall response times but what we’re most happy with is the new API – it means we get to write cleaner code and get features out faster.

However, this upgrade wasn’t trivial – as one of the largest and oldest Rails apps around, the adventure involved jumping through a few hoops. Here’s what we did and what you might consider if you’ve got an established Rails app that you’re thinking of upgrading.

First, some numbers

The first svn check-in to Shopify was on the release date of Rails 0.5. That was in July of 2004, six years ago, which according to @tobi is “roughly 65 years in internet time”.

At that time Shopify had only two active developers. Today it has eleven full time devs working on it.

The Shopify codebase has over 300 files in the app/models directory, over 130 controllers, and almost 100 gem dependencies.
$ find app/models/ -type f | wc -l
     327
$ find app/controllers/ -type f | wc -l
     131
$ bundle show | wc  -l
      95

Over the past 6 years Shopify has been under constant development, amassing nearly 12000 commits. This makes Shopify one of the oldest, most active Rails projects in existence.

Our process

There are many Rails 3 upgrade guides out there, but we didn’t try to follow any of them. We focused on doing as much as we could ahead of time to prepare for Rails 3, and then giving one big final push when 3.0 final was released.

When upgrading a large app to a major release like this we found there are some things you can do to prepare yourself, but at a certain point you’ve just got to bite the bullet and make the final push to get things working.

Bundler

Shopify had been using Bundler in production for 9 months before making the move to Rails 3. Like most, we weren’t convinced of its utility at first, but as the code got more stable we saw how much it helped with deployments and managing development environments. We think Bundler was absolutely the right choice for managing dependencies.

It was pretty painless to use Bundler with Rails 2.3.x, the Bundler documentation has everything that is needed. We’d definitely recommend doing this step ahead of time as it removes one more obstacle in the Rails 3 migration.

XSS

This was a big one. Some more numbers: Shopify has about 100 helper modules and 130 views. The task of updating all of our views/helpers for the new ‘safe by default’ XSS behaviour was a separate migration all its own. This too, we completed a few months before the release of 3.0.

There was no secret way to go about this, just the obvious back-breaking way. Here’s the basic process I followed:

  1. Run the functional tests. Fix any issues that show up there.
  2. Boot up Shopify in my development environment and click around, fixing any issues I see there.
  3. Manually scan through all of the modules in app/helpers, looking for anything suspicious.
  4. Deploy the code to our staging server. Have the team try it out and report any errors to a shared Google spreadsheet (great for collaborative editing).
  5. Code review.
  6. Deploy the code to production and hope that no issues slipped through.

N.B. When new issues come in, do your best to use ack (or some other project search tool) to find any instances of that issue in other views/helpers and correct those as well.

The rest

After getting Bundler and XSS out of the way, the rest of the migration was done as one large chunk. Some of the work in upgrading to Rails 3 was actually going on in parallel to the XSS work.

The first commit to our rails3 branch was made back in February when the first Rails 3 beta was released. At that point we didn’t know how much work it would be to get Shopify running on Rails 3. We were excited about the launch of the beta and the prospect of getting Shopify using it soon.

After a few days of work we ran into some major blockers that were keeping the app from functioning. Work was abondoned on the rails3 branch for 5 months while the 3.0 release became more stable. When the first release candidate came out in July work we resurrected the rails3 branch.

From then (mid-July) until mid-October the rails3 branch saw pretty constant action, never going more than a few days without a commit. There was a lull during the XSS migration, and as devs took on other projects while doing the migration. We remained mindful of the fact that 3.0 final wasn’t yet released and didn’t want to put our changes into production until we had the confidence of that final release.

Since this whole process took several months there was a lot of activity going on in the master branch at the same time. The only advice to offer is merge early and merge often.

When the final release came out we once again underestimated how much work would be involved in getting Shopify the rest of the way on to Rails 3. The day that it was released @tobi put something like the following into our Campfire room “Let’s get Shopify running on Rails 3! Any devs who want to help join the Meeting Room [campfire room].” It was another few weeks before all was finished.

Major stumbling blocks

Routes

Shopify also has lots of routes.

$ rake routes | wc -l
     846

At the beginning of the upgrade process we used the routes rake task that comes with the rails_upgrade plugin but we were still plagued with missing routes throughout the upgrade.

Although our routes tripled in size, the increase was worth it because the new routing API is much nicer to work with.

The old
map.namespace :admin do |admin|
  admin.resources :products, :collection => { :inventory => :get,
    :count => :get },  
    :member => { :duplicate => :post, 
      :sort => :post,
      :reorganize => :any,
      :update_published_status => :post } do |products|        
    products.resources :variants, :controller => "product_variants", :collection => { :reorder => :post, :set => :post, :count => :get }
  end
end
The new
namespace :admin do
  resources :products do
    collection do
      get :count
      get :inventory
    end 

    member do
      post :sort
      post :duplicate
      post :update_published_status
      match :reorganize
    end 

    resources :variants, :controller => 'product_variants' do
      collection do
        get :count
        post :set
        post :reorder
      end 
    end 
  end
end

Libraries

Like everyone else we were tripped up by libraries in need of upgrades for Rails 3 compliance. There was a lot less of this than you’d expect because Shopify implements so much of what it needs internally. Lots of code in Rails core began in Shopify’s code base.

There were updates required to the plugins that Shopify maintains. Otherwise, when we found issues with libraries we were happy to discover that other maintainers were diligent and had already pushed fixes for Rails 3 compatibility, it was just a matter of updating library versions we were tracking.

helper :all

helper(:all) was a configuration option in Rails 2.x. You could add it to a controller and that controller would have access to all helpers modules defined in your application. In 2.x this was part of the default Rails template, but it could be removed for users who didn’t want it.

In Rails 3.0 this has been moved into ActionController::Base and it can no longer be turned off. This can create very weird behaviour like the following: https://gist.github.com/517669

This was causing issues for us since a lot of our helpers define methods with the same name. We ended submitting a patch to Rails that let us continue to use routes with the default naming scheme. The fix is to use the
clear_helpers
method in your
ApplicationController
class ApplicationController
  clear_helpers
  ...
end

Documentation

External services

Shopify integrates with a myriad of external services. Payment gateways through ActiveMerchant, fulfillment services through ActiveFulfillment, shipping providers through ActiveShipping, product search engines, Google Analytics, Google Checkout, the list goes on.

Ensuring that these integrations continued working was very important for us and we would have had issues had we not thoroughly tested them. Don’t overlook this step.

Looking ahead

Towards the end of the upgrade we (jokingly) asked ourselves if it was really worthwhile to upgrade to Rails 3. After all, we were doing just fine with Rails 2.x, and upgrading to 3.0 was not trivial.

To give you an idea of how much code was changed, here’s the diffstat from Github:

But we soon came to realize that there are a lot of exciting things coming in future releases in the 3.x series and this is the way forward. We’re really excited about getting to use stuff like Arel 2.0, Automatic Flushing, Identity Map, and lots of other goodies.

The Rails project and its surrounding ecosystem are moving ahead quickly. By staying on top of it, we can provide the best tools for our developers and the best experience for our customers.

16 comments

  • Priit
    Priit
    November 16 2010, 01:46PM

    Thank you for this nice writeup. How do you compare it to migrating between previous versions in the past? For example, i still remember that moving from 2.0 to 2.1 wasn’t trivial either.

  • Jan M
    Jan M
    November 16 2010, 01:46PM

    Thanks for the write up, I’m in the middle of the upgrade myself and was seriously starting to doubt myself now that it is taking me so long.

    One tip for people reading this though: I found I could catch 95% of the XSS problems by adding a helper to my tests that scanned the response body with a few regexes that scanned for escaped HTML code on the page (things like “<”) and ran the result through HTML tidy with the tidy gem to spot broken HTML. That way you don’t have to check all your views manually.

  • Derek
    Derek
    November 16 2010, 01:46PM

    I very much appreciate what has been done here…just don’t know what I’m actually appreciating. Is there a way to explain all of this in layman’s terms?

  • Sohan
    Sohan
    November 16 2010, 01:46PM

    How long did it take you to complete the migration? Or better, how many man weeks/months?

    I converted a small project with 10 resources and it took me about 30 hours to get it done. I thought for a large project with dependencies it should be a tough task or at times hard to port completely.

  • Chris Moore
    Chris Moore
    November 16 2010, 01:46PM

    Great article! It’s always interesting to hear these sorts of stories.

    One question: How did you manage to keep the rails3 branch in sync with the bugfixes and features you added to master? Every time I try to work in a long-life branch in parallel to daily work I always seem to wind up in horrible merge somewhere. Even rebasing can be pretty hairy along the way.

  • adam
    adam
    November 16 2010, 01:46PM

    Hi guys, could you share with us your rake stats too? I mean, you told us how many models and controllers you have, but how many lines of code are they? And how much is for testing purpose? It would be a very interesting data :)
    Congrats for your app btw

  • Elliot
    Elliot
    November 16 2010, 01:46PM

    Love to read these kinds of technical posts.

  • @Shopify tobi
    tobi
    November 16 2010, 01:46PM

    @Chris Moore: Git is the solution to this. It totally revolutionized the way we work here. Merging is a non issue.

    @Derek: Admittedly this is very nerdy stuff. In effect it means that Shopify continues to use the latest technology so that our developers can be most effective. Usually updates are easy but in the case of Rails3 a lot of work was required. We are just sharing this story so that other people can potentially switch easier.

    @Sohan: Jesse worked on it pretty much by himself for a month. Almost 2 months if you count XSS part.

    @Priit: previous versions were a lot easier.

  • Avi Flombaum
    Avi Flombaum
    November 16 2010, 01:46PM

    Mind sharing the Google Spreadsheet you use to collect bug reports from your team? I find I can’t get my non-devs to use any bug tracking software and they just end up sending me emails that I then need to put somewhere. Maybe if we had a shared spreadsheet/form, they’d use that…

  • Piotr Sarnacki
    Piotr Sarnacki
    November 16 2010, 01:46PM

    Nice post and congrats for upgrade of such big application (300+ files in app/models seems really huge).

    You will be probably interested in that: https://github.com/drogus/rails/commit/333b32b278ec1e8ba9e302dc23ab24bf148a393a

    Unfortunately it will work on 3.1 only and probably you don’t want to be on the edge with such big application, but let’s hope that 3.1 will be stable in near future ;-)

  • Nathan Youngman
    Nathan Youngman
    November 16 2010, 01:46PM

    Wow, Shopify is a massive Rails app. I had no idea it was around over 6 years ago.

    I am wondering what you use, if anything, for user-translatable text. Globalize and its various plugins doesn’t seem quite there yet, our main blocker right now.

    I remember reading about Shopify crowd-sourcing translations some time ago, the way 37signals does with Tolk, but that’s a separate sort of i18n from what Globalize provides.

    Thanks, Nathan.

  • adam
    adam
    November 16 2010, 01:46PM

    Hi Tobi, any hope to have some more info about the code that you could share with us (rake stats, gems used, translation management, etc)?

    That would be really appreciated,

    Thank you!

  • adam
    adam
    November 16 2010, 01:46PM

    ok, i think that i can just stop to hope and check everyday for an answer… thanks anyway ;)

  • Jesse
    Jesse
    November 16 2010, 01:46PM

    @Piotr That’s great. We’ll definitely use that when 3.1 comes out.

  • postmodern
    postmodern
    November 16 2010, 01:46PM

    You can use DataMapper’s IdentityMap (http://datamapper.org/why#identity_map) right now with the dm-rails railtie (http://github.com/datamapper/dm-rails), thanks to the modularity of Rails3.

  • Andy Blake
    Andy Blake
    November 16 2010, 01:46PM

    Nice post. Do you refactor Shopify controllers to new style (with response_with and respond_to)?

Leave a comment ...