GitHub: upgrading Rails on a live app

In early 2017, GitHub.com, an application about ten years old, was still on Rails 3.2. Upgrading the framework under a codebase that size, while engineers keep shipping to it, usually means a long-lived branch that never catches up. GitHub instead ran the application on both versions at once, deployed the new one to a share of servers at a time, and reached Rails 5.2 in about a year and a half without stopping anyone's work.

1. The situation

GitHub's main application had run on Rails 3.2 for years. Rails 3.2 still received backported security fixes; 4.0 and 4.1 didn't, so GitHub deployed only 4.2 and 5.2 to production on the way up. The upgrade "started out as kind of a hobby": one full-time engineer and volunteers, later four full-time engineers plus volunteers.

2. What mattered

• Nobody stops shipping. Feature work continues on the same application throughout.
• No long-lived branch. A branch that diverges for a year never merges cleanly.
• Prove it on production traffic. Tests and CI miss what only real requests do.
• Keep momentum on a project with a small team and a long tail.

3. The options

4. The decision

Decision: Dual-boot the application on the current and next Rails versions

Context: GitHub.com on Rails 3.2, about ten years old, with feature work
  continuing every day; a long-running branch would never catch up.
Decision:
  - Two lockfiles: Gemfile.lock (current) and Gemfile_next.lock (next).
  - Code that must differ checks the version (GitHub.rails_3_2?), with the
    newest behaviour in the else branch so nothing falls back.
  - CI runs both versions; once the next version is green, its build
    becomes required for every push.
  - Roll out: testing environment and click-tests by each team, then a
    share of production servers off-hours, compare exceptions and
    performance, fix, repeat; merge after 30 minutes at peak traffic on
    full production with no visible impact.
Consequences:
  + Upgrade work ships daily in small changes; production is unaffected
  + Every regression is found on a small share of servers, then undone
  - Version conditionals and a second CI build until the upgrade ends
  - Breakage CI misses still happens (local development, slow queries)

5. How it works

One codebase, two versions, and a rollout that only moves forward when the new version behaves like the old one.

6. The technique in code

A dual-boot switch, a version conditional (using a real Rails 5 change: redirect_to :back was deprecated in 5.0 in favour of redirect_back, and removed in 5.1), and a rollout gate that advances only while the new version's error rate stays within 20% of the old one's. The program was run in all three languages; each prints the output below it.

CURRENT, NEXT = "4.2", "5.2"


def boot(env):
    """Dual boot: the same code runs on either version; the lockfile picks which gems load."""
    return (NEXT, "Gemfile_next.lock") if env.get("BOOT_NEXT") == "1" else (CURRENT, "Gemfile.lock")


def go_back(version):
    """Code that must work on both versions branches on the version, like GitHub.rails_3_2?."""
    if version == "4.2":
        return "redirect_to :back"                          # removed in Rails 5.1
    return "redirect_back(fallback_location: root_path)"    # added in Rails 5.0


def rollout(rounds, tolerance=1.2):
    """Advance only while the new version's error rate stays within tolerance x the old one's."""
    for share, new_rate, old_rate, note in rounds:
        ok = new_rate <= old_rate * tolerance
        verdict = "advance" if ok else "roll back and fix"
        print(f"  {share:>4} of servers: {new_rate:>2} vs {old_rate} errors per 10k requests -> {verdict}" + (f" ({note})" if note else ""))
        if share == "100%" and ok:
            return "merged: full production at peak with no visible impact"
    return "not merged yet"


for env in ({}, {"BOOT_NEXT": "1"}):
    version, lockfile = boot(env)
    print(f"boot Rails {version} with {lockfile}: {go_back(version)}")

print("rollout of", NEXT)
print(rollout([
    ("2%", 41, 12, "3 new exception types"),
    ("2%", 14, 12, ""),
    ("10%", 13, 12, ""),
    ("50%", 12, 12, ""),
    ("100%", 12, 12, "30 minutes at peak traffic"),
]))

const CURRENT = "4.2", NEXT = "5.2";

function boot(env: Record<string, string>): [string, string] {
  // Dual boot: the same code runs on either version; the lockfile picks which gems load.
  return env.BOOT_NEXT === "1" ? [NEXT, "Gemfile_next.lock"] : [CURRENT, "Gemfile.lock"];
}

function goBack(version: string): string {
  // Code that must work on both versions branches on the version, like GitHub.rails_3_2?.
  if (version === "4.2") return "redirect_to :back";              // removed in Rails 5.1
  return "redirect_back(fallback_location: root_path)";          // added in Rails 5.0
}

type Round = [share: string, newRate: number, oldRate: number, note: string];

function rollout(rounds: Round[], tolerance = 1.2): string {
  // Advance only while the new version's error rate stays within tolerance x the old one's.
  for (const [share, newRate, oldRate, note] of rounds) {
    const ok = newRate <= oldRate * tolerance;
    const verdict = ok ? "advance" : "roll back and fix";
    console.log(`  ${share.padStart(4)} of servers: ${String(newRate).padStart(2)} vs ${oldRate} errors per 10k requests -> ${verdict}` + (note ? ` (${note})` : ""));
    if (share === "100%" && ok) return "merged: full production at peak with no visible impact";
  }
  return "not merged yet";
}

for (const env of [{}, { BOOT_NEXT: "1" }] as Record<string, string>[]) {
  const [version, lockfile] = boot(env);
  console.log(`boot Rails ${version} with ${lockfile}: ${goBack(version)}`);
}

console.log("rollout of", NEXT);
console.log(rollout([
  ["2%", 41, 12, "3 new exception types"],
  ["2%", 14, 12, ""],
  ["10%", 13, 12, ""],
  ["50%", 12, 12, ""],
  ["100%", 12, 12, "30 minutes at peak traffic"],
]));

import java.util.*;

public class Main {
    static final String CURRENT = "4.2", NEXT = "5.2";

    static String[] boot(Map<String, String> env) {
        // Dual boot: the same code runs on either version; the lockfile picks which gems load.
        return "1".equals(env.get("BOOT_NEXT")) ? new String[]{NEXT, "Gemfile_next.lock"} : new String[]{CURRENT, "Gemfile.lock"};
    }

    static String goBack(String version) {
        // Code that must work on both versions branches on the version, like GitHub.rails_3_2?.
        if (version.equals("4.2")) return "redirect_to :back";              // removed in Rails 5.1
        return "redirect_back(fallback_location: root_path)";              // added in Rails 5.0
    }

    record Round(String share, int newRate, int oldRate, String note) {}

    static String rollout(List<Round> rounds, double tolerance) {
        // Advance only while the new version's error rate stays within tolerance x the old one's.
        for (Round r : rounds) {
            boolean ok = r.newRate() <= r.oldRate() * tolerance;
            String verdict = ok ? "advance" : "roll back and fix";
            System.out.println(String.format("  %4s of servers: %2d vs %d errors per 10k requests -> %s", r.share(), r.newRate(), r.oldRate(), verdict)
                + (r.note().isEmpty() ? "" : " (" + r.note() + ")"));
            if (r.share().equals("100%") && ok) return "merged: full production at peak with no visible impact";
        }
        return "not merged yet";
    }

    public static void main(String[] args) {
        for (Map<String, String> env : List.of(Map.<String, String>of(), Map.of("BOOT_NEXT", "1"))) {
            String[] b = boot(env);
            System.out.println("boot Rails " + b[0] + " with " + b[1] + ": " + goBack(b[0]));
        }
        System.out.println("rollout of " + NEXT);
        System.out.println(rollout(List.of(
            new Round("2%", 41, 12, "3 new exception types"),
            new Round("2%", 14, 12, ""),
            new Round("10%", 13, 12, ""),
            new Round("50%", 12, 12, ""),
            new Round("100%", 12, 12, "30 minutes at peak traffic")), 1.2));
    }
}

boot Rails 4.2 with Gemfile.lock: redirect_to :back
boot Rails 5.2 with Gemfile_next.lock: redirect_back(fallback_location: root_path)
rollout of 5.2
    2% of servers: 41 vs 12 errors per 10k requests -> roll back and fix (3 new exception types)
    2% of servers: 14 vs 12 errors per 10k requests -> advance
   10% of servers: 13 vs 12 errors per 10k requests -> advance
   50% of servers: 12 vs 12 errors per 10k requests -> advance
  100% of servers: 12 vs 12 errors per 10k requests -> advance (30 minutes at peak traffic)
merged: full production at peak with no visible impact

The BOOT_NEXT variable is invented for the demo; the post names the two lockfiles but not how the second is selected.

7. The numbers

• About 10 years: the age of the codebase.
• About 18 months in total: Rails 3.2 to 4.2 took about a year, 4.2 to 5.2 about five months. Rails 5.2.1 reached production on 15 August 2018.
• 1, then 4 full-time engineers, plus volunteers from across the company.
• 30 minutes on full production at peak traffic with no visible impact: the condition for merging.

Try the rollout gate

Invented error rates for the new version at each step. Change the tolerance and the rate at each step, and see where the rollout stops.

8. What happened

GitHub.com reached Rails 5.2 with no long-lived branch and no stop to feature work. Along the way the team cleaned up technical debt the old version had hidden, moved the test suite closer to standard Rails, and replaced custom code with Rails' own features. The post is honest that some breakage still got through: problems in CI, local development and slow queries that neither the test suite nor click-testing caught.

9. Lessons

• Upgrade early and often. The post's first lesson: the closer you are to the latest version, the easier each upgrade is. Falling years behind is how frameworks become legacy (Module 29a).
• Run both versions, not a branch. Dual boot turns one giant merge into many small daily changes, and production never notices.
• One minor version at a time. Deprecation warnings guide each step; skipping versions skips the warnings.
• Gate the rollout on behaviour. A share of servers, compared on exceptions and performance, then undone or widened: the canary from Module 30a, applied to a framework.
• Keep the machinery afterwards. The post advises keeping upgrade infrastructure, such as a build that runs against Rails' main branch, so the next upgrade starts early.

10. Your turn

1. Find your oldest dependency (5 min). Which framework or library in a system you know is furthest behind? How many versions?
2. Sketch the dual boot (15 min). How would your build load either version? Where would version conditionals live?
3. Define the gate (10 min). Which numbers would have to match before you widen the rollout, and over what time?

11. Check yourself

12. Sources and further reading