Stripe: moving live data without downtime

In Stripe's early days, each customer had at most one subscription, so subscriptions were stored inside the customer record. Years later there were hundreds of millions of subscriptions, and every change to one meant rewriting the whole customer. Stripe moved them into their own table while the API kept serving requests, in four phases, checking every step against production data. It's one of the clearest public accounts of moving live data without downtime.

1. The situation

Stripe Billing stored each customer's subscriptions as an array on the Customer object. That was a sensible start: "In the early days, each Customer object had, at most, one subscription." As customers gained many subscriptions, "any changes to a customer's subscriptions meant updating the entire Customer record". The fix was clear: give subscriptions their own table. The hard part was doing it to hundreds of millions of records behind a payments API that can't stop.

2. What mattered

• No downtime. The API serves businesses' billing around the clock.
• Correctness. A lost or stale subscription means a wrong bill.
• Scale. Hundreds of millions of objects. The post's own arithmetic: at one second per object, migrating 100 million one after another "would take over three years".
• A way back at every step until the old data is finally deleted.

3. The options

4. The decision

Decision: Move subscriptions to their own table online, in four phases

Context: Subscriptions live in an array on each Customer; hundreds of
  millions of them; any change rewrites the whole customer record.
Decision:
  1. Dual writing: write new subscriptions to both stores; copy existing
     ones when they're updated; backfill the rest from offline snapshots.
  2. Change all read paths: compare old and new reads in production
     (GitHub's Scientist) before switching reads to the new table.
  3. Change all write paths: write the new store first, then archive in
     the old one; make the old field raise an error if anything calls it.
  4. Remove old data once nothing reads or writes it.
Consequences:
  + No downtime; every phase checked against real data
  + Reversible until phase 4
  - Two stores and comparison code to run for the length of the migration
  - Every read and write path has to be found and changed, one by one

5. How it works

The four phases, with the tools the post names at each step.

6. The technique in code

A small store that moves subscriptions from an array on each customer to their own table. One update path is missed during dual writing, as happens in real migrations; the comparison catches it before reads switch. The program was run in all three languages; each prints the output below it.

class SubscriptionStore:
    """Moves subscriptions from an array on each customer (old) to their own table (new), in four phases."""

    def __init__(self):
        self.customers = {}        # old store: customer id -> list of subscriptions
        self.subscriptions = {}    # new store: subscription id -> subscription
        self.phase = "old only"    # old only -> dual write -> read new -> write new -> old removed
        self.update_path_fixed = False
        self.mismatches = []

    def create(self, customer, sub_id, plan):
        sub = {"id": sub_id, "customer": customer, "plan": plan}
        if self.phase in ("old only", "dual write"):
            self.customers.setdefault(customer, []).append(dict(sub))
        if self.phase != "old only":
            self.subscriptions[sub_id] = dict(sub)

    def change_plan(self, customer, sub_id, plan):
        if self.phase in ("old only", "dual write"):
            for sub in self.customers[customer]:
                if sub["id"] == sub_id:
                    sub["plan"] = plan
        # the bug: this update path was missed, so it only reaches the new store once fixed
        if self.phase in ("write new", "old removed") or (self.phase in ("dual write", "read new") and self.update_path_fixed):
            self.subscriptions[sub_id]["plan"] = plan

    def backfill(self, overwrite=False):
        """Copy subscriptions that exist only in the old store (or all of them, to repair)."""
        copied = 0
        for subs in self.customers.values():
            for sub in subs:
                if overwrite or sub["id"] not in self.subscriptions:
                    self.subscriptions[sub["id"]] = dict(sub)
                    copied += 1
        return copied

    def _old(self, customer):
        return sorted((s["id"], s["plan"]) for s in self.customers.get(customer, []))

    def _new(self, customer):
        return sorted((s["id"], s["plan"]) for s in self.subscriptions.values() if s["customer"] == customer)

    def list(self, customer):
        if self.phase == "old removed":
            return self._new(customer)
        if self.phase == "dual write":     # experiment: answer from old, compare with new, alert on a mismatch
            old, new = self._old(customer), self._new(customer)
            if old != new:
                self.mismatches.append((customer, old, new))
            return old
        return self._new(customer) if self.phase in ("read new", "write new") else self._old(customer)


def show(subs):
    return ", ".join(f"{i}:{p}" for i, p in subs) or "none"


store = SubscriptionStore()
for cus, sub, plan in [("cus_1", "sub_a", "basic"), ("cus_2", "sub_b", "pro"), ("cus_3", "sub_c", "basic")]:
    store.create(cus, sub, plan)
CUSTOMERS = ["cus_1", "cus_2", "cus_3", "cus_4"]

store.phase = "dual write"
store.create("cus_4", "sub_d", "pro")
print("1 dual write: backfill copied", store.backfill(), "subscriptions; new store has", len(store.subscriptions))
store.change_plan("cus_1", "sub_a", "pro")           # goes through the missed update path
for c in CUSTOMERS:
    store.list(c)
print("  experiment:", len(store.mismatches), "mismatch")
for c, old, new in store.mismatches:
    print(f"    {c}: old {show(old)} | new {show(new)}")
store.update_path_fixed = True
store.mismatches.clear()
print("  fixed the update path; repair backfill copied", store.backfill(overwrite=True))
for c in CUSTOMERS:
    store.list(c)
print("  experiment:", len(store.mismatches), "mismatches")

store.phase = "read new"
print("2 read new: cus_1 ->", show(store.list("cus_1")))
store.phase = "write new"
store.change_plan("cus_2", "sub_b", "basic")
print("3 write new: cus_2 ->", show(store.list("cus_2")), "| old store still says", show(store._old("cus_2")))
store.phase = "old removed"
store.customers.clear()
print("4 old removed: old store has", len(store.customers), "customers | cus_4 ->", show(store.list("cus_4")))

type Sub = { id: string; customer: string; plan: string };
type Phase = "old only" | "dual write" | "read new" | "write new" | "old removed";
type Row = [string, string];

class SubscriptionStore {
  // Moves subscriptions from an array on each customer (old) to their own table (new), in four phases.
  customers = new Map<string, Sub[]>();      // old store: customer id -> list of subscriptions
  subscriptions = new Map<string, Sub>();    // new store: subscription id -> subscription
  phase: Phase = "old only";
  updatePathFixed = false;
  mismatches: [string, Row[], Row[]][] = [];

  create(customer: string, id: string, plan: string) {
    if (this.phase === "old only" || this.phase === "dual write") {
      if (!this.customers.has(customer)) this.customers.set(customer, []);
      this.customers.get(customer)!.push({ id, customer, plan });
    }
    if (this.phase !== "old only") this.subscriptions.set(id, { id, customer, plan });
  }

  changePlan(customer: string, id: string, plan: string) {
    if (this.phase === "old only" || this.phase === "dual write")
      for (const sub of this.customers.get(customer)!) if (sub.id === id) sub.plan = plan;
    // the bug: this update path was missed, so it only reaches the new store once fixed
    if (this.phase === "write new" || this.phase === "old removed" ||
        ((this.phase === "dual write" || this.phase === "read new") && this.updatePathFixed))
      this.subscriptions.get(id)!.plan = plan;
  }

  backfill(overwrite = false): number {
    // Copy subscriptions that exist only in the old store (or all of them, to repair).
    let copied = 0;
    for (const subs of this.customers.values())
      for (const sub of subs)
        if (overwrite || !this.subscriptions.has(sub.id)) { this.subscriptions.set(sub.id, { ...sub }); copied++; }
    return copied;
  }

  old(customer: string): Row[] { return (this.customers.get(customer) ?? []).map((s): Row => [s.id, s.plan]).sort(); }
  fresh(customer: string): Row[] { return [...this.subscriptions.values()].filter((s) => s.customer === customer).map((s): Row => [s.id, s.plan]).sort(); }

  list(customer: string): Row[] {
    if (this.phase === "old removed") return this.fresh(customer);
    if (this.phase === "dual write") {   // experiment: answer from old, compare with new, alert on a mismatch
      const old = this.old(customer), fresh = this.fresh(customer);
      if (show(old) !== show(fresh)) this.mismatches.push([customer, old, fresh]);
      return old;
    }
    return this.phase === "read new" || this.phase === "write new" ? this.fresh(customer) : this.old(customer);
  }
}

const show = (subs: Row[]) => subs.map(([i, p]) => `${i}:${p}`).join(", ") || "none";

const store = new SubscriptionStore();
for (const [cus, sub, plan] of [["cus_1", "sub_a", "basic"], ["cus_2", "sub_b", "pro"], ["cus_3", "sub_c", "basic"]]) store.create(cus, sub, plan);
const CUSTOMERS = ["cus_1", "cus_2", "cus_3", "cus_4"];

store.phase = "dual write";
store.create("cus_4", "sub_d", "pro");
console.log(`1 dual write: backfill copied ${store.backfill()} subscriptions; new store has ${store.subscriptions.size}`);
store.changePlan("cus_1", "sub_a", "pro");           // goes through the missed update path
for (const c of CUSTOMERS) store.list(c);
console.log(`  experiment: ${store.mismatches.length} mismatch`);
for (const [c, old, fresh] of store.mismatches) console.log(`    ${c}: old ${show(old)} | new ${show(fresh)}`);
store.updatePathFixed = true;
store.mismatches = [];
console.log(`  fixed the update path; repair backfill copied ${store.backfill(true)}`);
for (const c of CUSTOMERS) store.list(c);
console.log(`  experiment: ${store.mismatches.length} mismatches`);

store.phase = "read new";
console.log(`2 read new: cus_1 -> ${show(store.list("cus_1"))}`);
store.phase = "write new";
store.changePlan("cus_2", "sub_b", "basic");
console.log(`3 write new: cus_2 -> ${show(store.list("cus_2"))} | old store still says ${show(store.old("cus_2"))}`);
store.phase = "old removed";
store.customers.clear();
console.log(`4 old removed: old store has ${store.customers.size} customers | cus_4 -> ${show(store.list("cus_4"))}`);

import java.util.*;
import java.util.stream.Collectors;

public class Main {
    static class Sub { final String id, customer; String plan; Sub(String id, String customer, String plan) { this.id = id; this.customer = customer; this.plan = plan; } }

    static class SubscriptionStore {
        // Moves subscriptions from an array on each customer (old) to their own table (new), in four phases.
        final Map<String, List<Sub>> customers = new LinkedHashMap<>();   // old store: customer id -> list of subscriptions
        final Map<String, Sub> subscriptions = new LinkedHashMap<>();     // new store: subscription id -> subscription
        String phase = "old only";   // old only -> dual write -> read new -> write new -> old removed
        boolean updatePathFixed = false;
        final List<String[]> mismatches = new ArrayList<>();

        boolean in(String... phases) { return Arrays.asList(phases).contains(phase); }

        void create(String customer, String id, String plan) {
            if (in("old only", "dual write")) customers.computeIfAbsent(customer, k -> new ArrayList<>()).add(new Sub(id, customer, plan));
            if (!in("old only")) subscriptions.put(id, new Sub(id, customer, plan));
        }

        void changePlan(String customer, String id, String plan) {
            if (in("old only", "dual write")) for (Sub s : customers.get(customer)) if (s.id.equals(id)) s.plan = plan;
            // the bug: this update path was missed, so it only reaches the new store once fixed
            if (in("write new", "old removed") || (in("dual write", "read new") && updatePathFixed)) subscriptions.get(id).plan = plan;
        }

        int backfill(boolean overwrite) {
            // Copy subscriptions that exist only in the old store (or all of them, to repair).
            int copied = 0;
            for (List<Sub> subs : customers.values())
                for (Sub s : subs)
                    if (overwrite || !subscriptions.containsKey(s.id)) { subscriptions.put(s.id, new Sub(s.id, s.customer, s.plan)); copied++; }
            return copied;
        }

        String old(String customer) { return show(customers.getOrDefault(customer, List.of())); }
        String fresh(String customer) { return show(subscriptions.values().stream().filter(s -> s.customer.equals(customer)).collect(Collectors.toList())); }

        String list(String customer) {
            if (in("old removed")) return fresh(customer);
            if (in("dual write")) {   // experiment: answer from old, compare with new, alert on a mismatch
                String o = old(customer), n = fresh(customer);
                if (!o.equals(n)) mismatches.add(new String[]{customer, o, n});
                return o;
            }
            return in("read new", "write new") ? fresh(customer) : old(customer);
        }
    }

    static String show(List<Sub> subs) {
        String s = subs.stream().map(x -> x.id + ":" + x.plan).sorted().collect(Collectors.joining(", "));
        return s.isEmpty() ? "none" : s;
    }

    public static void main(String[] args) {
        SubscriptionStore store = new SubscriptionStore();
        String[][] seed = {{"cus_1", "sub_a", "basic"}, {"cus_2", "sub_b", "pro"}, {"cus_3", "sub_c", "basic"}};
        for (String[] r : seed) store.create(r[0], r[1], r[2]);
        List<String> all = List.of("cus_1", "cus_2", "cus_3", "cus_4");

        store.phase = "dual write";
        store.create("cus_4", "sub_d", "pro");
        System.out.println("1 dual write: backfill copied " + store.backfill(false) + " subscriptions; new store has " + store.subscriptions.size());
        store.changePlan("cus_1", "sub_a", "pro");           // goes through the missed update path
        for (String c : all) store.list(c);
        System.out.println("  experiment: " + store.mismatches.size() + " mismatch");
        for (String[] m : store.mismatches) System.out.println("    " + m[0] + ": old " + m[1] + " | new " + m[2]);
        store.updatePathFixed = true;
        store.mismatches.clear();
        System.out.println("  fixed the update path; repair backfill copied " + store.backfill(true));
        for (String c : all) store.list(c);
        System.out.println("  experiment: " + store.mismatches.size() + " mismatches");

        store.phase = "read new";
        System.out.println("2 read new: cus_1 -> " + store.list("cus_1"));
        store.phase = "write new";
        store.changePlan("cus_2", "sub_b", "basic");
        System.out.println("3 write new: cus_2 -> " + store.list("cus_2") + " | old store still says " + store.old("cus_2"));
        store.phase = "old removed";
        store.customers.clear();
        System.out.println("4 old removed: old store has " + store.customers.size() + " customers | cus_4 -> " + store.list("cus_4"));
    }
}

1 dual write: backfill copied 3 subscriptions; new store has 4
  experiment: 1 mismatch
    cus_1: old sub_a:pro | new sub_a:basic
  fixed the update path; repair backfill copied 4
  experiment: 0 mismatches
2 read new: cus_1 -> sub_a:pro
3 write new: cus_2 -> sub_b:basic | old store still says sub_b:pro
4 old removed: old store has 0 customers | cus_4 -> sub_d:pro

7. The numbers

• Hundreds of millions of subscription objects.
• Over three years: the post's estimate for migrating 100 million objects one at a time, at one second each.
• A few hundred lines: "We never attempted to change more than a few hundred lines of code at one time."

Why the backfill can't be a simple loop

Change the numbers and see how long copying takes. Stripe found the records to copy offline, from database snapshots processed with Hadoop and MapReduce jobs written in Scalding, then copied them with a multi-threaded job. The post doesn't give the number of workers, so this is illustrative.

8. What happened

The subscriptions moved, and the new table became the source of truth. The comparison did its job: "Scientist experiments alerted us as soon as a single piece of data was inconsistent in production." Each step was small enough to review and undo, and a final Scalding job confirmed that no subscription was missing from the new table before the old data was removed.

9. Lessons

• Move data in phases, not in one switch. Write both, read new, write new, delete old. Every phase before the last can be undone (Module 30a).
• Compare in production before you trust. Tests use the data you imagined; a comparison on live reads uses the data you have. This is dark launching applied to data.
• Missed code paths are the real risk. The dual-write phase is where a forgotten update path shows up, so make the comparison cover every read path.
• Make the old path fail loudly. Stripe made the old field raise an error if called, so any remaining use was found, not silently served stale data.
• Copy offline, verify twice. Finding the records from snapshots kept load off production; running the same job afterwards proved nothing was missed (Module 23).

10. Your turn

1. Plan the phases (10 min). Pick a field you'd like to move to its own table. Write what each of the four phases changes, and how you'd undo it.
2. List the paths (10 min). Find every place that reads or writes it. Which one is most likely to be missed?
3. Design the comparison (10 min). What exactly would you compare, and what would you do with a mismatch at 3 a.m.?

11. Check yourself

12. Sources and further reading