Netflix: active-active across regions

Netflix runs its streaming service from several AWS regions at once. Each region serves real users all the time, data replicates between them asynchronously, and when one region is unhealthy, its users are moved to the others. In 2013 Netflix described how it built this active-active setup across the US; in 2018 it described how it cut the time to evacuate a region from about 50 minutes to under 10. Together they show what active-active really costs and why the speed of failover is an architecture problem, not an operations one.

1. The situation

Netflix's 2013 post starts from a principle: failure rates depend on scale and speed of change, and "if you're running at scale and still pursuing high velocity, things will break all the time". The hardest failures are the ones that take down a whole region. Netflix had already worked on surviving a region-wide load balancer outage; the next step was a full multi-regional active-active setup, serving users from more than one US region.

2. What mattered

• Availability through the loss of an entire region.
• Latency: "there should not be any cross-regional calls on user's call path". Each region must serve a request completely by itself.
• Consistency good enough for users, with replication between regions asynchronous.
• Failover speed, which by 2018 mattered more: with over 117 million members, "even short or partial outages affect many of our customers".

3. The options

4. The decision

Decision: Active-active across US regions

Routing: geo-DNS sends users to the nearest region, roughly 50/50; tooling
  (Denominator, over UltraDNS and Route53) can override it to move all users.
Edge: Zuul proxies detect and re-route mis-routed requests, shape traffic and
  shed load during failover.
Data: everything read and written for users lives in Cassandra, with
  asynchronous multi-region replication. Caches (EVCache) invalidate entries
  in the other region through a message (SQS) on every write.
Rule: no cross-region calls on the user's path; replication is asynchronous.
Verification: regular exercises, including "Chaos Kong", evacuating a region.
Consequences:
  + A region can be evacuated while users keep streaming
  + Both regions are exercised by real traffic every day
  - Every application must be deployable and correct in every region
  - Reads in one region may briefly miss a write made in the other

5. How it works

A normal evening, a write, a regional failure, and the failover before and after Project Nimble.

6. The numbers

Before testing active-active, Netflix wrote 1 million records in one region of a multi-region Cassandra cluster under production load, read them 500 ms later in the other region, and all were read successfully. In 2018, a failover took about 50 minutes; try changing the steps the Nimble post lists:

Durations from the Nimble post: about 5 minutes to decide, 3 to 5 to provision capacity, 25 for services to start, 10 or more to proxy traffic gradually to let new instances warm up, and about 5 for devices to follow the DNS change. The faster-decision option is an assumption for illustration.

7. What happened

By 2013, Netflix was running active-active across the US and had proved it with Chaos Kong, its largest outage simulation: most US traffic moved from US-East to US-West, kept there for over 24 hours, then shifted back. The post also reports a real failover that happened before multi-regional validation was finished.

Five years later, the problem was speed. Netflix's services autoscale with daily traffic and aren't over-provisioned to absorb another region's load, so a failover had to compute and add capacity, then wait about 25 minutes for instances to boot and warm up. Pinning every service high would have hidden performance regressions in a huge buffer and needed changes to hundreds of scaling policies. Nimble instead keeps dark capacity: pre-built instances in hidden "shadow" groups, isolated from traffic and metrics, added to the live services only on failover. Failover becomes mostly flipping DNS: under 10 minutes, which Netflix reports was achieved cost neutrally by using existing capacity more cleverly.

8. Lessons

• Active-active is a data decision first. It works because user data lives in a store with multi-region asynchronous replication, and caches are invalidated across regions. Decide what "eventually" may mean for each kind of data (Module 24, 15).
• No cross-region calls on the user's path. Each region must be a complete quantum; one cross-region call couples the regions back together (Module 09).
• Failover you don't exercise doesn't work. Chaos Kong moves real traffic on purpose (fitness functions at the scale of a region; compare GitLab's untested backups).
• RTO is the sum of the slowest steps. Instance start-up was half the failover. Capacity that is already booted, not more automation, removed it (DR strategies: Nimble moves Netflix from scaling on demand towards a warm standby inside an active-active design).
• Elasticity and failover pull against each other. Autoscaling to daily demand saves money and leaves no headroom for another region's traffic; dark capacity was Netflix's way out (22a).

9. Your turn

1. Find the cross-region call (10 min). Trace one user request in a system you know. Does any call leave the region?
2. Time your failover (10 min). List every step of moving traffic away from a region or zone, with the time each takes. Which is the long pole?
3. Choose per data type (10 min). For three kinds of data in your system, can they be written in two regions at once? What happens if they conflict?

10. Check yourself

11. Sources and further reading