System design · Module 27 of 29 · Level 5: Production and architecture

Deployment, multi-region and disaster recovery

Most outages are caused by changes (Module 04), and the worst ones by losing a whole data centre or region. This module covers both ends: how to ship changes many times a day while limiting the damage of a bad one, and how to design for a region going dark, from "we restore from backup in a day" to "users never notice". The key numbers are two: how much data you can lose, and how long you can be down.

1. The idea

A restaurant chain changing its menu doesn't swap every kitchen at once; it tries the new dish in one branch, watches the reactions, then rolls it out, and keeps the old recipe cards handy. It also keeps copies of its recipes and supplier lists off-site, and decides in advance whether a fire at headquarters means "closed for a week" or "the second kitchen takes over by lunchtime".

Shipping code

Containers package an app with its dependencies into an image that runs the same everywhere. Kubernetes (or a managed equivalent: ECS, Cloud Run, App Service) schedules them, restarts failures, scales them and rolls out new versions.
Immutable infrastructure: never patch running servers; build a new image and replace them. Every environment is reproducible.
Infrastructure as code (Terraform, OpenTofu, Pulumi, CloudFormation): networks, databases and permissions defined in reviewed, versioned files instead of console clicks.
CI/CD: every change is built, tested and deployed by a pipeline, the same way every time. Small, frequent releases are safer than big, rare ones: smaller changes are easier to understand and to roll back.

Release strategies

Strategy	How	Blast radius of a bad release	Rollback
All at once	Replace every instance together	Everyone	Redeploy the old version: minutes, with an outage
Rolling	Replace instances a few at a time, behind the load balancer	Grows as the rollout proceeds	Roll back the same way
Blue-green	Run the new version (green) beside the old (blue); switch traffic in one step	Everyone, but only after testing green	Instant: switch back to blue
Canary	Send 1%, then 5%, 25%, 100% of traffic to the new version, checking metrics at each step	The canary's share	Automatic, when metrics degrade
Feature flag	Ship code switched off; turn it on for staff, then a percentage of users	Whoever has the flag on	Flip the flag, no deploy

Two numbers for disasters

RPO (recovery point objective): how much data you can afford to lose, measured in time. "RPO 5 minutes" means at most the last 5 minutes of writes.
RTO (recovery time objective): how long you can be down before service is restored.

2. How it works

A canary that catches a bad release

Going multi-region

Why: survive a regional outage, serve distant users faster, and meet data residency laws (some data must stay in a country or the EU).
Active-passive: one region serves; a second holds asynchronously replicated data and scaled-down capacity, ready to take over. Simpler; failover loses the replication lag (RPO of seconds to minutes) and takes minutes.
Active-active: every region serves users and accepts writes. Best availability and latency, but writes to the same data in two regions must be reconciled (Module 16), or each user's data is homed in one region and only that region writes it. Global databases (Spanner, CockroachDB, DynamoDB global tables, Aurora DSQL) handle some of this, with their own trade-offs.
Routing: DNS or anycast sends users to the nearest healthy region (Module 06); health checks stop sending to a failed one.
Regional independence: each region should be able to run on its own. A "global" dependency (one auth service, one config store, one region's control plane) quietly turns two regions back into one.

Four disaster-recovery strategies

AWS's disaster-recovery whitepaper names four, from cheapest and slowest to most expensive and fastest. Pick by the RPO and RTO each workload really needs (try the chooser in section 3):

Strategy	What's running in the recovery region	Typical RPO	Typical RTO
Backup and restore	Nothing; backups are copied there	Hours (since the last backup)	Hours to a day
Pilot light	Data replicated; core infrastructure defined but switched off	Minutes	Tens of minutes to hours
Warm standby	A scaled-down but working copy of everything	Seconds to minutes	Minutes
Multi-site active-active	Full capacity, serving users	Near zero	Near zero

Backups you can trust

Replication is not a backup: deletes and corruption replicate instantly (Module 14). Keep point-in-time recovery (database log archiving) and periodic snapshots.
Follow the 3-2-1 habit: three copies, on two kinds of storage, one off-site (another region or account). Make at least one copy immutable, so ransomware or a compromised admin can't delete it.
Test restores regularly. An untested backup is a hope, not a plan. Measure how long a full restore takes: that's your real RTO.

Backup vs disaster recovery

The two get treated as one thing, but they answer different questions. A backup answers "can we get the data back as it was?" Disaster recovery answers "can we keep the service running when a whole site is gone?" You need both: a perfect standby region happily replicates a DROP TABLE, and a perfect backup still leaves you down for a day while you rebuild servers to restore it onto.

	Backup	Disaster recovery
Protects against	Mistakes, bugs that corrupt data, ransomware, a bad migration	Losing a data centre or region: fire, power, a cloud outage
What it is	Copies of the data from the past, kept somewhere separate	A plan plus infrastructure: where traffic goes, who decides, in what order things start
Measured by	How far back you can go, and how long a restore takes	RPO and RTO for the whole service
Proved by	Restoring it, regularly	Failing over for real, in a scheduled exercise

RPO and RTO are easiest to see on a timeline. The disaster happens in the middle. Everything written in the red stretch before it is lost; the amber stretch after it is downtime. Pick a strategy:

3. Try it yourself

How many users does a bad release hit?

A release has a bug that makes 20% of requests fail. Choose the strategy and how long it takes to notice.

Model: rolling takes 30 minutes for the whole fleet; canary holds at 1% for 15 minutes and 10% for 15 minutes with automatic rollback when errors rise; blue-green switches everyone at once and switches back in a minute; detection time is how long until a human or alert notices (canary's automated check takes 3 minutes). Each user makes about 2 requests a minute.

Which disaster-recovery strategy?

4. Trade-offs

Speed vs safety of releases

Canaries and staged rollouts take longer to reach everyone, but catch most bad releases on a sliver of traffic. Fast fixes still need a fast path, with the same checks.

Multi-region costs a lot

Double (or more) infrastructure, cross-region data transfer, harder data consistency, and many more ways to be partially broken. Do it for workloads whose RTO and RPO truly need it.

Per workload, not per company

Checkout might need warm standby; the analytics warehouse can restore from backup tomorrow. Tier your systems.

Failover is a feature

A failover nobody has practised usually fails. Run game days: fail over on purpose, on a schedule, and fix what breaks.

5. In real systems

Kubernetes Deployments do rolling updates with configurable surge and unavailability, and roll back to a previous revision with one command; tools such as Argo Rollouts and Flagger add canaries with automatic analysis of metrics. GitOps tools (Argo CD, Flux) deploy whatever is in a repository.
Feature flag services (LaunchDarkly, Unleash, Flagsmith, cloud equivalents, or a simple in-house table) decouple deploying code from releasing features.
Database changes deploy separately from code and must be backwards compatible with the version still running (expand and contract, Module 11).
Cloud building blocks: cross-region replicas and global tables, cross-region backup copies, DNS failover with health checks, and global load balancers. AWS's whitepaper Disaster Recovery of Workloads on AWS describes the four strategies above in detail.
Data residency: regulations such as the EU's GDPR shape where personal data may be stored and processed, which can force region-per-jurisdiction designs.

In an interview, deployment rarely needs more than a sentence ("canary releases with automatic rollback on SLO metrics, feature flags for risky changes"). Multi-region comes up when the prompt says "global" or "must survive a region failure": state active-passive or active-active, how data replicates, the RPO you accept, and how users are routed.

Lessons from real outages

Most big outages start with a change. Two well-documented ones show why rollouts and failover need care:

Incident	What happened	The lesson
GitHub, 21 October 2018	A 43-second network partition during maintenance made the database failover tool promote primaries on the US West Coast. Both coasts then held writes the other lacked, and failing back safely took 24 hours of degraded service.	Automatic failover across regions can create split brain (Module 21); require quorum, prefer staying in-region, and practise failover with fault injection.
CrowdStrike, 19 July 2024	A security content update with a field-count mismatch made the Windows sensor read out of bounds and crash at boot. It went to all hosts at once; Microsoft estimated about 8.5 million Windows devices were affected.	Staged rollouts and canaries apply to data and configuration, not just code; validate inputs; let customers control rollout timing.

The technologies up close

The default answer in an interviewDeployment needs one sentence: canary releases with automatic rollback on error rate and latency; risky features behind flags; infrastructure in Terraform; database changes expand-and-contract. When the prompt says global or "survive a region", state active-passive or active-active, the RPO and RTO, how data replicates, and how traffic moves. And always: backups with point-in-time recovery, tested.

Feature flags

The default · separate deploy from release

A runtime switch around new code: the code ships dark, then is turned on for staff, then 1% of users, then everyone, without a deploy. Turning it off is the fastest rollback there is.

How it works

Evaluation: SDKs download the flag rules and evaluate them locally in microseconds (by user, tenant, country, percentage), streaming updates; no network call per check.
Percentage rollouts hash the user ID so each user consistently sees the same variant.
Kinds: release flags (short-lived), operational kill switches (long-lived), experiments (A/B tests), and entitlements (plan features).
Debt: every flag is a branch in the code. Remove release flags once they reach 100%.

Status today (checked October 2026)

LaunchDarkly, Unleash, Flagsmith, GrowthBook, Statsig, cloud services (AWS AppConfig, Firebase Remote Config) and in-house tables. OpenFeature, a CNCF incubating project, standardises the SDK API across vendors.

Use it when

Risky changes, gradual rollouts, kill switches for expensive features
Trunk-based development: merge unfinished work behind a flag

Skip it when

Database schema changes: those need expand-and-contract migrations

In the interview

The new ranking ships behind a flag: staff first, then 1%, 10%, 100%, watching error rate and latency; turning the flag off is the rollback.

🚩 Red flags

Hundreds of stale flags nobody dares delete.
A remote call per flag check on the request path.

Rolling, blue-green and canary on Kubernetes

The default deploy strategies

Three ways to replace running code. Rolling swaps pods a few at a time; blue-green runs a full new copy and switches traffic at once; canary sends a small share of traffic to the new version and grows it while metrics stay healthy.

How it works

Strategy	How	Rollback	Cost
Rolling	Deployment replaces pods gradually (`maxSurge`, `maxUnavailable`); readiness probes gate traffic	Roll the Deployment back (another rolling update)	None extra
Blue-green	A full second environment; flip the Service or load balancer	Flip back: seconds	2× capacity during the switch
Canary	1% → 5% → 25% → 100% by traffic weight, with automated metric analysis	Automatic abort on bad metrics	Small; needs good metrics

Both versions run side by side in every strategy, so APIs, events and database schemas must be compatible with old and new code at once.

Status today (checked October 2026)

Built-in Deployments do rolling updates. Argo Rollouts and Flagger add blue-green and canary with automatic analysis (querying Prometheus or Datadog), using a service mesh or Gateway API for traffic weights.

Use it when

Rolling for routine changes
Canary with automatic rollback for user-facing services at scale
Blue-green when you need an instant, whole switch

Skip it when

Blue-green for stateful systems whose data can't be shared or switched cleanly

In the interview

Canary through Argo Rollouts: 5% for 10 minutes, automatic analysis on error rate and p99, then 25%, 50%, 100%.

🚩 Red flags

No readiness probe: pods get traffic before they're ready.
A canary nobody measures.

Infrastructure as code: Terraform and friends

The default for cloud resources

Cloud resources (networks, databases, queues, permissions) declared in files kept in Git, reviewed like code, and applied by a tool that compares what's declared with what exists.

How it works

Plan, then apply: Terraform reads its state (what it created before), queries the cloud, and shows a plan of creates, updates and destroys before changing anything.
State lives in a remote backend with locking (S3, GCS, Terraform Cloud) so two people can't apply at once.
Drift: someone changes a setting by hand; the next plan shows it and puts it back.
Modules package repeatable pieces ("a service with its database and alarms").

Status today (checked October 2026)

Terraform moved to the Business Source License in 2023 and HashiCorp became part of IBM in 2025; OpenTofu, the open-source fork, joined the CNCF in April 2025. Alternatives: Pulumi (general-purpose languages), AWS CDK and CloudFormation, Azure Bicep, and Crossplane (cloud resources managed from Kubernetes).

Use it when

Every cloud environment beyond a demo
Recreating a region or an environment from scratch

Skip it when

One-off experiments you'll delete today

In the interview

All infrastructure is Terraform in Git, applied by CI after review, so a second region is mostly a new set of variables.

🚩 Red flags

Hand-made production resources nobody can recreate.
Secrets in state files stored unencrypted.

GitOps

Situational · Kubernetes deploys

The desired state of each environment is a Git repository; an agent inside the cluster continuously makes the cluster match it. A deploy is a commit; a rollback is a revert.

How it works

Pull, not push: CI builds and tests an image and updates a version in the config repo; the in-cluster agent pulls and applies it. CI needs no cluster credentials.
Continuous reconciliation: manual changes in the cluster are detected and reverted (or flagged).
Audit for free: Git history says who changed what, when, and who approved it.

Status today (checked October 2026)

Argo CD and Flux are both CNCF graduated projects and the common choices; they pair with Argo Rollouts or Flagger for progressive delivery.

Use it when

Kubernetes with many services or clusters
Teams that need an audit trail of every production change

Skip it when

Non-Kubernetes platforms where the provider's deploy pipeline is simpler

In the interview

CI publishes images; Argo CD syncs each environment from its Git repo, so every production change is a reviewed commit.

🚩 Red flags

kubectl apply from laptops in production.

Global failover

Situational · surviving a region

Moving traffic and writes to another region when one fails. Traffic is the easy half; data is the hard half, because the standby only has what was replicated.

How it works

Traffic: DNS failover with health checks (TTLs of 60 s or less), anycast global load balancers, or a routing control you flip deliberately (AWS Application Recovery Controller).
Data: asynchronous cross-region replicas lose the last moments of writes (Aurora Global Database replicates with typically about a second of lag); synchronous options give an RPO of zero at the cost of write latency (Spanner, CockroachDB, DynamoDB global tables with multi-region strong consistency, GA since June 2025).
Who decides? Fully automatic regional failover risks split-brain on a flaky network (GitHub 2018). Many teams automate detection and make the final switch a single, rehearsed human decision.
Rehearse: game days that actually fail over; a plan never run doesn't work.

Status today (checked October 2026)

All major clouds provide the pieces: Route 53, Cloud DNS and Azure Traffic Manager health-checked routing, global load balancers, cross-region database replicas and global tables.

Use it when

Stated requirements to survive a region loss
Regulated or revenue-critical services with low RTO

Skip it when

Most products: multi-zone in one region plus cross-region backups is enough

In the interview

Active-passive: Aurora Global Database replicating to a second region with about a second of RPO; failover is a rehearsed runbook flipped through ARC, target RTO 15 minutes.

🚩 Red flags

Never tested failover.
Automatic promotion on any blip, causing split-brain.

Backups and point-in-time recovery

The default · the last line of defence

Replicas copy mistakes instantly; backups let you go back. Point-in-time recovery (PITR) restores a database to any second in a window, such as one minute before a bad migration.

How it works

How: periodic full snapshots plus a continuous archive of the write-ahead log (WAL). To restore to 14:31:59, start from the snapshot before it and replay the log up to that moment.
Restore makes a new database, so a restore takes time proportional to data size and log replayed: measure it, because that's your real RTO.
3-2-1: three copies, two kinds of storage, one off-site; for ransomware, an immutable or separate-account copy (S3 Object Lock, backup vault locks).

Status today (checked October 2026)

Managed databases do this for you: Amazon RDS and Aurora restore to any second within a retention of up to 35 days; Cloud SQL, Azure Database and others are similar. For self-managed PostgreSQL, pgBackRest, Barman and WAL-G handle base backups and WAL archiving.

Use it when

Every database holding data you can't regenerate

Skip it when

Caches and derived data you can rebuild

In the interview

PITR with 14 days of retention, plus daily snapshots copied to a locked vault in another account and region; we test a restore monthly.

🚩 Red flags

"We have replicas, so we don't need backups": a DROP TABLE replicates too.
Backups never restored in a test.

One region, on purpose

The default for most products

Running in several regions is often presented as the mature choice. For most products it is the expensive one: roughly double the infrastructure, cross-region data costs, harder consistency, and failover machinery that must be tested or it won't work. One region spread over three zones, with a tested way back, is the right answer more often than not.

How it works

Signal	Points to one region	Points to several
Downtime the business can accept if a whole region fails	Hours (a region-wide outage is rare)	Minutes, written into contracts
Where users are	Mostly one continent	Several continents, and latency matters (Module 01)
Data rules	One jurisdiction	Laws require data to stay in each country or region
Team	One team, one on-call rotation	A platform team that can run and rehearse failover

What a good single-region setup still has: every tier spread across three zones; backups copied to a second region and restored there on a schedule; infrastructure as code that can rebuild the stack in that region; and a written, timed plan saying how long that would take. That is the backup-and-restore or pilot-light strategy above, chosen deliberately.

Say when you'd change it: "If a contract demands under an hour of downtime for a regional failure, or if we open in Asia, we add a second region, starting with read traffic."

Status today

Cloud providers design each region as several isolated zones so that a single-region, multi-zone deployment survives the common failures (a data centre, a power feed, a network link).

Use it when

Most products, most of the time
Startups and new products

Skip it when

Contracts or laws that demand regional failover or local data
Global products where cross-continent latency hurts

In the interview

One region, three zones, with backups restored weekly in a second region. A regional outage would cost us a few hours, which the business accepts; a second region would double the cost for a rare event.

🚩 Red flags

Active-active across regions for a product with one country's users.
A DR plan that has never been run.

Game days: rehearsing failure on purpose

Situational · before you rely on a recovery plan

A failover or restore that has never been run is a hope, not a plan. A game day is a scheduled exercise where the team breaks something deliberately, with everyone watching, to find out whether recovery works and how long it takes.

How it works

Pick one question. "Can we restore the orders database from last night's backup in under 2 hours?" "Does traffic move when we turn off zone B?"
Write down what you expect, and what will make you stop early (errors above 1%, a real incident elsewhere).
Start small. Staging first; in production, a fraction of traffic or a quiet hour, with the people who'd normally respond.
Run it and time every step, including the slow human ones: finding the runbook, getting access, waiting for approvals.
Fix what you found and run it again. The usual findings are unglamorous: an expired credential, a missing permission, a step that only one person knew.

Good first exercises: restore a backup into a fresh database and check it; turn off one zone; make a dependency slow (add 2 seconds) rather than down; fail over the primary database; let a certificate expire in staging.

Status today

Widely practised since Netflix's Chaos Monkey and Amazon's GameDay exercises; tools such as AWS Fault Injection Service, Azure Chaos Studio, Gremlin and Chaos Mesh inject failures safely (Module 24).

Use it when

Before trusting any failover or restore plan
After big changes to infrastructure or the team

Skip it when

Production experiments without a stop condition or without the responders present

In the interview

Our restore target is 2 hours; we rehearse it every quarter by restoring last night's backup into a fresh database and timing it. The last run took 3 hours, mostly waiting on access, so we fixed that first.

🚩 Red flags

"Backups are fine" with no restore ever timed.
Breaking production without abort criteria.

6. Remember it

Memory hook: "Taste in one branch; keep the recipes off-site"Release to a small slice first and roll back automatically; keep tested, off-site copies of data and decide in advance how much you can lose (RPO) and how long you can be closed (RTO).

How this hook works: what "Taste in one branch; keep the recipes off-site" means

What the slogan means, phrase by phrase

"Taste in one branch": Canaries, staged rollouts and feature flags: expose a change to a small share of traffic, watch the metrics, and roll back before most users ever see a bad version.
"keep the recipes off-site": Backups and replicas in another region or account, immutable and tested, so a lost region or a bad delete is recoverable.
(and decide how long you can be closed): RPO and RTO per workload pick the strategy: backup and restore, pilot light, warm standby, or active-active.

Why this picture fits: a chain of restaurants deals with both everyday changes and rare catastrophes; the disciplines are the same in software.

The story → the concept

Restaurant chain	System
Trying a dish in one branch	A canary release
Taking it off the menu quickly	Automatic rollback, or flipping a feature flag
Recipe copies in a safe elsewhere	Off-site, immutable backups
A second kitchen that's kept warm	Warm standby in another region
Two kitchens both serving all day	Active-active
"Closed for a week" vs "open by lunch"	RTO

Watch it on an example

A payments company tiers its systems: payment processing (RPO ≈ 0, RTO 5 minutes), merchant dashboard (RPO 15 minutes, RTO 1 hour), reporting (RPO 24 hours, RTO 1 day).
Payments runs active-active across two regions with synchronous replication for the ledger; the dashboard runs warm standby; reporting restores from backups.
Every release goes 1% → 10% → 100% per region, with rollback on error-rate or latency SLO burn.
Each quarter, a game day fails one region over on purpose; the measured times are the real RTOs.

The trap it saves you from

A "highly available" architecture diagram with no measured recovery time, and releases that reach every user before anyone checks a graph.

It still works when…

The change is a new AI model or prompt (Module 29): shadow it, canary it on a slice of traffic, compare quality and cost metrics, and keep the old one ready to switch back.

7. Common mistakes

Never testing a restoreBackups that fail to restore, or take three days, are discovered during the disaster. Restore on a schedule and time it.

Releasing to everyone at onceA bad release hits every user before detection. Stage rollouts and automate rollback on metrics.

Hidden single-region dependenciesA second region is useless if login, DNS management or a config service lives only in the first. Map every dependency.

Schema changes that break the running versionDuring a rollout, old and new code run together. Make every migration compatible with both.

Over-building DRActive-active for an internal reporting tool wastes money and adds complexity. Match the strategy to the workload's RPO and RTO.

8. Design exercise (25 minutes)

Design a release process for a mobile banking API: environments, checks, canary steps, the metrics that trigger rollback, and how database migrations fit.
Tier the systems of an e-commerce site (checkout, catalogue, search, recommendations, analytics, admin) with an RPO and RTO each, and pick a DR strategy for each.
Plan multi-region for a social app with users in the US, EU and India and EU data-residency rules. Active-passive or active-active? Where does each user's data live?
Write a game-day plan: what you break, what you measure, what "success" means.
Write it down: RPO vs RTO, and the four DR strategies in order.

9. Self-check quiz

What does RPO measure?

Recovery point objective: the age of the newest data you're guaranteed to recover. RTO is the downtime.

Which strategy limits a bad release to a small share of users and can roll back automatically?

Blue-green switches everyone at once (with fast rollback); a canary exposes only a slice while metrics are checked.

RPO of a minute and RTO of 15 minutes, at reasonable cost. Which DR strategy?

A scaled-down running copy with continuous replication can take over in minutes; backups take hours.

Why isn't a cross-region replica a backup?

A replica faithfully copies mistakes. Backups and point-in-time recovery let you go back to before them.

Why did GitHub's 43-second network partition in 2018 become 24 hours of degraded service?

Cross-region automatic failover created divergent writes; reconciling them safely took a day.

An engineer runs DELETE without a WHERE clause. Why don't read replicas save you?

Replicas protect against machine failure, not mistakes. Point-in-time recovery from backups does.

What makes rollback fastest for a risky new feature?

No build or deploy needed: the code path is switched off at runtime.

A B2B tool's users are all in Europe and contracts allow 4 hours of downtime for a regional outage. What setup fits?

Zones cover common failures; a tested restore in a second region meets a 4-hour target for the rare regional one at a fraction of the cost.

What is the main point of a game day?

Untested recovery plans fail on small things: expired credentials, missing access, unknown steps.

10. Go deeper

Resource	Time	Why
Disaster recovery options in the cloud (AWS whitepaper)	45 min	The four strategies with architectures, RPO and RTO for each.
Canary release, blue-green deployment and feature toggles (Martin Fowler's site)	45 min	The standard descriptions of each release technique.
Release Engineering (Google SRE book)	30 min	Hermetic builds, staged rollouts and configuration management at scale.
The Twelve-Factor App	30 min	Habits that make apps easy to deploy, scale and replace.