Design a deployment system

Two thousand services, fifty thousand machines, several regions, and engineers who ship thousands of times a day. Most outages start with a change (Module 04), so the system that makes changes is also the best place to stop them: release in waves, watch each wave, and undo a bad one before most users notice.

1. The prompt

"Design the internal system that deploys new versions of services across the company's fleet, safely and quickly, with automatic rollback of bad releases."

What makes it hard:

• Speed and safety pull in opposite directions. Waiting longer between steps catches more problems and delays every fix.
• "Bad" has to be detected by a machine, from noisy metrics, within minutes.
• The deployment system can become the outage: one global push of a bad change, or a control plane failure that stops services.

2. Frame the problem

3. What it must do

1. An engineer can deploy a release (an immutable artifact plus its configuration) to a service.
2. The system rolls it out in waves, watches health, and stops and rolls back automatically when something's wrong.
3. Anyone can see what's running where, and the full history of changes.

Not now: building artifacts (CI), schema migrations, the feature-flag system itself (we use it), cost allocation.

4. How well it must do it

5. Numbers that change the design

6. The nouns

• Service and its targets (clusters, zones, regions).
• Release: an immutable artifact (by content digest) plus a configuration version.
• Deployment: a plan of waves, each with targets, a bake time and health gates.
• Desired state per target, and the actual state agents report.
• Audit event.

releases      release_id, service, artifact_digest, config_version, created_by, created_at
deployments   deployment_id, service, release_id, strategy, status, current_wave, started_by
waves         deployment_id, n, targets, bake_minutes, gates, status (pending | rolling | baking | done | rolled_back)
desired       target_id → release_id, generation                      (written by the control plane)
actual        host_id → release_id, generation, healthy, reported_at  (written by agents)
audit         event_id, who, what, where, when                        (append-only)

7. The contract

POST /v1/services/{svc}/deployments  {release_id, strategy: "staged" | "fast"}   Idempotency-Key: …
     → 202 {deployment_id}
GET  /v1/deployments/{id}            → {status, waves: [{n, targets, status, gates}]}
POST /v1/deployments/{id}/pause | /resume | /rollback
GET  /v1/services/{svc}/versions     → [{release_id, targets, share_of_capacity}]
agent: GET /v1/targets/{target}/desired?since_generation=…   (long poll) → {release_id, generation}
agent: POST /v1/hosts/{host}/actual  {release_id, generation, healthy}

Data flow: release → deployment plan → wave n: set desired state for its targets → agents pull and switch → health gates over the bake time → next wave, or rollback → audit at every step.

8. First working version

A deployment API, a workflow that walks the waves, a desired-state store, an agent on every host, and an artifact registry.

8.1 Starting a deployment

The API checks permissions and approvals, records the release and the plan, and starts a workflow for this deployment. Only one active deployment per service at a time (a lease, Module 21), so two engineers can't fight over the same fleet.

8.2 Hosts converge

The control plane never pushes commands into hosts. It writes desired state; each host's agent (or the cluster's own controller in Kubernetes) watches it, pulls the artifact, drains, switches and reports. Repeating that is harmless: it's reconciliation, not a script.

8.3 What's running where

Agents report actual state; the API joins desired and actual per service so anyone can see versions, stragglers and failures.

9. Make it hold up

9.1 Waves and blast radius blast radius

9.2 Knowing a release is bad blast radius · rollback in 5 minutes

Rollback means setting desired state back to the last good release: agents converge exactly as they do for a deploy, and the previous artifact is still cached on every host, so it takes minutes.

9.3 The deploy system can fail; services can't static stability

• Control plane vs data plane: the deployment system decides; hosts and clusters run. If the control plane is down, agents keep the last desired state and services carry on (AWS calls this static stability).
• Agents cache desired state and the current and previous artifacts locally.
• Regional control planes, so a fault in one region's deploy tooling doesn't affect others.

9.4 Getting a terabyte of images to the fleet rollout speed

• Content-addressed layers: hosts download only layers they don't have, so a small code change moves a small layer.
• Regional registry mirrors and a cache per cluster.
• Peer-to-peer distribution for big fleets: hosts share pieces with each other instead of all pulling from the registry (Uber's Kraken, Dragonfly).
• Pre-fetch the next wave's artifact during the current wave's bake.

9.5 Configuration is a deployment too blast radius

Flags, limits, firewall rules and routing tables can break everything as fast as code, and they're often pushed globally in seconds. Send them through the same staged waves, validate them before the first wave (schema, limits, a dry run), and keep an emergency path that is still staged, just faster. Cloudflare's 2019 outage and CrowdStrike's 2024 outage were both global pushes of content rather than code (Modules 24 and 27). Feature flags then separate deploying code from releasing behaviour to users.

9.6 A rollout that survives crashes exact

A rollout lasts hours and touches thousands of hosts; the orchestrator will restart in the middle of some of them. Run each deployment as a durable workflow (Module 22): every wave's progress is recorded, so a restart resumes at the right wave instead of starting over or skipping one. Desired state carries a generation number so agents ignore stale instructions.

9.7 The whole design after the deep dives

10. Interviewer follow-ups

11. When things break

12. Wrap-up and the clock

Trade-offs we made:

• Slower, staged rollouts with bakes, in exchange for small blast radius.
• Desired state and reconciliation instead of push scripts.
• Automated canary analysis, accepting occasional false rollbacks.
• Configuration treated exactly like code.
• A control plane that's never on the request path.

What we'd watch: deployments per day and their success rate · automatic rollbacks (true and false) · time from merge to fleet-wide · desired vs actual drift · artifact pull latency · incidents caused by changes.

At 10×: more regional control planes, more artifact caching and peer-to-peer; the workflow logic stays the same.

Common mistakes:

• A script that SSHes into machines.
• Deploying all regions at once "because it's urgent".
• Fleet-wide alarms as the only detection.
• Control plane on the request path.
• Treating config as safe.
• No answer for rollbacks after schema changes.

13. Try it, check yourself, read more

Blast-radius simulator. One region's rollout. Pick a strategy and a bug, and watch the share of the fleet running the new version (green) and the errors users get (red). Detection needs a few minutes of data; rolling back takes 5 minutes.

Model: fleet-wide alarms fire once the extra error rate across the whole fleet exceeds 1% for 5 minutes; canary analysis compares new and old hosts directly and needs 5 minutes of data. Illustrative numbers.