System design · Module 24 of 29 · Level 5: Production and architecture

Reliability patterns: keeping failures small

In a system of many services, something is always slow or broken. The danger isn't the first failure; it's the chain reaction: callers wait, threads run out, retries multiply the load, and a hiccup in one service takes down five. This module is the toolkit that stops the spread: timeouts, sensible retries, circuit breakers, bulkheads, rate limits and load shedding, plus how to degrade gracefully and test it all on purpose.

1. The idea

A house's electrical system expects faults. Each circuit has a breaker that trips before a short becomes a fire. Circuits are separate, so a broken kettle doesn't turn off the fridge. The main fuse limits how much the whole house can draw. And when power is scarce, you turn off the heated towel rail, not the freezer.

A cascading failure usually goes like this: one dependency slows down → callers' requests pile up waiting → their thread pools and connection pools fill (Little's Law, Module 02) → they start failing their own callers → everyone retries → load multiplies on the service that was struggling in the first place. Each pattern below breaks one link of that chain.

Pattern	Breaks the chain by…
Timeouts	Never waiting forever: a slow call fails fast and frees its thread.
Retries with backoff and jitter	Recovering from brief faults without hammering a struggling service.
Circuit breaker	Stopping calls to a dependency that's clearly failing, so it can recover and callers fail fast.
Bulkhead	Separate pools per dependency, so one slow dependency can't use up all the threads.
Rate limiting	Capping how much any one client can send, protecting the service from abuse and accidents.
Load shedding	Rejecting some work early when overloaded, so the rest still succeeds.
Graceful degradation	Serving a reduced but useful response instead of an error.

2. How it works

How one slow database takes down a whole site

The chain in section 1 is easier to believe once you watch it. Nothing below crashes: one database gets slow, and every layer above it runs out of something while waiting. Press Play.

Two things make cascades nasty. First, the failure travels upwards, towards services that had nothing wrong with them, and the home page goes down because of a stock query. Second, it outlives its cause: once retries and queued work pile up, the system can stay down after the database has recovered, because it is now handling three times its normal load. That's why the fixes below are about letting go quickly, not about trying harder.

Now the same system with a defence on every link. Each number marks where one pattern cuts the chain:

Timeouts and retries

Every network call gets a timeout, set from the dependency's normal latency (a little above its p99), and smaller than the caller's own deadline. Propagate deadlines down the call chain (gRPC does this, Module 07), so downstream work stops when the user has given up.
Retry only what's safe and useful: idempotent requests, on transient errors (timeouts, 503, 429), not on 400 or 404.
Exponential backoff: wait 100 ms, then 200, 400, 800… so retries thin out over time. Jitter: randomise each wait (for example, a random value between 0 and the backoff), so thousands of clients don't retry in synchronised waves.
Cap retries in layers. Three retries at each of five layers is 3⁵ = 243 attempts at the bottom for one user click. Retry at one layer (usually closest to the user or the edge), and use a retry budget (retries may add at most ~10% to normal traffic).

Circuit breakers

Press Play to watch a breaker protect a caller from a failing dependency.

Bulkheads

Named after a ship's watertight compartments. Give each dependency its own connection pool or concurrency limit (say, 20 for recommendations, 50 for payments). If recommendations hang, they can tie up at most 20 threads; checkout keeps working. The same idea at larger scale: separate clusters or cells for different customers or features (Module 28).

Rate limiting

Token bucket (most common): a bucket holds up to B tokens and refills at R per second; each request takes one. Allows bursts up to B, then a steady R. Try it in section 3.
Fixed window: at most N per minute, counted per clock minute. Simple, but allows 2N across a window boundary.
Sliding window (log or counter): counts requests in the last 60 seconds at every moment; smoother, a bit more work.
Return 429 Too Many Requests with Retry-After. Limit per user, per API key, per IP, and per endpoint; at many servers, keep counters in a shared store such as Redis (Design D02).

Load shedding and graceful degradation

When a service is past capacity, accepting everything makes every request slow and most time out. Shedding rejects some work immediately (cheaply, early: at the load balancer or the first thing in the handler), so the requests you do accept finish in time. Shed the least valuable first: background syncs before checkouts, prefetches before user clicks. Degrading serves something useful when a dependency is down: a cached recommendation list, a page without reviews, "estimated delivery unavailable".

3. Try it yourself

Retry storm

A service can handle 1,000 requests a second and normally gets 800. It has an outage. Compare what each client retry policy does to its load, and whether it recovers when the outage ends.

Model: each failed request is retried up to 5 times. Immediate retries wait 100 ms; backoff waits a random time up to 0.5, 1, 2, 4, 8 s; the budget limits retries to 10% of normal traffic. Above capacity, useful throughput drops (time is wasted on requests that time out), as it does in real overloads.

Token bucket

Capacity 10 tokens, refilling 2 per second. Send single requests or bursts and watch what's allowed.

4. Trade-offs

Timeouts: too short vs too long

Too short and healthy-but-slow calls fail (and get retried, adding load). Too long and threads pile up during an incident. Base them on measured percentiles, and revisit them.

Retries can make it worse

They turn short blips into successes, and long outages into overload. Backoff, jitter, caps and budgets keep the first benefit without the second.

Fail fast vs fail open

When an authorisation service is down, failing closed (deny) is safe; failing open (allow) keeps users working. Decide per dependency, deliberately.

Degraded is still a decision

Stale data, missing sections and approximate counts must be acceptable to the product. Agree on the fallbacks before the incident, not during it.

5. In real systems

Libraries: Resilience4j (Java) provides circuit breakers, retries, bulkheads and rate limiters; Polly does the same for .NET. Netflix's Hystrix popularised circuit breakers but has been in maintenance mode for years.
In the network: service meshes and proxies (Envoy, Istio, Linkerd) apply timeouts, retries with budgets, circuit breaking (connection and request limits) and outlier ejection without code changes.
At the edge: API gateways and CDNs enforce rate limits and absorb abusive traffic; cloud WAFs add bot and DDoS protection.
Practice: AWS's Builders' Library describes timeouts, backoff with jitter and load shedding as used inside Amazon; Google's SRE book covers cascading failures and overload in depth.
Chaos engineering: deliberately inject failures (kill instances, add latency, break a dependency) in a controlled way, ideally in production with a small blast radius, to prove these patterns actually work. Netflix's Chaos Monkey started the practice.

The technologies up close

The default answer in an interviewWhen you draw a call to another service (especially a third party), say: timeout at about 2× its p99, at most 2 retries with exponential backoff and jitter, a circuit breaker, and a fallback of X. For public APIs, add per-user rate limits returning 429.

Timeouts, retries, backoff and jitter

The default · on every network call

Every call gets a deadline; only safe failures are retried, with growing, randomised waits.

How it works

Timeout a little above the dependency's normal p99, and shorter than the caller's own deadline; propagate deadlines down the chain so work stops when the user has given up.
Retry only what's safe and useful: idempotent requests, on transient errors (timeouts, 503, 429), never on 400 or 404.
Exponential backoff (100, 200, 400, 800 ms…) thins retries out; jitter (a random wait up to the backoff) stops thousands of clients retrying in synchronised waves.
Retry at one layer. Three retries at each of five layers is 3⁵ = 243 attempts at the bottom for one click. Add a retry budget (retries may add at most ~10% to normal traffic).

Status today

Built into AWS SDKs, gRPC, Envoy and service meshes, Resilience4j and Polly; AWS's Builders' Library documents the approach.

Use it when

Every call to another service, database or third-party API

Skip it when

Retrying non-idempotent writes without an idempotency key

In the interview

Timeout at about 2× its p99, at most 2 retries with exponential backoff and jitter, only on timeouts and 5xx.

🚩 Red flags

Retries without jitter: synchronised waves hammer a recovering service (a thundering herd).
Retries at every layer, multiplying load during an outage.
No timeout at all: threads pile up waiting on a dead dependency.

Circuit breakers

Situational · around flaky dependencies

After repeated failures, stop calling the dependency for a while and fail fast; then test with one request before resuming.

How it works

Closed (calls pass) → too many failures → open (calls fail immediately, often to a fallback) → after a cool-down, half-open (one trial request) → close on success. Prevents cascades and stops retry storms against a struggling service.

Status today

Resilience4j, Polly, Envoy and service meshes; Netflix's Hystrix popularised it but is in maintenance mode.

Use it when

Third-party APIs, databases and service-to-service calls whose failure could pile up threads

Skip it when

Calls with no sensible fallback where failing fast doesn't help anyone

In the interview

A circuit breaker around the payment provider; while open, orders queue for retry and the user sees "processing".

🚩 Red flags

A breaker with no fallback or user-facing behaviour defined.

Bulkheads

Situational · isolate failures

Separate pools (threads, connections, even clusters) per dependency or customer, so one slow part can't consume everything.

How it works

Covered in section 2: the recommendation client gets at most 20 threads, so when it hangs, checkout still has its own. At larger scale: cells per customer group (Module 28).

Status today

Standard in resilience libraries and in cell-based architectures at large companies.

Use it when

Several dependencies of very different importance in one service

Skip it when

Simple services with one dependency

In the interview

Recommendations get their own small connection pool, so if they hang, checkout is unaffected.

🚩 Red flags

One shared pool for everything: one slow dependency starves the rest.

Load shedding and graceful degradation

Situational · past capacity

Reject the least valuable work early when overloaded, and serve something useful when a dependency is down.

How it works

Shed cheaply and early (at the load balancer or the first line of the handler): background syncs before checkouts, impressions before clicks, a stale location ping (a fresher one arrives in seconds). Degrade: cached recommendations, a page without reviews. Overload becomes reduced service instead of an outage.

Status today

Common at large companies (described in AWS's Builders' Library and Google's SRE book); proxies and gateways support priority-based limits.

Use it when

Traffic spikes beyond capacity
Optional features that can disappear temporarily

Skip it when

Critical writes that must never be dropped (use a queue)

In the interview

Under overload we drop analytics events and prefetches first, and show cached recommendations if that service is down.

🚩 Red flags

Accepting everything when overloaded: every request slows down and most time out.

Service mesh resilience (Envoy, Istio, Linkerd)

Situational · many services, one policy

Sidecar proxies beside each service apply timeouts, retries, circuit breaking and outlier ejection without code changes.

How it works

Policies are configured centrally and enforced by the proxies on every call, along with mutual TLS and telemetry.

Status today

Common in large Kubernetes deployments; adds operational complexity and some latency per hop.

Use it when

Dozens of services in several languages needing consistent policies

Skip it when

A handful of services: a library is simpler

In the interview

At this many services, timeouts and retry budgets are set once in the mesh.

🚩 Red flags

Adding a mesh to a three-service design to sound modern.

Chaos engineering

Rare in interviews · a practice

Deliberately inject failures (kill instances, add latency, break a dependency) in a controlled way to prove the defences work.

How it works

Start small (one instance, a short window), measure against SLOs, and widen the blast radius as confidence grows. Netflix's Chaos Monkey started the practice.

Status today

Tooling exists on every cloud (AWS Fault Injection Service, Azure Chaos Studio) and in open source (Chaos Mesh, LitmusChaos).

Use it when

Mature systems with good monitoring and on-call

Skip it when

Systems without observability: you won't learn anything

In the interview

Only if asked how you'd verify resilience: Game days and fault injection in production with a small blast radius.

🚩 Red flags

Proposing chaos testing before basic timeouts and monitoring exist.

Leaky bucket

Situational · smooth output rate

A rate-limiting algorithm that queues requests and lets them out at a fixed rate, like water leaking from a bucket.

How it works

Requests enter a bounded queue (the bucket); a worker drains it at a constant rate; when the bucket is full, new requests are rejected. Compared with the token bucket (section 2): the token bucket allows bursts up to its size, while the leaky bucket smooths output to a steady rate, adding queueing delay.

Status today

Used for traffic shaping in networks and in some gateways (NGINX's limit_req behaves like a leaky bucket with an optional burst).

Use it when

A downstream that needs a steady, even rate (a partner API with a strict limit)

Skip it when

User-facing APIs where short bursts are fine (token bucket)

In the interview

Calls to the partner API go through a leaky bucket at 50 a second, matching their limit; our own API uses a token bucket so users can burst.

🚩 Red flags

Confusing the two: a token bucket permits bursts; a leaky bucket smooths them out.

Lessons from real outages

Public post-mortems are the best evidence for these patterns. One sentence citing a real incident ("this is how Cloudflare's 2019 outage spread") shows you know why a defence exists.

Incident	What happened	The lesson
Cloudflare, 2 July 2019	A new firewall rule with a badly written regular expression backtracked catastrophically and pinned CPUs on every edge server. Rules were pushed globally within seconds, so the whole network failed at once (about 27 minutes).	Configuration is code: roll it out in stages; keep resource guards (CPU limits) in place; prefer regex engines with runtime guarantees.
AWS S3, 28 February 2017	A mistyped input to a maintenance command removed far more servers than intended, including S3's index and placement subsystems in us-east-1. They hadn't been fully restarted in years, and the restart took hours.	Tools need guardrails (remove capacity slowly, never below a minimum); rehearse cold restarts; split big subsystems into cells.
AWS us-east-1, 19–20 October 2025	A race between two automated DNS updaters applied a stale plan and cleanup deleted it, leaving DynamoDB's regional endpoint with no addresses. When DNS returned, EC2's lease manager tried to re-establish leases for all hosts at once and fell into congestive collapse; recovery took about 14.5 hours.	Guard automation against stale writes (versions, fencing, Module 21); a recovering system needs throttled, jittered reconnects (section 2), not a stampede.
Facebook, 4 October 2021	A maintenance command meant to assess backbone capacity disconnected all data centres; an audit tool's bug failed to stop it. DNS servers then withdrew their BGP routes, so Facebook vanished from the internet, and engineers' own tools were down too.	Keep out-of-band access for emergencies; validate dangerous commands; your recovery tools mustn't depend on the system that failed.

6. Remember it

Memory hook: "Fuses, separate circuits, a main switch"Breakers trip on a failing dependency, bulkheads keep circuits separate, and rate limits and load shedding cap the total draw, so one fault never burns the house down.

How this hook works: what "Fuses, separate circuits, a main switch" means

What the slogan means, phrase by phrase

"Fuses": Timeouts and circuit breakers: stop waiting on, and stop calling, something that's failing.
"separate circuits": Bulkheads: separate pools and limits per dependency, so one fault can't drain shared resources.
"a main switch": Rate limits and load shedding: cap and prioritise incoming work, so overload degrades service instead of collapsing it.

Why this picture fits: electrical safety is designed on the assumption that faults will happen, which is the right assumption for distributed systems too.

The story → the concept

In the house	In the system
A breaker that trips	A circuit breaker opening
Resetting it to test	The half-open state
Kitchen and bedroom on different circuits	Bulkheads
The main fuse rating	Rate limits and capacity
Turning off the towel rail first	Load shedding by priority
Candles when the lights fail	Graceful degradation
Testing the smoke alarm	Chaos engineering

Watch it on an example

A product page calls pricing, reviews and recommendations. Reviews starts taking 30 seconds per call.
Fuse: a 300 ms timeout and an open circuit after repeated failures mean the page stops waiting for reviews.
Separate circuits: reviews has its own pool of 20 connections, so pricing calls still get through.
Degradation: the page renders without reviews ("reviews are temporarily unavailable"). The page stays fast; sales continue.

The trap it saves you from

A shared thread pool, no timeouts and eager retries, which together turn one slow, unimportant service into a full outage.

It still works when…

You call an AI model API (Module 29): timeouts, retries on 429 and 529-style overload responses, a fallback model, and per-user rate limits are exactly this toolkit.

7. Common mistakes

No timeoutMany HTTP clients and database drivers default to waiting a very long time, or forever. Set one explicitly on every call.

Retrying without backoff or jitterImmediate, synchronised retries turn a brief outage into a sustained overload.

Retries at every layerThey multiply. Retry in one place, and budget it.

Retrying non-idempotent callsA timed-out payment may have succeeded. Use idempotency keys (Module 07) before retrying writes.

Shedding too lateRejecting a request after doing most of its work saves nothing. Reject early and cheaply.

8. Design exercise (25 minutes)

Protect checkout: it calls inventory, pricing, a fraud check (third party) and email. For each, choose a timeout, a retry policy, a breaker setting and a fallback.
Count the retries: a request passes through gateway → API → service → database driver, each retrying 3 times. How many database attempts can one click cause? Fix it.
Rate-limit an API: free users 100 requests a minute with bursts of 20; paid users 1,000. Write the token-bucket parameters and where the counters live with 50 API servers.
Plan shedding: rank five request types in a food delivery app by what to shed first under overload.
Write it down: the cascading-failure chain, and which pattern breaks each link.

9. Self-check quiz

Why add jitter to exponential backoff?

Without randomness, clients that failed together retry together, in waves that re-overload the service.

A circuit breaker has been open for its cool-down period. What happens next?

If the trial requests succeed it closes; if they fail it opens again for another cool-down.

What does a token bucket with size 10 and refill 2/s allow?

A full bucket permits a burst of 10; after that, requests are allowed as tokens refill, 2 per second.

Which pattern stops a slow recommendations service from using up all of checkout's threads?

A separate, limited pool for each dependency contains the damage.

Five layers each retry 3 times. How many attempts can one user click cause at the bottom layer?

3⁵ = 243. Retry at one layer, with a retry budget.

Why add jitter to exponential backoff?

Identical schedules make every client hit the recovering service at the same moments.

A partner API allows exactly 50 calls a second, evenly. Which limiter fits your outgoing calls?

A leaky bucket drains at a constant rate; a token bucket would let bursts through.

10. Go deeper

Resource	Time	Why
Timeouts, retries and backoff with jitter (Amazon Builders' Library)	25 min	How Amazon sets timeouts and retries, and why jitter matters.
Addressing Cascading Failures (Google SRE book)	45 min	How overload spreads, and the defences, from long experience.
Using load shedding to avoid overload (Amazon Builders' Library)	25 min	Rejecting work early and by priority.
Circuit Breaker (Martin Fowler)	10 min	The pattern's classic short description.
Principles of Chaos Engineering	10 min	How to test resilience deliberately and safely.