Software architecture · Level 4: Communication and integration · Module 22 of 46

Resilience and observability

In a distributed system something is always failing somewhere. The question is whether one slow or broken service takes everything else down with it, and whether you can tell what's happening while it does. Resilience patterns stop failures spreading; observability lets you see and explain them. This module covers circuit breakers, timeouts, retries and bulkheads, and the three signals every distributed system needs.

See it builtThe Architecture styles track builds this style for one bike-share app, with code you can run: 15 Cell-based.

1. The idea

The electrics in a house. When a heater short-circuits, the breaker for that circuit trips: the heater stops, the rest of the house keeps its lights. Without breakers, one faulty appliance melts the wiring for everything. And the meter and the fuse box labels are how you find out which circuit tripped and why. Resilience patterns are the breakers; observability is the meter and the labels.

Four patterns that stop failures spreading

Timeout: never wait forever. Every remote call gets a deadline shorter than the caller's own.
Retry with backoff and jitter: retry brief failures, waiting a little longer each time, with randomness so thousands of clients don't retry in lockstep. Only retry idempotent operations, and cap the attempts.
Circuit breaker: after enough failures, stop calling the broken service for a while and fail fast (with a fallback), then let a test call through to see if it has recovered.
Bulkhead: give each dependency its own pool of threads or connections, so one slow dependency can exhaust only its own pool.

The circuit breaker's three states

State	What happens to calls	Moves to
Closed	Pass through; failures are counted	Open, when failures cross the threshold
Open	Fail immediately, without calling; a fallback answers	Half-open, after a set wait
Half-open	A few trial calls go through	Closed if they succeed; open again if they fail

Observability: three signals

Logs: what happened, event by event, with a correlation ID (Module 10) so one request can be followed across services.
Metrics: numbers over time: request rate, error rate, latency percentiles, queue depth, breaker state. They tell you that something is wrong.
Traces: one request's path through every service with the time spent in each. They tell you where.

2. How it works

A circuit breaker, call by call

The checkout service calls a recommendations service. The breaker opens after 5 failures in its last 10 calls and waits 10 seconds before testing again. Watch the counters.

Bulkheads: one slow dependency, one pool

Checkout calls Payment and Recommendations, using 40 worker threads. Recommendations becomes very slow. Watch the threads.

3. Try it yourself

An outage, with and without a breaker

Checkout calls recommendations 50 times a second with a pool of 50 threads. Recommendations goes down at 0:20 and comes back at 1:10. Choose the protection and watch how many of checkout's threads are stuck waiting.

Simulation in one-second steps. A call to a down service holds a thread for the full timeout. The breaker opens once 5 of the last 10 calls fail, fails fast while open (a fallback shows popular items), then sends one trial call. With 50 calls a second, threads needed ≈ 50 × time each call is held.

4. Trade-offs

Fail fast vs try harder

Retries recover from blips; circuit breakers stop hammering a service that's truly down. Retries without a breaker turn an outage into a retry storm.

Short vs long timeouts

Short timeouts free threads quickly and fail requests that would have succeeded slowly. Base timeouts on the dependency's measured p99, not on a round number.

Breaker wait time

A short wait detects recovery quickly but probes a struggling service often; a long wait gives it room to recover and keeps the fallback in place longer.

Observability cost

Logs, metrics and traces cost storage, money and some CPU. Sample traces, keep detailed logs for a short time, and alert on symptoms users feel (error rate, latency), not every internal blip.

5. In real projects

Decide the fallback before you need it. A breaker that opens and shows an error page is better than a hung page, but a fallback (cached results, popular items, "we'll email your receipt") is better still. The fallback is a product decision.

Where to put these patterns: libraries (Resilience4j, Polly and similar) inside each service, or a service mesh's sidecars (Module 16) so every service gets the same behaviour without code.

Observability is not optional in a distributed architecture. OpenTelemetry is the common standard for producing logs, metrics and traces; dashboards and alerts on service-level objectives (Module 05) turn them into action.

Say it in the interview

Two-sentence answers

"How does a circuit breaker work?" It counts failures while closed, opens past a threshold and fails fast with a fallback, then after a wait lets a trial call through: success closes it, failure reopens it. It stops one failing service consuming the caller's threads.
"What's a bulkhead?" Separate thread or connection pools per dependency, so one slow dependency can only exhaust its own pool and the rest of the service keeps working.
"Logs, metrics or traces?" All three: metrics show that something is wrong, traces show where, logs show what happened in detail. Correlation IDs tie them together.

Red flags interviewers notice

Remote calls with no timeout.
Retrying non-idempotent operations, or retrying without backoff.
"We'll check the logs" as the whole observability plan.

6. Remember it

Memory hook: "Trip the breaker, label the fuse box"Contain failures with breakers, timeouts and bulkheads; make them findable with logs, metrics and traces.

How this hook works: what "trip the breaker, label the fuse box" means

What the slogan means, phrase by phrase

"Trip the breaker": Stop calling what's broken, fail fast and fall back, so the failure doesn't spread.
"label the fuse box": Observability: know which circuit tripped, when and why.

Why this picture fits: nobody runs a house without breakers or labels; a distributed system without them is that house.

The story → the concept

House electrics	Distributed system
A breaker per circuit	A circuit breaker per dependency
Separate circuits for kitchen and lights	Bulkheads
Flicking it back on carefully	Half-open trial calls
The meter and labelled fuse box	Metrics, logs and traces

The trap it saves you from

A non-essential feature taking the whole checkout down.

7. Common mistakes

No timeoutsThe default timeout in many clients is infinite or very long. One hung dependency then holds every thread.

Retry stormsEvery layer retrying three times multiplies load on a struggling service: 3 × 3 × 3 = 27 calls per user request.

One shared poolAll dependencies sharing one thread pool lets the slowest decide for everyone.

Alerting on everythingHundreds of alerts on internal metrics train people to ignore them. Alert on what users feel.

8. Design exercise (30 minutes)

Audit the calls (10 min). List every remote call one service makes. Write its timeout, retry policy and fallback. Note any that have none.
Size the pools (10 min). For each dependency, estimate threads needed = calls per second × p99 latency. Would one slow dependency exhaust a shared pool?
Trace one request (10 min). Could you find, today, which service made one slow request slow? Write what's missing.

9. Self-check quiz

What does a circuit breaker do in the half-open state?

It tests whether the dependency has recovered before letting full traffic back.

50 calls a second to a dead service, each holding a thread for a 2-second timeout. How many threads are busy waiting?

Concurrent threads ≈ arrival rate × holding time = 50 × 2.

A slow recommendations service is exhausting the threads checkout needs for payments. What helps most?

The slow dependency can only exhaust its own pool.

Why add jitter to retry backoff?

Synchronised retries hit a recovering service with a wave of load.

What does a distributed trace show that metrics don't?

Metrics say something is slow; traces say which hop.

10. Go deeper

Resource	Time	Why
CircuitBreaker (Martin Fowler)	10 min	The pattern with a worked example.
Timeouts, retries and backoff with jitter (Amazon Builders' Library)	20 min	How a large provider sets timeouts and avoids retry storms.
Observability primer (OpenTelemetry)	15 min	Logs, metrics and traces, and how they fit together.