System design · Module 04 of 29 · Level 1: Foundations

Availability, reliability and SLOs

Everything fails eventually: disks, servers, networks, whole data centres, and most often, our own changes. This module gives you the vocabulary and arithmetic of failure: what "four nines" really allows, why chaining services lowers availability and spares raise it, and how teams turn "be reliable" into a measurable target with a budget for mistakes.

1. The idea

A car with one tyre puncture stops. A car carrying a spare stops for ten minutes. A lorry with twin wheels on each side keeps driving. And a car whose route passes through five toll gates is late whenever any one of them is jammed. Redundancy adds safety; every extra thing you depend on in a row takes some away.

Availability: The fraction of time (or of requests) the system answers correctly. Usually written in nines: 99.9% is "three nines".
Reliability: How consistently it does the right thing over time: not just up, but correct.
Durability: Data you were told is saved isn't lost. A system can be down for an hour (unavailable) without losing anything (durable), or up while having lost data.
Fault vs failure: A fault is one part going wrong (a disk dies). A failure is the whole system letting users down. Fault tolerance is stopping faults from becoming failures.
Single point of failure (SPOF): A part with no backup: if it breaks, everything breaks. Finding and removing SPOFs is a big part of design.

Availability has two levers. Fail less often (better hardware, fewer risky changes) and recover faster. In numbers: availability = MTBF ÷ (MTBF + MTTR), where MTBF is the mean time between failures and MTTR the mean time to recover. Cutting recovery from an hour to a minute is worth as much as making failures 60 times rarer, and it's usually far easier: automatic failover, fast rollbacks, good alerts.

2. How it works

What the nines allow

Availability	Downtime per year	per 30-day month	per week

Series and parallel

If a request needs components A and B, both must be up: multiply their availabilities. Two 99.9% services in a row give 99.9% × 99.9% ≈ 99.8%. If it needs A or its copy A′ (redundancy), it fails only when both fail: 1 − (1 − a)ⁿ. Two independent 99% servers give 1 − 0.01² = 99.99%.

That word independent carries a lot of weight. Two servers in the same rack share power and a network switch; two in the same data centre share a building. Copies only add nines if they fail separately, so you spread them across failure domains: different machines, racks, availability zones (separate data centres in one region) and, for the strictest needs, regions.

Press Play to raise a small web app's availability one change at a time. The component figures are the cloud providers' published SLA levels where they exist (verified October 2026), used as rough failure rates.

MTBF and MTTR: fail less, or recover faster

Section 1 gave the formula: availability = MTBF ÷ (MTBF + MTTR). MTBF (mean time between failures) is how long the system usually runs before something breaks; MTTR (mean time to recover) is how long it then takes to work again. MTTR itself has four parts: noticing (alerting), finding the cause (dashboards, logs), fixing it (rollback, failover) and confirming it's fixed. Move the sliders, or compare the two presets:

The presets make the point: a system that breaks every week but recovers in two minutes is more available than one that breaks every two months but takes an hour to fix. Rare failures are also the ones nobody has practised. That's why mature teams invest in recovery (automatic failover, one-click rollbacks, alerts that page within a minute) at least as much as in prevention.

High availability vs fault tolerance

The two terms are often mixed up. A highly available system recovers from a fault quickly: users may see a short blip of errors while a standby takes over. A fault-tolerant system carries on through the fault with no visible interruption, because enough copies are already doing the work. Same database fault, two designs:

SLIs, SLOs, SLAs and error budgets

"Highly available" means nothing until you measure it. Google's Site Reliability Engineering practice, now used almost everywhere, splits it into three:

SLI (service level indicator): a measured ratio of good events to all events. "Share of requests that returned non-5xx in under 300 ms."
SLO (service level objective): the target for an SLI over a window. "99.9% of requests are good, over a rolling 30 days."
SLA (service level agreement): a contract with customers, with penalties (usually credits) if missed. Always looser than your internal SLO, so you find out before your customers do.

The gap between the SLO and 100% is the error budget. A 99.9% SLO over a month that serves 100 million requests allows 100,000 bad ones. While budget remains, teams ship features and take risks. When it's spent, the agreed error budget policy kicks in: slow down releases and spend the time on reliability. It turns "how reliable should we be?" from an argument into arithmetic.

Why not 100%?Users can't tell 99.999% from 100%, because their own Wi-Fi, phone and ISP fail far more often than that. Each extra nine costs a lot more (more redundancy, slower releases), so aim for the level users actually notice, and spend the rest of the budget on moving faster.

3. Try it yourself

Availability composer

A request passes through each component in order. Change how many independent copies each has, or take a dependency off the critical path (for example, the app can still take orders when the email service is down).

Error budget

4. Trade-offs

Each nine costs more

Going from 99% to 99.9% is mostly good practice. 99.99% needs multiple zones, automatic failover and careful releases. 99.999% needs multiple regions and a team built around it. Pick the target the product needs.

Redundancy adds complexity

Copies must be kept in sync, failover must be tested, and two nodes that each think they're in charge ("split brain") can corrupt data. Modules 14 and 21 handle this properly.

Availability vs consistency

When parts of a system can't talk to each other, you can keep answering (maybe with stale data) or refuse until they reconnect. That choice is CAP (Module 16), and it's made per feature.

Degrade instead of failing

Not every dependency is critical. If recommendations are down, show the page without them. If email is down, queue the email. Taking a component off the critical path is often the cheapest nine you'll ever buy.

5. In real systems

Published SLA (October 2026)	Monthly uptime
Amazon EC2: a single instance	99.5%
Amazon EC2: the service across a region (instances in two or more zones)	99.99%
Amazon RDS: a Multi-AZ database (standby in another zone, automatic failover)	99.95%
Amazon S3 Standard (designed for 99.99% availability and 99.999999999% durability)	99.9%

Notice the jump from one instance (99.5%) to instances spread across zones (99.99%): that's the parallel formula in a cloud contract. Google Cloud and Azure publish similar tiers. Remember that an SLA is a promise of credits, not a guarantee; design for the failures anyway.

Failure domains in the cloud: an instance, a host, a rack, an availability zone (one or more data centres with separate power and networking), a region (several zones in one area). Spread replicas across zones by default; across regions when the product needs to survive a regional outage (Module 27).
The biggest cause of outages is us. Google's SRE book reports that roughly 70% of outages are due to changes in a live system. That's why progressive rollouts, fast rollback and feature flags matter as much as redundancy.
Typical targets: 99.9% for many internal services, 99.95–99.99% for important user-facing ones. Payments, identity and core infrastructure often aim higher, and pay for it.
Durability is a separate goal. It's protected by replicas, backups you've actually restored, and point-in-time recovery, measured as RPO (how much data you can lose) and RTO (how long recovery takes). Module 27 covers disaster recovery.

The techniques up close

The default answer in an interviewState availability as a number per feature (99.9% for reads, 99.99% for checkout), name the failure domains you spread across (two zones), and say how failover happens. Reliability comes from redundancy, automatic recovery and safe changes, not from hoping components don't fail.

Redundancy across failure domains

The default fault-tolerance technique

Run spare copies of every component on hardware that fails independently, and switch to them automatically.

How it works

Two copies only add nines if they fail separately (section 2). Spread them across hosts, racks and availability zones; across regions only when the product must survive a regional outage. Pair each copy with automatic detection and failover, or the spare sits idle while users wait for a human.

Status today

Built into managed cloud services (multi-zone load balancers, Multi-AZ databases, S3); your own services need at least two instances in two zones.

Use it when

Every component on the critical path

Skip it when

Copies in the same rack or zone and calling it redundancy

In the interview

Every tier runs in at least two zones, with health checks and automatic failover.

🚩 Red flags

Redundancy without failover: a standby nobody switches to.
Correlated copies (same rack, same deploy, same config push): they fail together.

Checkpointing and restartable work

Situational · long-running work

Save progress regularly so a crash resumes from the last checkpoint instead of starting over.

How it works

Stream processors (Flink) snapshot state and source offsets together; batch jobs write intermediate results; workflow engines record each step (Module 22); Kafka consumers commit offsets after the work is safe. Combined with idempotent steps, a restart redoes at most the work since the last checkpoint.

Status today

Standard in stream processing, batch frameworks and workflow engines.

Use it when

Jobs that run for minutes to days
Stateful stream processing

Skip it when

Short requests: just retry them

In the interview

The import checkpoints every 10,000 rows, so a crash costs at most one batch.

🚩 Red flags

A six-hour job with no checkpoints, restarted from zero by every failure.

Maintainability

The forgotten requirement · mention it once

How easily people can operate, understand and change the system. Most of a system's cost comes after launch.

How it works

Operability: good monitoring, runbooks, safe deploys and rollbacks, sensible defaults.
Simplicity: fewer moving parts and clear boundaries; every extra technology is something to run at 3 a.m.
Evolvability: versioned APIs, expand-and-contract schema changes, feature flags.

Status today

Rarely listed in interviews, but it's the honest reason for choosing "boring" technology and fewer components.

Use it when

Justifying a simpler design over a cleverer one
Senior and staff interviews, where operating cost matters

Skip it when

Listing it as a top non-functional requirement for a small prompt

In the interview

I'd keep this as one service and one database for now: fewer moving parts to operate, and we can split it when a team boundary or a scaling limit demands it.

🚩 Red flags

Five technologies for a problem one database solves, with no word on who operates them.

6. Remember it

Memory hook: "Chains multiply down, spares add nines"Things you need in a row multiply your availability down. Independent copies of the same thing add nines. Recovery speed counts as much as rarity.

How this hook works: what "Chains multiply down, spares add nines" means

What the slogan means, phrase by phrase

"Chains": Components in series: the request needs all of them. Load balancer and app and database and payment API.
"multiply down": Availability = a₁ × a₂ × …. Every link is a little below 100%, so the product is always lower than the weakest link.
"spares": Independent copies, in different failure domains, any one of which can serve.
"add nines": 1 − (1 − a)ⁿ. Two 99% copies give 99.99%: each copy adds roughly as many nines as one copy has.

Why this picture fits: a chain is only as strong as its weakest link, and a rope of many strands survives one snapping. You already know both intuitions; the formulas just put numbers on them.

The story → the concept

In the story	In the system
Links of a chain	Services on the critical path (in series)
Strands of a rope	Redundant replicas (in parallel)
Strands cut by the same knife	Replicas in the same failure domain (correlated failures)
Unhooking a link you don't need	Graceful degradation, async calls, queues
How fast you can re-tie a strand	MTTR: failover, rollback, alerting

Watch it on an example

An app promises 99.99% but calls five internal services, each 99.95%, one after another.

Chain: 0.9995⁵ ≈ 99.75%, before counting its own bugs. The promise is already broken on paper.
Spares can't fix a dependency you don't own; you can only change the chain.
Make two of the calls asynchronous through a queue, and cache the results of a third with a fallback: the chain is now two links, 0.9995² ≈ 99.9%.
Be honest about the rest: 99.99% needs the remaining dependencies to offer better than 99.99% themselves.

The trap it saves you from

Promising more nines than your dependencies have, and adding "redundancy" by putting both copies in the same zone.

It still works when…

You reason about durability: data copied to three independent places is lost only if all three fail before a copy is repaired. Fast repair (MTTR again) is what keeps that probability tiny.

7. Common mistakes

Correlated "redundancy"Two replicas on the same host, rack or zone, or sharing one config change, fail together. Spread copies across failure domains, and roll out changes gradually.

Promising more nines than your dependenciesA service on top of a 99.9% database can't offer 99.99% unless it can work without the database. Do the series multiplication first.

Measuring "server is up" instead of "users succeed"A server that answers health checks but returns errors is up and useless. Define SLIs from the user's point of view: successful, fast-enough requests.

Ignoring recovery timeRedundancy with a 30-minute manual failover is 30 minutes of downtime per incident. Automate failover, and practise it.

Aiming for 100%It's unreachable, infinitely expensive, and freezes releases. Pick an SLO and use the error budget.

8. Design exercise (25 minutes)

Find the SPOFs. Draw an app with one load balancer, two app servers in the same zone, one database and a third-party email API called during sign-up. Circle every single point of failure.
Do the math. Estimate the end-to-end availability using the composer above. Which one change adds the most?
Redesign. Change the design to reach 99.95% for sign-ups, using spread replicas and by taking email off the critical path. Draw it.
Write an SLO. For one screen of an app you use: the SLI (what counts as good), the target, the window, and what happens when the error budget runs out.
Write it down. In your design notebook: the nines table for 99.9%, 99.95% and 99.99%, and the two formulas.

9. Self-check quiz

How much downtime does 99.9% availability allow in a 30-day month?

0.1% of 43,200 minutes = 43.2 minutes. 99.99% allows about 4.3 minutes.

A request needs two services in a row, each 99.9% available. What's the combined availability?

In series, multiply: 0.999 × 0.999 ≈ 0.998.

Why should replicas be placed in different availability zones?

The parallel formula assumes independent failures. Replicas sharing power, network or a building can all go down at once.

What's the difference between an SLO and an SLA?

The SLO is your own goal; the SLA is what you promise customers, set looser than the SLO so you have a safety margin.

Two app servers run on the same rack. Do they make the tier highly available?

Copies add nines only when they fail independently. Spread them across zones.

10. Go deeper

Resource	Time	Why
Service Level Objectives (Google SRE book)	30 min	The chapter that defined SLIs, SLOs and SLAs as the industry uses them.
Embracing Risk (Google SRE book)	30 min	Why 100% is the wrong target, and where error budgets come from.
Implementing SLOs (Google SRE workbook)	45 min	A practical walk-through: choosing SLIs, setting targets, writing an error budget policy.
Availability table (SRE book appendix)	5 min	The nines and their downtime, on one page.
Reliability Pillar (AWS Well-Architected)	1–2 h	Failure domains, redundancy and recovery as practised in the cloud.