System design · Module 02 of 29 · Level 1: Foundations

Latency, throughput and the numbers to know

"Is it fast?" is two different questions: how long does one request take, and how many can we handle at once? This module separates them, shows why the average hides the users who suffer, explains why systems slow down sharply when they get busy, and gives you the hardware numbers that keep your designs grounded in 2026, not 2010.

New to this?Start with B2 · Inside one machine in Level 0; this module assumes it.

1. The idea

Picture a motorway. Latency is how long your trip takes. Throughput is how many cars reach the city per hour. Bandwidth is the number of lanes. Adding lanes lets more cars through, but doesn't make your trip shorter. And when the road gets close to full, everyone's trip gets much slower, suddenly, not gradually.

Latency is the time one request takes, start to finish, usually in milliseconds (ms) or microseconds (µs). It's what a user feels.
Throughput is the work done per second: requests per second (QPS or RPS), messages per second, megabytes per second. It's what your capacity plan is about.
Bandwidth is the most a link or disk can carry per second. Throughput can't exceed it.
Concurrency is how many requests are in progress at the same moment. Latency, throughput and concurrency are tied together by one simple law (section 2).

Latency is never one number. Run the same request a thousand times and you get a spread: most are quick, a few are slow because of a cache miss, a garbage-collection pause, a busy disk or a lost packet. So we describe latency with percentiles: p50 (the median: half are faster), p95, p99 (99% are faster). The slow end is the tail latency, and in big systems it matters more than the average.

2. How it works

Percentiles from real requests

Press Play: twenty requests to the same endpoint, then the average, then the percentiles.

Fan-out makes the tail everyone's problem

Many pages are built by asking dozens of servers at once (a search query fans out to every index shard; a feed asks many services) and waiting for all of them. If each server is slow just 1% of the time, a request that needs 100 servers is slow whenever any of them is:

P(at least one slow) = 1 − 0.99¹⁰⁰ ≈ 63%

A server's rare p99 becomes the user's typical experience. This is the core point of Google's paper The Tail at Scale, and why large systems set targets on p99, not on the average.

Busy systems get slow suddenly

A server that takes 10 ms per request can, in theory, do 100 per second. But requests don't arrive evenly; they bunch up and wait in a queue. In the simplest queueing model, the average time in the system is:

response time = service time ÷ (1 − utilization)

At 50% busy that's 2× the service time; at 80%, 5×; at 90%, 10×; at 95%, 20×. Real systems differ in the details, but the shape is always the same: a gentle slope, then a cliff. That's why capacity plans keep headroom, typically running servers at 50–70% of their limit at peak.

Little's Law

For any stable system, averaged over time:

requests in flight = throughput × latency (L = λ × W)

A service handling 2,000 requests per second at 50 ms each has about 2,000 × 0.05 = 100 requests in progress at any moment. That tells you how many threads, connections or database connections you need. It also shows what happens when a dependency slows down: if latency jumps to 500 ms, the same traffic needs 1,000 slots, and a pool sized for 100 runs out. Slowness turns into failures.

3. Try it yourself

Fan-out and the tail

Each dot is one user request. A request is slow (red) if any of the servers it waits for is slow.

Utilization, queueing and Little's Law

Drag the utilization up and watch the response time. Then see how many requests are in flight for a given traffic level.

The curve is the M/M/1 queueing model: random arrivals, one server. It's a simplification, but it captures why "90% busy" is dangerous. Servers per request in the tiles assume each server runs at the chosen utilization.

4. Trade-offs

Batching: throughput up, latency up

Grouping work (writing 100 rows in one statement, sending messages in batches) raises throughput a lot, but each item waits for its batch. Use it where a few extra milliseconds don't hurt: logs, analytics, background jobs.

Headroom costs money, saves you

Running at 90% looks efficient on a bill and terrible on a latency graph. Keep spare capacity for spikes and for losing a server, and let autoscaling add more before the cliff.

Averages vs percentiles

An average can look fine while 1 in 100 users waits seconds. Set goals and alerts on p95 or p99, and watch p50 for the typical experience.

Cutting the tail

Timeouts stop one slow call holding everything up. Hedged requests send a second copy to another replica if the first is slow and take whichever answers first, trading a little extra load for a much better p99.

5. The numbers to know

You don't need exact figures, only the right order of magnitude. The ladder below uses rounded figures from Simon Eskildsen's napkin-math benchmarks and typical cloud measurements. The last column scales everything up so a memory read takes one second, which makes the gaps feel real.

Operation	Typical time	If a memory read took 1 s

How much one machine can do (2026)

Hardware has grown enormously, and many "you must shard" instincts come from an older era. A large general-purpose cloud server today has around 192 vCPUs, 768 GiB of memory and 50 Gbit/s of network (for example AWS m7i.48xlarge); memory-optimised machines go up to 32 TiB (U7i). Rough capacity of one well-tuned instance, as a starting point for estimates:

Component	Rough capacity	Latency	Think about scaling when…
In-memory cache (Redis, Valkey)	100k–200k+ simple operations per second; memory up to hundreds of GB	Under 1 ms	Sustained 100k+ ops/s, or the data won't fit in one machine's memory
Relational database (PostgreSQL, MySQL)	Tens of thousands of simple indexed reads per second; roughly 10k–20k simple writes per second; tens of TB of data	1–5 ms reads from memory, more from disk	Sustained writes beyond ~10k/s, or data approaching tens of TB
App server	Thousands to tens of thousands of simple requests per second; 100k+ open connections	Depends on your code	CPU above ~70% at peak
Message log (Kafka broker)	Hundreds of thousands to about a million small messages per second	A few ms end to end	Approaching those rates, or consumers can't keep up

Treat these as starting points, not promisesReal numbers depend on payload size, query shape, indexes, hardware and configuration. In an interview, quoting the right order of magnitude ("one Postgres box handles tens of thousands of reads a second") is what counts. At work, measure with a load test.

The numbers in the interview

The default answer in an interviewQuote the order of magnitude, not the decimal: one PostgreSQL box handles tens of thousands of indexed reads a second. Do the math before adding shards, caches or queues, and say what each number means for the design.

Caches can often hold the whole hot dataset; the limit is usually operations per second or bandwidth, not memory.
One database carries millions to tens of millions of users. Still run replicas for availability: replication (staying up) and sharding (scale) are separate decisions.
App servers run out of CPU before memory; spare RAM is good for local caches, and autoscaling is quick enough that you needn't over-provision.
Queues are fast enough to sit inside a synchronous request flow when there's no backlog.
Distance can't be optimised away: same zone ≈ 0.25 ms, another zone ≈ 1–2 ms, another continent 60–180 ms. Keep data close to compute and users.

🚩 Red flags: the four classic mistakes with numbers

Premature sharding. Yelp-sized data: 10 million businesses × 1 KB = 10 GB; ×10 for reviews = 100 GB. That's one database. Even a 400 GB leaderboard fits in one large cache.
Overestimating database latency. An indexed row lookup on SSD takes well under a millisecond to a few milliseconds. Simple lookups don't need a cache; expensive queries might.
A queue for "high" writes. PostgreSQL handles thousands of simple writes a second per core, so 5,000 writes a second needs no queue. Try batching, better indexes and connection pooling first; use queues for guaranteed delivery, decoupling or spikes beyond capacity.
Old hardware assumptions. Designing for 2015-era machines ("8 GB of RAM, so we need 50 servers") over-engineers everything. Think in orders of magnitude and avoid 100 machines where one will do.

Hedged and tied requests

Situational · cutting the tail

Ask a second replica when the first is slow, and use whichever answers first. It attacks the slow 1% directly instead of trying to make every server perfect.

How it works

Hedge after a delay, not immediately: send the backup only once the first request has taken longer than about the 95th-percentile latency, so only ~5% of requests are duplicated.
The classic result (Dean and Barroso, "The Tail at Scale", 2013): reading 1,000 keys spread over 100 servers, hedging after 10 ms cut the 99.9th-percentile latency from 1,800 ms to 74 ms, while sending just 2% more requests.
Tied requests: send to two servers at once, each told about the other; whichever starts first cancels its twin. Lower latency, little duplicated work.
Only for safe reads: duplicating a write is a correctness bug unless it's idempotent (Module 24).

Status today (checked October 2026)

Available in gRPC (a hedging policy in the service config, supported by some language implementations), Envoy (hedging on per-try timeout) and some database drivers (Cassandra drivers' speculative execution). Usually a config switch, not code.

Use it when

Fan-out reads where one slow replica delays the whole answer
Read paths with tight p99 targets and spare capacity

Skip it when

Writes that aren't idempotent
Systems already near capacity: duplicates add load at the worst moment

In the interview

Search fans out to 100 shards, so a single slow shard sets our p99. We hedge reads to a second replica after the p95 delay, about 5% extra load.

🚩 Red flags

Hedging immediately: doubles the load for little gain.
Hedging writes without idempotency keys.

Load-testing tools

The default way to find real capacity

Estimates say what you need; a load test says what one server really does. You generate realistic traffic, raise it until latency or errors break the target, and read off the capacity.

How it works

Tool	Write tests in	Known for
k6 (Grafana)	JavaScript	Developer-friendly scripts, thresholds in CI, arrival-rate executors
Locust	Python	Simple user behaviour as code, distributed workers
Gatling	Scala, Java, Kotlin, JS	High load from one machine, detailed reports
Apache JMeter	GUI / XML	Long-established, many protocols
wrk2, Vegeta	Command line	Constant request rate with correct latency measurement

Closed vs open model: a closed test runs N virtual users who each wait for a response before sending the next, so a slow server automatically receives fewer requests. Real users don't wait for each other: an open test sends requests at a fixed arrival rate regardless. Use open-model tests to find capacity.
Test types: load (expected peak), stress (until it breaks), soak (hours, to find leaks), spike (sudden burst).

Status today (checked October 2026)

All of these are actively maintained; k6 and Locust are the most common choices for new projects. Cloud-hosted runners (Grafana Cloud k6, Azure Load Testing and others) generate traffic from many regions.

Use it when

Before launch and before big events
In CI against a staging environment to catch performance regressions

Skip it when

Production without a plan: test in a production-like environment, or carefully against production with limits

In the interview

I'd load test one instance with k6 at a constant arrival rate, find where p99 crosses 200 ms, and size the fleet at about 70% of that.

🚩 Red flags

Testing with a closed model only and reporting its latency as real.
Load generator on the same machine as the system under test.
Testing one endpoint with cached responses and calling it capacity.

Coordinated omission

Situational · why benchmarks lie

A measurement error where the load generator, waiting for a slow response, quietly stops sending the requests that would have arrived during the stall. The slowest moments get the fewest samples, so reported percentiles look far better than what users experience.

How it works

Example: a tester sends one request every 10 ms. The server freezes for 1 second. A naive tester sends one request, waits the whole second, records one slow sample, then carries on. In reality 100 requests should have arrived during that second, waiting on average half a second each. One bad sample out of thousands barely moves the p99; 100 would.
The fix: measure latency from when each request should have been sent (its schedule), not when it was actually sent; or use an open-model tool that keeps sending regardless. wrk2, k6's arrival-rate executors and Gatling's open injection all do this. HdrHistogram can correct recorded data.
Also in monitoring: averaging percentiles across servers, or averaging p99s over time, is a similar trap. Merge histograms instead (Module 25).

Status today

Named by Gil Tene (Azul); now widely understood, but still the default behaviour of simple closed-loop tools and hand-written benchmarks.

Use it when

Reading or designing any latency benchmark
Explaining why production p99 is worse than the test said

Skip it when

Throughput-only tests, where latency isn't reported

In the interview

I'd use a constant-arrival-rate test, so stalls show up in the percentiles instead of being skipped.

🚩 Red flags

Quoting p99 from a closed-loop benchmark as the user experience.

Profilers and flame graphs

The default way to find where time goes

A profiler samples what each thread is doing many times a second. A flame graph draws those samples: each box is a function, its width is the share of samples it was on the stack, and boxes stack on top of their callers. The widest boxes are where the time goes.

How it works

Reading one: the x-axis is not time: stacks are merged and sorted alphabetically. Look for wide plateaus at the top: functions that are themselves burning CPU.
On-CPU vs off-CPU: a CPU flame graph shows computation; an off-CPU graph shows waiting (locks, disk, network). Slow requests with idle CPUs need the second kind, or tracing (Module 25).
Tools: Linux perf and eBPF profilers for anything; async-profiler for Java; pprof for Go; py-spy for Python; browser and Node.js dev tools. Continuous profiling (Grafana Pyroscope, Parca, cloud profilers) samples production all the time at about 1% overhead, so you can compare before and after a deploy.

Status today (checked October 2026)

Flame graphs were invented by Brendan Gregg in 2011 and are now built into most profilers and observability platforms.

Use it when

CPU is high and you need to know why
Comparing a slow release with the previous one

Skip it when

Latency spent waiting on other services: use distributed tracing

In the interview

Before adding servers I'd profile: a flame graph of the hot path often shows one function, like JSON serialisation or a regex, taking a third of the CPU.

🚩 Red flags

Scaling out a CPU problem nobody has profiled.
Reading the flame graph's x-axis as time.

6. Remember it

Memory hook: "Lanes, trip time, traffic jam"Bandwidth is lanes, latency is trip time, utilization is how full the road is. Near full, every trip gets slow, and the unluckiest trip is the one users remember.

How this hook works: what "Lanes, trip time, traffic jam" means

What the slogan means, phrase by phrase

"Lanes": Bandwidth: how much can flow at once. More lanes raise throughput, not speed.
"trip time": Latency, described by percentiles. The median trip and the worst 1% trip are different numbers, and both matter.
"traffic jam": Queueing. As utilization nears 100%, waiting time explodes. Keep headroom.

Why this picture fits: everyone has sat in traffic that went from fine to gridlock in minutes. Servers behave the same way, for the same reason: arrivals bunch up.

The story → the concept

On the motorway	In the system
Number of lanes	Bandwidth, or the number of servers
Cars per hour reaching the city	Throughput (QPS)
Your trip time	Latency for one request
The slowest trip of the day	p99 or p999 latency
Cars on the road right now	Concurrency (Little's Law)
Rush hour gridlock	High utilization and long queues

Watch it on an example

An API's database has a p50 of 4 ms and runs at 85% busy at peak.

Traffic grows 10% on a sale day: utilization goes from 85% to about 93.5%.
Response time goes from about 6.7× to about 15× the service time: it more than doubles, for 10% more traffic.
By Little's Law, slower requests mean more in flight: the connection pool runs out and requests start failing.
The fix wasn't a faster database; it was headroom (a bigger instance, a read replica, or a cache) before the sale.

The trap it saves you from

Planning capacity with averages ("we're at 85%, there's 15% left"). The last 15% is the traffic jam.

It still works when…

You think about any shared resource: CPU, a disk, a lock, a database connection pool, a Kafka partition. Each has lanes, trip times and a point where the jam begins.

7. Common mistakes

Reporting only the average latencyA 50 ms average can hide a p99 of 2 seconds. Always ask for percentiles, and set goals on the tail.

Planning to run at 90%+ utilizationQueueing makes latency explode near the limit, and there's no room to lose a server. Plan peak load at 50–70%.

Using 2010-era hardware numbersDesigns that shard a 100 GB dataset across ten machines are solving a problem one server doesn't have. Do the math with current numbers.

Confusing bandwidth with latencyA 10 Gbit/s link to another continent still has an 80 ms round trip. Many small calls across it are slow however wide the pipe is.

Averaging percentilesThe p99 of a service isn't the average of each server's p99. Combine raw data or histograms, then compute the percentile.

8. Design exercise (25 minutes)

Read your own percentiles. Run curl against one endpoint 50 times in a loop and write the total times to a file (for i in $(seq 50); do curl -o /dev/null -s -w '%{time_total}\n' https://example.com; done > t.txt). Sort them and read off p50, p90 and the maximum.
Fan-out. A search page queries 40 shards and waits for all of them. Each shard has a p99 of 200 ms and a p50 of 20 ms. Roughly what share of searches take at least 200 ms? What two changes would you make?
Little's Law. Your service gets 5,000 requests per second at 40 ms each, and each request holds one database connection for 10 ms. How many requests are in flight? How many database connections do you need on average, and how many would you configure?
Headroom. A database runs at 60% CPU at peak. Traffic is forecast to grow 50% next year. What utilization will that be, and what would you do now?
Write it down. Five numbers you'll memorise from section 5, in your design notebook.

9. Self-check quiz

What does a p99 latency of 300 ms mean?

p99 is the 99th percentile: only the slowest 1% take longer than 300 ms.

What happens to latency as a server approaches 100% utilization?

Queueing time is roughly service time ÷ (1 − utilization): 10× at 90%, 20× at 95%.

A service handles 1,000 requests per second, each taking 50 ms. About how many requests are in progress at once?

Little's Law: 1,000 per second × 0.05 s = 50 in flight.

Why do systems with large fan-out care so much about p99?

A request that needs 100 servers is slow if any one of them is. A rare per-server event becomes common per request.

Yelp: 10 million businesses at about 1 KB each, plus 10× that in reviews. How many database shards do you need for storage?

10M × 1 KB = 10 GB; with reviews ≈ 100 GB. One database holds tens of TB. Premature sharding is the classic mistake.

Your service writes 5,000 simple rows a second to PostgreSQL. Do you need a queue in front?

Queues are for guaranteed delivery, decoupling or spikes beyond capacity, not for a load the database already handles.

When should a hedged request be sent?

Waiting until p95 keeps the extra load to about 5% while still cutting the tail.

A load tester waits for each response before sending the next. The server stalls for a second. What happens to the reported p99?

Coordinated omission: requests that should have queued during the stall were never sent, so never measured.

10. Go deeper

Resource	Time	Why
The Tail at Scale (Dean and Barroso, Google)	30 min	The classic short paper on why tail latency dominates at scale, and how hedged requests help.
napkin-math (Simon Eskildsen)	20 min	Measured, modern numbers for memory, disks, networks and cloud storage, with practice problems.
Latency numbers every programmer should know	5 min	The original list popularised by Jeff Dean. Some figures are dated; the ratios still teach.
Little's law (Wikipedia)	10 min	The law, its conditions, and worked examples.