System design · Level 0: Basics · Module B2 of 6

Inside one machine: CPU, memory, disk, network and how a server juggles requests

Every server is an ordinary computer: a processor that computes, memory that holds what it's working on, a disk that keeps things, and a network card that talks to other machines. They work at wildly different speeds, and most of a request's time is spent waiting for the slow ones. Knowing how a server handles thousands of waiting requests at once explains why one machine tops out, and what "scaling" actually has to fix.

1. The idea

A restaurant kitchen. The chef (a CPU core) works fast but can only do one thing at a time. Ingredients on the counter (memory) are in reach instantly; ingredients in the store room (disk) take a walk; anything from a supplier (another machine over the network) takes a delivery. A good chef doesn't stand still waiting for a delivery: they start the next order. That is the whole trick behind serving many requests at once.

Part	What it does	What to remember
CPU (cores)	Runs instructions. Each core runs one thing at a time, billions of steps a second.	A server with 16 cores can truly run 16 things at the same moment.
Memory (RAM)	Holds running programs and the data they're using.	Fast (about 100 nanoseconds) but lost when power goes off, and limited in size.
Disk (SSD or HDD)	Keeps data permanently.	An SSD read takes tens to hundreds of microseconds: about a thousand times slower than memory. Spinning disks are slower again.
Network card	Sends and receives data from other machines.	A round trip inside a data centre is about half a millisecond; across an ocean, 100 ms or more.
Process	A running program with its own private memory.	Processes can't see each other's memory; that isolation makes them safe but heavier.
Thread	One line of execution inside a process. Threads in a process share its memory.	Cheap to share data, easy to get wrong (two threads changing the same value).
Concurrency vs parallelism	Concurrency is juggling many tasks over time; parallelism is doing several at the same instant.	One chef juggling five orders is concurrent; four chefs are parallel.
Blocking vs non-blocking I/O	Blocking: the thread stops until the disk or network answers. Non-blocking: it starts the request and does other work meanwhile.	The difference decides how many requests one server can keep in flight.

2. How it works

The latency ladder

Each rung is roughly a hundred to a thousand times slower than the one above. The bars use a logarithmic scale; in real time the bottom rung is hundreds of millions of times longer than the top one. Switch to "human scale", where one nanosecond becomes one second, to feel the gaps.

Rounded figures from commonly cited measurements; your hardware will differ by small multiples, not by the orders of magnitude between rungs.

Serving many requests: threads vs an event loop

A request usually spends a little time computing and a lot of time waiting (for the database, a disk, another service). There are two classic ways to keep a server busy while requests wait:

Thread per request: each request gets its own thread, which simply blocks while it waits. Simple code, but every waiting request holds a thread and its memory.
Event loop: one thread per core starts the I/O, moves on to other requests, and comes back when the answer arrives. Thousands of waiting requests cost almost nothing, but one slow computation stalls everybody on that loop.

Modern runtimes blur the line: Go's goroutines and Java's virtual threads let you write simple blocking-style code while the runtime multiplexes thousands of them onto a few real threads.

3. Try it yourself

How many requests can one server handle?

Set the machine and the work each request does. The lab finds which limit is hit first: CPU, threads, or memory.

Model: CPU limit = cores × 1000 ÷ CPU ms; a pool of N blocking threads can finish N × 1000 ÷ (CPU + waiting) ms per second; each thread reserves about 1 MB of stack. Real servers lose some capacity to the operating system and garbage collection.

4. Trade-offs

Threads vs event loops

Threads keep code simple and use all cores naturally, but each waiting request costs memory and switching between thousands of threads costs CPU. Event loops handle huge numbers of waiting connections cheaply, but code must never block, and CPU-heavy work must move elsewhere.

Memory vs disk

Memory is fast and small; disk is slow and large. Most performance work is keeping the data you need next in memory: that's what caches, buffer pools and the operating system's page cache all do.

More cores vs more machines

A bigger machine helps until one shared resource (a lock, the disk, the network card) becomes the bottleneck, and it's still one machine that can fail. See B6.

Speed vs safety on disk

Writes first land in memory buffers. Forcing them to disk (fsync) on every write is safe but slow; batching them is fast but risks losing the last moments on a crash. Databases let you choose.

5. In real systems

Runtime	How it handles many requests
Node.js	One JavaScript thread per process on an event loop; a small thread pool for file and crypto work. Run one process per core.
Go	Goroutines: very cheap threads scheduled by the Go runtime onto a few OS threads. Blocking code is fine.
Java	Thread pools traditionally; virtual threads (since Java 21) make thread-per-request cheap again.
Python	The global interpreter lock limits one process to one core for Python code, so servers run several worker processes, or use asyncio for waiting-heavy work.
Nginx, Redis	Event loops: Nginx runs one per worker process; Redis runs its commands on one thread, which is why one slow command blocks everyone.

When a server is slow, check four resources for each of utilisation, saturation and errors (Brendan Gregg's USE method): CPU, memory, disk and network. The busiest one is your bottleneck; adding anything else won't help.

Say it in the interview

Two-sentence answers

"Why is a request slow if the CPU is idle?" Because it's waiting: on the database, a disk or another service. Most request time is I/O, not computation.
"Thread per request or event loop?" Threads are simpler and fine up to thousands of concurrent requests; event loops or lightweight threads handle tens of thousands of mostly-waiting connections, like WebSockets.
"What limits one server?" Whichever runs out first: CPU for heavy computation, threads or memory for many waiting requests, disk or network bandwidth for big data.

Red flags interviewers notice

Assuming disk is as fast as memory, or a network call as cheap as a function call.
"Add more threads" as the answer to a CPU-bound server.
Ignoring the order-of-magnitude gaps when estimating latency.

6. Remember it

Memory hook: "The chef never waits for the delivery"Computation is fast and waiting is slow, so a good server keeps working on other requests while some wait.

How this hook works: what "The chef never waits for the delivery" means

What the slogan means, phrase by phrase

"The chef": A CPU core: fast, but only one task at a time.
"never waits": Doesn't sit idle while I/O is in flight: an event loop or many lightweight threads keep it busy.
"the delivery": Disk and network: thousands to millions of times slower than computing.

Why this picture fits: it explains both why requests are slow (waiting) and how servers stay efficient (overlapping the waits).

The story → the concept

Kitchen	Machine
Chef	CPU core
Counter	Memory
Store room	Disk
Supplier delivery	Network call
Juggling orders	Concurrency (event loop or threads)
Hiring more chefs	More cores, or more machines

7. Common mistakes

Blocking the event loopA synchronous file read, a big JSON parse or a tight loop on a Node.js or Redis thread stops every other request it was serving.

Thread pools sized by guessworkToo small and requests queue while the CPU idles; too large and memory and context switches eat the machine. Size from CPU time and waiting time, as in the lab.

Forgetting memoryEach connection, thread and cached object costs memory. Running out doesn't slow things gracefully: the process is killed.

Treating I/O as freeTen database calls in a loop is ten round trips. Batch them, or the waiting dominates.

8. Design exercise (20 minutes)

Profile on paper (10 min). For an API that reads a user from the database (2 ms), calls a recommendation service (40 ms) and renders JSON (3 ms CPU), say what dominates and estimate requests per second on 8 cores with a 200-thread pool.
Choose a model (5 min). A chat server holds 50,000 mostly idle WebSocket connections. Threads or an event loop? Why?
Check with the lab (5 min). Enter your numbers and compare.

9. Self-check quiz

Where does most of a typical web request's time go?

Compute takes microseconds to milliseconds; network and disk waits are usually larger.

Roughly how much slower is an SSD read than a memory read?

Around 100 ns vs around 100 µs.

What happens when code on an event loop does a slow computation?

One thread serves them all; while it computes, nothing else runs.

One core switching between many requests as they wait is…

Concurrency is interleaving; parallelism needs several cores at once.

Requests need 20 ms of CPU each and almost no waiting. With 4 cores, what limits throughput at about 200 requests a second?

4 cores × 1000 ÷ 20 ms = 200 per second. Only more cores, or less CPU per request, helps.

10. Go deeper

Resource	Time	Why
Latency numbers every programmer should know (interactive)	10 min	The classic table, adjusted by year.
The USE method (Brendan Gregg)	15 min	A checklist for finding which resource is the bottleneck.
The Node.js event loop	20 min	How an event loop actually schedules work.
Concurrency is not parallelism (Rob Pike)	30 min	The distinction, explained by one of Go's designers.

11. Under the hood (optional)

What the operating system, language runtime and orchestrator really do with your requests. Not needed for most interviews; open it when you want to know how the products really do it. All the deep dives are listed on the Under the hood page.

Runtimes, the OS and containersConcurrency models, page cache and fsync (animated), a CPU-throttling lab, the Kubernetes reconcile loop (animated), an autoscaler lab, serverless cold starts, 9 runtimes comparedabout 35 min

Capacity estimates assume a server does work at a steady rate. Real servers are processes and threads scheduled by an operating system, inside containers with limits, run by an orchestrator, sometimes created on demand. Each layer has behaviour that shows up as latency spikes, lost writes or wasted machines.

Concurrency models

Model	How waiting is handled	Cost per in-flight request	Examples
Process per request	The OS switches processes	Megabytes; heavy	Classic CGI, Python servers with worker processes
Thread per request	A blocked thread sleeps; the OS runs another (a context switch costs microseconds)	A stack (often 1 MB reserved) plus kernel bookkeeping	Traditional Java and .NET servers
Event loop	One thread multiplexes many sockets with epoll/kqueue; callbacks or async/await resume work	Kilobytes	Node.js, NGINX, Netty, Python asyncio
M:N (green threads)	Cheap user-space threads scheduled onto a few OS threads; blocking calls park only the lightweight thread	Kilobytes	Go goroutines, Java virtual threads, Erlang processes, Tokio tasks

Readiness vs completion I/O: epoll and kqueue tell you a socket is ready, then you read; io_uring (Linux) and IOCP (Windows) let you submit operations and collect completions through shared queues, with fewer system calls.
The event-loop trap: one slow synchronous call (a large JSON parse, a regex, a sync file read) blocks every request on that loop. Node.js offloads file, DNS and crypto work to a small libuv thread pool (4 threads by default).
Garbage collection pauses add to tail latency: Java's G1 is the default; ZGC and Shenandoah keep pauses around a millisecond; Go's collector is concurrent with short pauses; Rust has no collector at all.

Memory, the page cache and durable writes

The page cache: Linux keeps recently read and written file pages in spare RAM. Reads of hot files never touch the disk, and write() returns once data is in memory, not on disk.
fsync forces the file's dirty pages and metadata to stable storage; fdatasync skips metadata not needed to read the data; O_DIRECT bypasses the cache (databases with their own buffer pools). Databases call fsync at commit, which is why commit latency follows disk flush latency.
Zero-copy: sendfile and splice move file bytes to a socket inside the kernel; Kafka and NGINX use this.
Out of memory: when memory runs out, the kernel's OOM killer kills a process; inside a container, exceeding the memory limit kills the container. Memory limits are hard; CPU limits are not (next).
Limits per process: open file descriptors (every socket is one) and the TCP accept backlog cap how many connections a server can hold or queue; raise them on busy servers.

Containers: namespaces, cgroups and CPU throttling

A container is an ordinary Linux process with namespaces (its own view of processes, network, mounts and hostname) and cgroups (limits on CPU, memory and I/O). Images are stacks of read-only layers merged by OverlayFS with a thin writable layer on top.

A CPU limit is enforced as a quota per period: a limit of 1 CPU means 100 ms of CPU time per 100 ms period, shared by all the container's threads. A request that runs on several threads at once can burn the quota early and then wait, doing nothing, until the next period, even on an idle machine:

Runtimes that size thread pools from the number of host CPUs make this worse: a Go or JVM process on a 64-core node with a 2-CPU limit starts 64 workers. Go 1.25 now sets GOMAXPROCS from the container's CPU limit automatically; modern JVMs read cgroup limits too. Many teams set CPU requests (for scheduling) but no CPU limit, and always set memory limits.

Kubernetes: the reconcile loop

Desired vs actual state: you store what you want (a Deployment with 3 replicas) through the API server into etcd; controllers watch for differences and act until reality matches. Nothing is a one-off command.
Scheduling: the scheduler filters nodes that fit the pod's requests, scores the rest (spreading, affinity), and binds the pod. Requests and limits set the QoS class: Guaranteed (requests = limits), Burstable, or BestEffort (none), which decides who is evicted first under pressure.
Probes: readiness decides whether a pod receives traffic; liveness restarts a stuck pod; startup probes protect slow starters. A liveness probe that checks a database turns a database blip into a restart storm.

Serverless: instances, isolation and cold starts

AWS Lambda runs functions in Firecracker microVMs; each execution environment handles one request at a time, so concurrency = instances. SnapStart and provisioned concurrency hide cold starts.
Cloud Run runs containers that can each serve many concurrent requests (80 by default), so fewer instances and fewer cold starts.
Cloudflare Workers run V8 isolates inside a shared process: starting one takes milliseconds, but only JavaScript/WebAssembly with tight CPU and memory limits.
Cold start costs come from creating the sandbox, starting the runtime and running your initialisation (loading libraries, opening connections); keep packages small and initialise lazily.

Runtime by runtime

Runtime or platform	Concurrency and memory	The detail worth knowing
Node.js	One JavaScript thread on the libuv event loop; a small thread pool; worker threads for CPU work	One slow synchronous function stalls the whole process
Go	Goroutines scheduled M:N onto OS threads (G-M-P); network poller on epoll; concurrent GC	Hundreds of thousands of goroutines are normal; GOMAXPROCS follows container CPU limits since 1.25
Java	Platform threads 1:1; virtual threads (Java 21) M:N; Netty event loops; G1 default, ZGC and Shenandoah for low pauses	Virtual threads make thread-per-request code scale like async
Python	The GIL limits CPU parallelism per process; the free-threaded build is officially supported (optional) since 3.14; asyncio for I/O	Servers run several worker processes to use every core
Rust (Tokio)	Async tasks on a work-stealing scheduler; no GC	Predictable latency
Erlang / Elixir (BEAM)	Millions of lightweight processes, preemptive schedulers, per-process GC, supervisors	Fault isolation by design
Containers	Namespaces, cgroups, OverlayFS layers	CPU limits throttle by time slice
Kubernetes	API server and etcd, scheduler, controllers, kubelets	HPA: desired = ceil(replicas × current ÷ target)
Lambda, Cloud Run, Workers	MicroVM per request stream; containers with concurrency; V8 isolates	Cold starts come from creating instances

Say it in the interview

Answers that show depth

"p99 latency jumped after moving to Kubernetes, but CPU looks low. Why?" Likely CFS throttling: a multi-threaded burst used the quota early in each 100 ms period. Check throttling metrics, right-size the thread pool to the limit, or remove the CPU limit and keep the request.
"Is the write durable when write() returns?" No: it's in the page cache. It's durable after fsync (or after the database commits, which fsyncs its log).
"Lambda or containers?" Lambda for spiky, event-driven work with no servers to manage; containers for steady traffic, long connections or when cold starts and per-request pricing hurt.

A container with a 1-CPU limit runs an 8-thread burst. Why can a request stall even though the host is idle?

8 threads consume 100 ms of quota in about 12.5 ms, then the container is throttled for the rest of the 100 ms period.

4 replicas average 90% CPU against a 60% target. How many replicas does the Horizontal Pod Autoscaler want?

ceil(4 × 90 ÷ 60) = 6.

When is data written by an application actually safe from a power loss?

write() and close() can leave data in the page cache.

Inside one machine: CPU, memory, disk, network and how a server juggles requests

1. The idea

2. How it works

The latency ladder

Serving many requests: threads vs an event loop

3. Try it yourself

How many requests can one server handle?

4. Trade-offs

Threads vs event loops

Memory vs disk

More cores vs more machines

Speed vs safety on disk

5. In real systems

Say it in the interview

6. Remember it

What the slogan means, phrase by phrase

The story → the concept

7. Common mistakes

8. Design exercise (20 minutes)

9. Self-check quiz

10. Go deeper

11. Under the hood (optional)

Concurrency models

Memory, the page cache and durable writes

Containers: namespaces, cgroups and CPU throttling

Kubernetes: the reconcile loop

Serverless: instances, isolation and cold starts

Runtime by runtime

Say it in the interview

Sources