System design · Module 08 of 29 · Level 2: Building blocks

Load balancing and scaling out

When one server isn't enough, you can buy a bigger one or run many. Running many is how almost every large system grows, and it needs two things: servers that don't care which requests they get, and something in front that spreads the work wisely and routes around failures. This module covers both, plus the gateways and autoscalers that sit around them.

New to this?Start with B6 · From one server to millions in Level 0; this module assumes it.

1. The idea

A supermarket with one till and a long queue can hire a faster cashier (scale up) or open more tills (scale out). More tills only help if any cashier can serve any customer, and if someone at the entrance points each shopper to a till that's actually open and not already swamped. That person is the load balancer.

Vertical scaling (scale up): a bigger machine. No code changes and often the right first step (Module 02 showed how big machines are), but there's a ceiling, upgrades can mean downtime, and one machine is one failure.
Horizontal scaling (scale out): more machines, each running the same code. No ceiling, survives failures, can grow and shrink with demand. It needs the servers to be stateless.
Stateless means a server keeps nothing between requests that the next request needs. Sessions, uploads in progress and caches of user data live in shared stores (a database, Redis), or in a signed token the client carries. Then any server can handle any request, and a server can be added, removed or killed at any moment.
A load balancer receives every request and forwards it to one healthy server. It's also where TLS is usually terminated, and where a server is taken out of rotation for a deploy.

L4 or L7?

	Layer 4 (transport)	Layer 7 (application)
Looks at	IP addresses and ports only	The full HTTP request: path, host, headers, cookies
Balances	Connections	Individual requests (many per connection)
Can do	Very high throughput, very low latency, any TCP/UDP protocol, long-lived connections (WebSockets, game servers, databases)	Route `/api` and `/images` to different services, terminate TLS, retries, header-based canary routing, sticky sessions by cookie
Examples	AWS Network Load Balancer, Google's passthrough network LB, HAProxy or Envoy in TCP mode	AWS Application Load Balancer, Google's application LB, NGINX, Envoy, HAProxy in HTTP mode

2. How it works

Choosing a server

Round robin: next server in turn. Simple and fair when every request costs about the same.
Weighted: bigger servers get a bigger share; also used to send 5% to a new version (a canary).
Least connections / least outstanding requests: pick the server with the fewest requests in progress. Adapts when some requests are slow or a server is struggling.
Power of two choices: pick two servers at random and send to the less busy one. Nearly as good as checking every server, much cheaper, and it avoids every balancer stampeding to the same "least busy" server at once.
Hash-based (IP hash, consistent hashing): the same key (user ID, session, cache key) always goes to the same server. Needed when servers hold state for a key, such as a cache shard or a chat connection (Module 15).

Press Play to see why round robin breaks down when request costs vary.

Health checks and draining

Active health checks: the balancer calls /healthz every few seconds and removes a server after a few failures. Passive checks watch real traffic and eject servers returning errors. A good health check tests that the server can do useful work (it can reach its database), but not so deeply that one shared dependency failing marks every server unhealthy at once.

For deploys, connection draining (AWS calls it deregistration delay, 300 seconds by default) stops new requests to a server while letting in-flight ones finish, so a rolling deploy drops nothing.

API gateways

Once a product is split into several services, every one of them needs the same chores done for each request: check who the caller is, stop callers who send too much, log the call, and send it to the right place. An API gateway is a single L7 proxy at the edge that does those chores once, for every service behind it. Clients see one address and one API; the services see only requests that have already passed the checks.

Typical gateway jobs, beyond the ones in the figure: request validation (reject a malformed body before it costs a service anything), protocol translation (public REST outside, gRPC inside), response caching, versioning (/v1 and /v2 to different services) and aggregation (one call from a phone fans out to three services and comes back as one response). Managed options are AWS API Gateway, Google Apigee and Azure API Management; self-run ones include Kong and Envoy-based gateways.

Why bother? Compare a mobile app calling each service directly with the same app calling through a gateway:

The costs are real: one more hop (usually a millisecond or two), one more system that must never go down (run several copies behind an L4 balancer, across zones), and a tempting place to put business logic, which should stay in the services. A gateway that knows what an order is has become a bottleneck that every team must change.

Load balancer, reverse proxy, gateway: what's the difference?

All three sit between clients and servers and forward requests, and real products blur the lines (NGINX and Envoy can be all three). The difference is the main job:

	Main job	Knows about	Example question it answers
Load balancer	Spread traffic over copies of the same service and route around failures	Server health and load	Which of these 12 identical servers takes this request?
Reverse proxy	Stand in front of servers so clients never reach them directly: TLS, compression, caching, hiding the servers	HTTP	Can I answer this from cache, or must a server see it?
API gateway	Be the one front door to many different services, with the cross-cutting rules applied	Your API: routes, callers, quotas	Is this caller allowed, under quota, and which service owns `/orders`?

A common setup uses all three in a row: an L4 load balancer spreads connections over several gateway copies; the gateway authenticates and routes; each service has its own internal load balancing (often in a service mesh) across its copies.

A reverse proxy works for the servers. Its mirror image, the forward proxy, works for the clients. Same box in the middle, opposite loyalties:

The rest of the front door

Service discovery: how callers find healthy instances of internal services as they come and go: DNS names in Kubernetes, a registry (Consul), or a service mesh sidecar that load-balances on the client side.
Autoscaling: add or remove servers based on a metric. Kubernetes' Horizontal Pod Autoscaler checks every 15 seconds by default; cloud autoscaling groups work similarly. New servers take time (seconds for containers, minutes for VMs), so scale on a leading signal, keep headroom, and pre-scale for known events.

3. Try it yourself

Five servers, four worker slots each. Requests vary in cost: most are quick, a few are slow. Pick an algorithm and a load level, optionally make one server slow (a noisy neighbour, a bad disk), and run 4,000 requests.

Simulation: random arrivals; 90% of requests take 10 ms and 10% take 100 ms; each server runs 4 requests at once and queues the rest. "Least connections" sees every server's live count; "power of two" samples two at random. Results are from one seeded run, so they're repeatable.

4. Trade-offs

Sticky sessions

Pinning a user to one server (by cookie) lets that server keep their session in memory. It's convenient, but load gets uneven, and when that server dies the user's session goes with it. Prefer shared session storage; use stickiness only as an optimisation.

The balancer is a single point too

Managed cloud balancers are redundant across zones for you. Self-run ones need at least two, with a floating IP, DNS with several addresses, or anycast in front.

Smart vs simple algorithms

Least-connections and power-of-two adapt to slow servers and uneven requests; round robin is predictable and good enough when requests are uniform. Hashing gives locality but rebalances badly without consistent hashing.

Autoscaling isn't instant

A spike that doubles traffic in 30 seconds arrives before new servers are ready. Keep headroom, shed low-priority load (Module 24), and scale ahead of scheduled events.

5. In real systems

AWS Application Load Balancer routes by round robin by default, and also offers "least outstanding requests" and "weighted random"; the Network Load Balancer works at layer 4 for TCP, UDP and TLS passthrough. Google Cloud and Azure offer the same two families.
Envoy, the proxy inside most service meshes and many gateways, uses power of two choices for its least-request balancer, and also supports ring-hash and Maglev consistent hashing.
Global load balancing picks a region first (by DNS or anycast, using geography, latency and health), then a server within it. CDNs and the big clouds' global load balancers do this at their edge.
Kubernetes gives every service a stable name and spreads traffic across its pods; an Ingress or Gateway API controller (often NGINX- or Envoy-based) is the L7 front door; the HPA scales pods, and a cluster autoscaler adds nodes.
Stateful connection servers (WebSockets for chat, game servers) are balanced at L4 or by consistent hashing, with least-connections to spread long-lived connections (Module 19).

The technologies up close

The default answer in an interviewAn L7 load balancer in front of stateless app servers, autoscaled on CPU. Say it in one sentence and spend your time elsewhere, unless the design has long-lived connections, stateful servers or extreme traffic. That's when the blocks below matter.

L7 (application) load balancer

The default · draw it

A proxy that ends the client's connection, reads each HTTP request, and forwards it to a healthy server chosen per request.

How it works

Terminates TLS, so it can read the path, host, headers and cookies.
Routes each request (not each connection): /api/* to the API, /images/* to the image service, a header to the canary.
Runs health checks, retries idempotent requests on another server, and drains servers during deploys.

Status today

Standard everywhere: AWS Application Load Balancer, Google Cloud's application load balancer, Azure Application Gateway; NGINX, Envoy and HAProxy when you run your own. Managed ones are redundant across zones for you.

Use it when

Any HTTP or gRPC API
Routing by path or host to different services
Canary releases by header or weight

Skip it when

Non-HTTP protocols (raw TCP, UDP game traffic, database connections)
Millions of long-lived connections where per-request work adds nothing

In the interview

Clients hit an L7 load balancer that terminates TLS and spreads requests across stateless app servers, autoscaled on CPU. One sentence, then move on.

🚩 Red flags

Not saying the servers are stateless. A load balancer only helps if any server can handle any request; otherwise you need stickiness, and the interviewer will ask what happens when that server dies.
Sticky sessions as the way to keep user state. Use shared session storage or tokens; stickiness is only an optimisation.
Spending minutes on algorithms for a stateless HTTP service. Round robin or least-requests is fine; the interesting parts are elsewhere.

L4 (transport) load balancer

Situational · long-lived connections

Forwards whole TCP or UDP connections by IP address and port, without reading what's inside them.

How it works

Picks a backend when a connection opens; every byte of that connection then goes to the same server.
No TLS termination needed (it can pass encrypted traffic straight through), so it's very fast and handles any protocol.
At huge scale, L4 balancers often sit in front of a fleet of L7 proxies.

Status today

Managed everywhere: AWS Network Load Balancer (TCP, UDP, TLS), Google Cloud's passthrough network load balancer, Azure Load Balancer; HAProxy or Envoy in TCP mode. Google's Maglev is a well-known software L4 balancer.

Use it when

WebSocket servers and other long-lived connections (use least connections)
Non-HTTP protocols: game servers over UDP, database or MQTT traffic
TLS passthrough, when the backend must see the client's certificate

Skip it when

Normal REST APIs: you lose path routing, retries and per-request balancing
HTTP/2 or gRPC traffic, where one connection carries many requests (one busy connection lands on one server)

In the interview

Only where connections are long-lived: Chat connections go through an L4 load balancer using least connections, because each socket stays open for hours.

🚩 Red flags

Expecting an L4 balancer to route by URL path or cookie. It never reads the request.
Assuming it spreads requests evenly. It spreads connections; a few heavy connections can overload one server.
Choosing L4 for an ordinary API without a reason. L7 is the expected default.

Client-side load balancing

Situational · internal services

No balancer in the middle: the caller gets the list of healthy servers and picks one itself.

How it works

The client reads the server list from a registry or DNS (Kubernetes endpoints, Consul, a service mesh's control plane) and keeps it up to date.
It chooses per request (round robin, least requests) and retries elsewhere on failure.
Smart database and cache clients do the same: a Redis Cluster client caches which node owns which keys; Cassandra drivers send each query straight to a replica.

Status today

Common inside companies: gRPC's built-in balancing, service meshes (an Envoy sidecar beside each service balances on its behalf), and most distributed-database drivers. Plain DNS round robin is the simplest form, but caching makes it slow to react.

Use it when

High-volume internal service-to-service calls (saves a network hop)
Long-lived HTTP/2 or gRPC connections that an L4 balancer would pin to one server
Clients of distributed databases and caches

Skip it when

Browsers and mobile apps you don't control
Small systems: a managed load balancer is simpler to run

In the interview

Mention it only for internal traffic: Services call each other through gRPC with client-side balancing from the service registry, so there's no extra hop.

🚩 Red flags

Suggesting it for public clients. You can't push server lists to every phone, or trust them to balance fairly.
Ignoring stale lists. If clients cache a dead server (or a DNS answer with a long TTL), failover is slow; say how lists are refreshed.

API gateway

Situational · many services behind one front door

An L7 proxy at the edge that adds API features: authentication, rate limits, request validation, and routing to many backend services.

How it works

Checks the token once at the edge and passes the verified identity on.
Applies rate limits and quotas per user or API key (returning 429).
Routes /orders, /payments… to different services, and can transform or version requests.

Status today

Managed: AWS API Gateway, Google Apigee and API Gateway, Azure API Management. Self-run: Kong and Envoy-based gateways. Modern L7 load balancers and Kubernetes Gateway API controllers cover part of the same ground.

Use it when

A public API, or many microservices behind one domain
When auth and rate limiting must be enforced consistently in one place

Skip it when

One or two services: the load balancer plus middleware in the service is enough
Putting business rules in it: it should stay thin

In the interview

Draw it as one box at the edge when there are several services: An API gateway handles auth and rate limits, then routes to the services.

🚩 Red flags

Stacking a gateway, a load balancer and a service mesh in front of a single service.
Business logic in the gateway. It becomes a bottleneck every team must change.
Forgetting it's on every request's path: it must be replicated, and its latency counts.

Hardware load balancers

Legacy · rarely the answer

Dedicated appliances (such as F5 BIG-IP) that balance traffic in specialised hardware.

How it works

Physical boxes in a data centre, usually deployed in pairs, doing L4 and L7 balancing, TLS offload and firewall features at very high throughput.

Status today

Still common in on-premises data centres, banks and telecoms. Cloud designs use managed or software balancers, which scale out instead of up.

Use it when

On-premises designs where the company already runs them
The interviewer explicitly asks about extreme throughput in your own data centre

Skip it when

Any cloud-based design (which is nearly every interview)

In the interview

At most a one-liner if asked about raw throughput: On-prem you might use hardware balancers; in the cloud we'd use the managed L4/L7 balancers, which scale horizontally.

🚩 Red flags

Proposing hardware in a cloud design. It suggests outdated assumptions, one of the failure signs interviewers look for.

Scalability laws: Amdahl and the Universal Scalability Law

The default way to reason about "just add servers"

Adding servers never gives a perfectly straight line. Two laws say why, and how bad the curve gets: Amdahl's law (the part that can't run in parallel caps the speed-up) and the Universal Scalability Law (coordination between nodes can make throughput go down after a point).

How it works

Amdahl's law (1967): if a fraction p of the work can run in parallel, N workers give a speed-up of at most 1 ÷ ((1 − p) + p ÷ N). With p = 95%, 10 workers give 6.9×, 100 give 16.8×, and no number of workers ever gives more than 20× (1 ÷ 5%). The serial 5% is a single lock, a single leader, a single database row everyone updates.
Universal Scalability Law (Neil Gunther): capacity C(N) = N ÷ (1 + α(N − 1) + βN(N − 1)). α (contention) is time spent waiting in a queue for a shared resource; it alone gives Amdahl's flattening curve. β (coherence, or crosstalk) is time nodes spend keeping each other in sync: every node talking to every other node, cache invalidations, distributed locks. Because it grows with N², the curve has a peak and then falls.
A worked example (α = 5%, β = 0.05%, 1,000 requests per second per node): 8 nodes give 5,800, 32 give 10,500, the peak is about 44 nodes at 10,700, and 128 nodes give only 8,300. The peak sits at N = √((1 − α) ÷ β).
Fit it from measurements: load test at 1, 2, 4, 8 and 16 nodes, fit α and β, and the curve tells you where adding nodes stops paying. A near-zero α and β is what a shared-nothing, stateless tier looks like (Module 01).

Nodes	Ideal	Contention only (α = 5%)	Contention + coherence (β = 0.05%)
1	1,000	1,000	1,000
8	8,000	5,900	5,800
32	32,000	12,500	10,500
64	64,000	15,400	10,400
128	128,000	17,400	8,300

Status today

Both are textbook results, not products. Amdahl's law dates from Gene Amdahl's 1967 paper; Gunther published the Universal Scalability Law in the 1990s and it is used in capacity planning for databases and clusters (Percona, for example, has written up USL fits for MySQL).

Use it when

Someone proposes scaling out a tier that shares a lock, a leader or a hot row
Explaining why a cluster got slower after you added nodes
Capacity planning from a handful of load tests

Skip it when

Stateless tiers behind a load balancer, where scaling really is close to linear: just say so

In the interview

The web tier is stateless, so it scales almost linearly. The inventory counter is one row everyone updates; that's Amdahl's serial fraction, so I'd shard it (Module 15) instead of adding app servers.

🚩 Red flags

Assuming 10× servers means 10× throughput without naming what they share.
Adding nodes to a gossiping or all-to-all cluster past its peak: the β term makes it slower.
Quoting the formula without saying where α and β come from in your design.

6. Remember it

Memory hook: "Any till, open till, shortest queue"Stateless servers so any till can serve anyone; health checks so only open tills get customers; a balancing rule that prefers the shortest queue.

How this hook works: what "Any till, open till, shortest queue" means

What the slogan means, phrase by phrase

"Any till": Servers are stateless and identical, so the balancer may send any request anywhere. This is what makes scaling out possible at all.
"open till": Health checks, ejection of failing servers, and draining before deploys: traffic only goes to servers that can serve it.
"shortest queue": Prefer the least-loaded server (least connections, power of two choices) when request costs vary or a server is slow.

Why this picture fits: everyone has picked a supermarket queue. You don't pick tills in strict rotation; you avoid the closed one and the one behind the customer with three trolleys.

The story → the concept

At the supermarket	In the system
Tills that all sell everything	Stateless, identical app servers
The person directing shoppers	The load balancer
A till's "closed" sign	A failed health check, or draining before a deploy
A customer with three trolleys	A slow, expensive request
Opening more tills at 5 pm	Autoscaling
Your regular cashier who knows your order	A sticky session (convenient, fragile)

Watch it on an example

A checkout service keeps each cart in server memory: not "any till". A deploy restarts servers and users lose their carts.
Move carts to Redis: now any server can serve any user, and deploys roll freely.
One server's disk degrades and it slows to 3× latency. Round robin keeps feeding it; p99 jumps.
Switch to least outstanding requests ("shortest queue"): the slow server gets fewer requests automatically, and passive health checks eject it when errors start.

The trap it saves you from

Adding servers to a service that keeps state in memory, and wondering why users get logged out and load is uneven.

It still works when…

You balance work in a queue: workers pulling jobs is "shortest queue" by design, because a busy worker simply doesn't ask for more (Module 18).

7. Common mistakes

State in server memorySessions, carts or upload progress in one server's memory break on every deploy, crash and scale-in. Keep state in shared stores.

Health checks that are too deepIf every server's health check calls the database, a database blip marks all servers unhealthy and the balancer has nowhere to send traffic. Check the server itself; handle dependency failures in the request path.

Round robin with uneven requestsWhen some requests cost 100× others, rotation piles work onto unlucky servers. Use least outstanding requests or power of two choices.

Autoscaling as the only plan for spikesNew capacity arrives minutes late. Combine headroom, queues and load shedding.

L7 balancing for long-lived connectionsWebSockets stay on the server they opened on. Balance them by connection count, and plan for reconnect storms when a server restarts.

8. Design exercise (25 minutes)

Make it stateless. A photo-editing web app keeps uploaded images and edit history in server memory. List what must move where so you can run 20 servers behind a balancer.
Pick the balancer. For: a REST API; a chat service using WebSockets; a Redis-like cache cluster; a game server using UDP. L4 or L7, and which algorithm?
Run the simulator at 85% load with the slow server on. Record p99 for each algorithm. Why does power of two choices come so close to least connections?
Plan a launch. Traffic will be 8× normal for two hours tomorrow at 9:00. Write down what you'd pre-scale, what you'd let autoscale, and what you'd shed if it's not enough.
Write it down: L4 vs L7 in one line each, and the four algorithms.

9. Self-check quiz

Why must app servers be stateless to scale out easily?

With no per-user state on the server, the balancer can route anywhere and servers can come and go freely.

Most requests take 5 ms, but some take 2 seconds. Which algorithm spreads load best?

It sends new work away from servers busy with slow requests. Round robin ignores how busy each server is.

Which kind of balancer suits millions of long-lived TCP connections to game servers?

L4 forwards connections without parsing them: fast, protocol-agnostic and suited to long-lived connections.

What does connection draining do during a deploy?

New requests go elsewhere while existing ones complete, so rolling deploys drop nothing.

Your chat servers hold WebSocket connections open for hours. What's the usual choice?

Long-lived sockets are balanced per connection. Least connections sends each new socket to the emptiest server.

When should a hardware load balancer appear in your design?

Cloud designs use managed or software balancers that scale out. Hardware appliances suggest outdated assumptions unless the interviewer sets up an on-premises scenario.

95% of a job can run in parallel. What's the most speed-up any number of workers can give?

Amdahl's law: the serial 5% always takes the same time, so the best possible speed-up is 1 ÷ 0.05 = 20×.

A cluster got slower after you doubled its size. Which term of the Universal Scalability Law explains that?

Contention only flattens the curve; the coherence term grows with N² and makes throughput fall after a peak.

10. Go deeper

Resource	Time	Why
Supported load balancers (Envoy docs)	20 min	Round robin, least request with power of two choices, ring hash and Maglev, explained by the proxy that implements them.
Load balancing (Sam Rose)	20 min	A beautiful interactive essay that animates each algorithm.
Load Balancing at the Frontend and in the Datacenter (Google SRE book)	1 h	DNS and anycast at global scale, then subsetting and weighted balancing inside a data centre.
Horizontal Pod Autoscaling (Kubernetes docs)	20 min	How autoscaling decides, and the knobs that stop it flapping.