System design · Module 19 of 29 · Level 4: Distributed and async

Real-time updates: from polling to WebSockets and pub/sub

HTTP is built for the client to ask and the server to answer. Chat messages, live scores, delivery tracking, collaborative cursors and streamed AI answers need the opposite: the server telling the client the moment something changes. That's two problems: how the update reaches the user's device (the protocol), and how it reaches the right server in the first place (the routing). This module solves both.

1. The idea

Waiting for a parcel, you can check the doorstep every ten minutes (polling), wait at the door until the courier comes and then go back to waiting (long polling), sign up for text alerts (server-sent events), or keep a phone line open to the depot for chatting both ways (a WebSocket). And the depot has to know which of its vans is driving down your street, which is the routing problem.

Option	How it works	Direction	Use when
Polling	The client asks every N seconds: "anything new?"	Client pulls	Updates aren't urgent; real-time isn't the core of the problem. The simplest thing that works.
Long polling	The server holds the request open until there's news (or 20–30 s pass), then the client immediately asks again	Server → client, via repeated requests	Infrequent updates where you want low delay with plain HTTP, e.g. waiting for a payment to complete
Server-sent events (SSE)	One long HTTP response that the server keeps writing events into; the browser's `EventSource` reconnects automatically and resumes with the `Last-Event-ID` header	Server → client only	Live feeds, dashboards, notifications, streaming AI responses. Writes go over normal requests.
WebSocket	An HTTP request upgraded into a persistent, two-way connection with small frames	Both ways	Frequent two-way messages: chat, multiplayer games, collaborative editing
WebRTC	Peer-to-peer media and data over UDP, with a signalling server to connect and STUN/TURN servers to get through NAT	Peer to peer	Voice and video calls, screen sharing, low-latency peer data

Rule of thumb: not latency-critical → polling; one-way push → SSE; frequent two-way → WebSockets; audio and video → WebRTC. Senior designs often start with the simplest one that meets the requirement.

2. How it works

Hop 2: getting the update to the right server

A user's WebSocket is connected to one of many servers. When someone sends them a message, the sender is connected to a different server. How does the message find its way? Press Play.

Two ways to route

Pub/sub (the usual default): connection servers are thin and interchangeable. Each subscribes to channels for the users connected to it (user:42, room:7) in Redis or Valkey pub/sub, NATS, or a Kafka-backed bus. Publishers just publish to the channel. Easy to scale and balance; the pub/sub layer must be sharded at scale, and plain pub/sub forgets messages for anyone not connected.
Consistent hashing to a stateful server: each user or document is owned by one server on a hash ring (Module 15); senders hash the ID to find it. Best when the server keeps heavy per-session state in memory (a collaborative document being edited). Harder: a coordinator (etcd, ZooKeeper) tracks membership, and moving ownership needs care.

Reconnects and missed messages

Mobile connections drop constantly. Design for it:

Presence and huge fan-out

Presence ("online", "typing…") is soft state: heartbeats refresh a key with a short TTL; expiry means offline. Batch and throttle presence updates; they can outnumber real messages.
Fan-out to millions (a live stream's comments, a celebrity's post) can't send each message individually through one channel. Use layers: messages go to regional or "broadcast" servers, each of which pushes to its own connected viewers, and aggregate or sample where exactness doesn't matter (Design D22).

3. Try it yourself

What does each option cost?

Compare the load and delay of each option for your numbers.

Pick the mechanism

4. Trade-offs

Stateless vs stateful

Polling keeps every server stateless. Persistent connections pin users to servers: deploys drop connections, load gets uneven, and you need least-connections balancing at layer 4 or 7 with long timeouts.

SSE vs WebSockets

SSE is plain HTTP: works through proxies, reconnects by itself, simple to scale. Over HTTP/1.1 browsers allow only 6 connections per domain, so serve it over HTTP/2. WebSockets are two-way and lighter per message, but you build reconnects and replay yourself.

Keep connection servers dumb

Run WebSockets in a dedicated gateway that only holds connections and forwards messages. Business logic lives in ordinary stateless services, which deploy freely without dropping anyone.

Real-time enough?

"Live" dashboards refreshed every 5 seconds by polling are often indistinguishable from push, at a fraction of the complexity. Ask what delay users would notice.

5. In real systems

WebSockets are standardised in RFC 6455; server-sent events in the HTML standard. Most AI chat products stream tokens to the browser with SSE (or a similar streamed HTTP response).
Managed options: AWS API Gateway WebSocket APIs and AppSync, Azure Web PubSub, Google Firebase, and hosted services such as Ably and Pusher handle connections and fan-out for you.
WebRTC needs a signalling channel (often a WebSocket) to exchange connection details, STUN to discover public addresses, and TURN relays when peers can't connect directly. Group calls usually route media through an SFU (selective forwarding unit) rather than full peer-to-peer.
WebTransport, built on HTTP/3 and QUIC, offers multiplexed streams and unreliable datagrams to browsers. It's newer and support varies by browser, so check before relying on it.
Scale numbers: one well-tuned server can hold hundreds of thousands of mostly idle WebSocket connections; memory per connection and the message rate are the limits, not the count alone.

The technologies up close

The default answer in an interviewName both hops. Hop 1: polling, unless real-time is core; SSE for one-way push; WebSockets only for frequent two-way messages. Hop 2: a stateless gateway tier subscribed to per-user channels in a sharded pub/sub, with messages stored first so reconnecting clients catch up by sequence number.

Polling and long polling

The default start · simplest

The client keeps asking. Plain polling asks every few seconds; long polling asks once and the server holds the request open until there's news.

How it works

Polling: GET /messages?since=917 every N seconds. Every server stays stateless, but delay is up to N, and most requests return nothing: 1 million clients polling every 10 s is 100,000 requests per second.
Long polling: the server answers when data arrives (or after 20–30 s), and the client immediately asks again. Lower delay with plain HTTP; align load balancer timeouts with the hold time.

Status today

Works everywhere, through every proxy and firewall. Still the right answer for many "live" screens refreshed every few seconds.

Use it when

Real-time isn't the core of the problem
Infrequent updates, or waiting for one async result (payment status)
You want to spend the interview time on something else

Skip it when

Chat, games and anything needing sub-second, frequent updates
Huge numbers of clients polling often (wasted load)

In the interview

I'll start with polling every few seconds so we can focus on the matching logic; if latency matters we can move to SSE. Senior interviews reward the simpler choice when it's enough.

🚩 Red flags

Jumping to WebSockets for updates nobody needs within a second.
Not doing the load math: clients ÷ interval = requests per second, mostly empty.

Server-sent events (SSE)

Situational · one-way push

One long HTTP response that the server keeps writing events into. The browser's EventSource reconnects automatically and resumes from the last event ID.

How it works

Plain HTTP, server to client only; the client sends its own actions as normal POSTs. Serve it over HTTP/2, because HTTP/1.1 browsers allow only six connections per domain. Some proxies buffer streamed responses, so disable buffering on that path.

Status today

Part of the HTML standard and supported by all modern browsers. Most AI chat products stream tokens this way (or with a similar streamed response).

Use it when

Live feeds, dashboards, notifications, auction prices
Streaming AI answers token by token

Skip it when

Frequent messages from the client too (chat, games)
Binary data

In the interview

Prices only flow to the client, so SSE: plain HTTP, automatic reconnect with the last event ID. Bids go over normal POSTs.

🚩 Red flags

Using WebSockets for one-way push without a reason; SSE is simpler to run.
Forgetting writes still need a normal endpoint, since SSE can't carry client messages.

WebSockets

Situational · frequent two-way traffic

An HTTP request upgraded into a persistent, two-way connection with small message frames.

How it works

Low overhead per message once open, in both directions.
Stateful: each user is pinned to one server. Deploys drop connections, load gets uneven, and you build heartbeats, reconnects and replay of missed messages yourself.
Run them in a dedicated connection-gateway tier that only holds sockets, behind an L4 (or long-timeout L7) balancer using least connections.

Status today

Standardised (RFC 6455) and universal. Managed options include AWS API Gateway WebSocket APIs, Azure Web PubSub, Ably and Pusher.

Use it when

Chat, multiplayer games, collaborative editing
High-frequency messages in both directions

Skip it when

One-way updates (SSE)
Updates every few seconds or slower (polling)

In the interview

Clients hold WebSockets to a thin gateway tier behind an L4 load balancer; business logic stays in stateless services.

🚩 Red flags

Choosing it too eagerly. The interviewer will ask about reconnects, deploys and uneven load.
Business logic inside the socket servers, so every deploy disconnects everyone.
No plan for missed messages: store first, then let reconnecting clients fetch by sequence number.

WebRTC

Specialised · audio and video only

Peer-to-peer media and data between browsers or apps, over UDP.

How it works

Peers exchange connection details through a signalling server (often a WebSocket).
STUN lets each peer discover its public address so they can connect through NAT; when that fails, a TURN server relays the traffic.
Group calls usually send media through an SFU (selective forwarding unit) instead of every peer connecting to every other.

Status today

Built into all major browsers and mobile platforms; it powers browser video calling. Running TURN and SFU servers at scale is real infrastructure work.

Use it when

Voice and video calls, screen sharing
Very low-latency peer data (some games, cursors in collaborative tools)

Skip it when

Chat, notifications or any server-to-client update
Anything that must be stored or moderated centrally

In the interview

Only for calls: Media goes peer-to-peer over WebRTC with TURN as fallback; signalling over our WebSocket gateway.

🚩 Red flags

Proposing WebRTC for chat messages or feeds.
Naming it without signalling, STUN and TURN: peers can't simply connect through NAT.

Pub/sub fan-out (hop 2)

The default for hop 2

How an update reaches the server holding the recipient's connection: connection servers subscribe to channels (per user or per room) and publishers post to them.

How it works

Gateways subscribe to user:42 when 42 connects; any service publishes to user:42 without knowing where 42 is.
Redis pub/sub is at-most-once: a subscriber that's offline misses the message. So store messages first, and use pub/sub only as the "wake up" signal.
At scale, shard the bus (sharded pub/sub in Redis 7+ Cluster) or use Kafka or NATS.

Status today

The common production pattern (Redis or Valkey pub/sub, NATS, Kafka-backed buses, or a managed service).

Use it when

Chat, notifications, broadcasting small messages to many clients
Whenever connection servers are interchangeable

Skip it when

Heavy per-session state that must live on one server (next block)

In the interview

Gateways subscribe to per-user channels in a sharded Redis pub/sub; messages are written to the database first, so reconnecting clients catch up by sequence number.

🚩 Red flags

Treating Redis pub/sub as durable. It forgets messages for anyone not connected.
Building your own subscriber registry in a key: extra hops and stale entries; let the pub/sub layer do it.

Stateful servers on a hash ring

Specialised · heavy per-session state

Each document or session is owned by one server, chosen by consistent hashing; every client for that document connects to its owner.

How it works

A coordinator (ZooKeeper or etcd) tracks which servers are alive; clients and senders hash the document ID to find the owner. Scaling means moving ownership carefully: announce the change, move clients gradually, send to both owners meanwhile.

Status today

Used where one server must hold heavy in-memory state, such as collaborative editors (Design D19).

Use it when

Collaborative documents, game rooms, anything with large per-session state

Skip it when

Chat and notifications: pub/sub is simpler

In the interview

Only for heavy state: Each document is owned by one server on a consistent-hash ring, so all edits for it meet in one place.

🚩 Red flags

Reaching for it in a chat design. It adds a coordinator and rebalancing for no gain.

6. Remember it

Memory hook: "Last mile, then routing"Pick the last-mile protocol from the direction and urgency of updates. Then solve routing: how an update finds the server holding the user's connection, and how a reconnecting user catches up.

How this hook works: what "Last mile, then routing" means

What the slogan means, phrase by phrase

"Last mile": Hop 1, server to device: polling, long polling, SSE, WebSockets or WebRTC, chosen by direction (one-way or two-way) and how fast it must arrive.
"then routing": Hop 2, source to the right server: pub/sub channels or consistent hashing, plus a durable store so missed messages can be replayed.

Why this picture fits: parcel delivery has the same split. The last mile is the van to your door; routing is the depot system that knows which van covers your street. Getting one right doesn't fix the other.

The story → the concept

Parcel delivery	Real-time system
Checking the doorstep every 10 minutes	Polling
Text alerts when it's out for delivery	Server-sent events
An open phone line to the depot	A WebSocket
The depot knowing which van covers your street	Pub/sub channels or consistent hashing
"Missed delivery, collect from depot"	Replaying missed messages by sequence number

Watch it on an example

A delivery app shows the courier on a map. Updates are one-way and every 5 seconds is fine: SSE, or even polling.
Routing: courier location updates go to a service that publishes to order:881; the gateway holding the customer's SSE connection is subscribed.
The customer's phone loses signal in a lift; EventSource reconnects with the last event ID and receives the latest position.
Chat with the courier needs two-way: the same gateway upgrades that screen to a WebSocket.

The trap it saves you from

Choosing WebSockets, drawing one server, and forgetting that with many servers the sender and receiver are almost never on the same one.

It still works when…

The "client" is another service: webhooks (Module 07) are the last mile between companies, and the routing problem becomes "which customer endpoint, with what retries".

7. Common mistakes

Relying on pub/sub for deliveryPlain pub/sub drops messages for disconnected clients. Store messages durably first, and replay on reconnect.

WebSockets by reflexFor one-way updates, SSE or polling is simpler to build and run. Choose WebSockets for genuinely two-way traffic.

Business logic in connection serversEvery deploy then drops every connection. Keep gateways thin and rarely deployed.

Reconnect stormsA restart makes all clients reconnect at once. Use exponential backoff with random jitter, and drain servers gradually.

Load balancer timeoutsDefault idle timeouts (often 60 seconds) silently cut long-lived connections. Raise them and send heartbeats.

8. Design exercise (30 minutes)

Choose the last mile for: a stock ticker web page, a group chat, a "your report is ready" notice, a multiplayer drawing game, a video call. One line each.
Draw hop 2 for a group chat with 10M connected users across 200 gateway servers: what each gateway subscribes to, and what happens when user A messages a group of 50.
Design reconnect: what the client stores, what it sends on reconnect, and where the missed messages come from.
Estimate with the calculator: 2M users, one update a minute. Compare polling every 10 s with SSE.
Write it down: the rule of thumb for the five options, and the two routing approaches.

9. Self-check quiz

Streaming an AI model's answer token by token to a browser. Best fit?

One-way, server to client, over plain HTTP. The question itself was a normal POST.

User A (on server 1) messages user B (on server 7). How does it reach B?

Server 7 subscribed to B's channel when B connected; server 1 publishes to it.

What lets a reconnecting client get messages it missed?

The client reports its last sequence number; the server replays newer messages from storage.

A dashboard updated once a minute, viewed by 50 people. Simplest good option?

Tiny load, no persistent connections, nothing new to operate.

An auction page shows live prices; bids are rare and go through a button. Which delivery fits best?

Prices only flow server → client, so SSE is enough and simpler to run; bids are ordinary requests.

Why store a chat message before publishing it to Redis pub/sub?

Redis pub/sub is at-most-once. Storing first means a reconnecting client can fetch what it missed by sequence number.

10. Go deeper

Resource	Time	Why
Using server-sent events (MDN)	20 min	EventSource, the event format, reconnects and connection limits.
Real-time updates (Hello Interview)	30 min	Both hops in interview form, with chat, live comments and collaborative editing.
WebRTC API (MDN)	30 min	Signalling, ICE, STUN and TURN explained with diagrams.
Redis pub/sub (Redis docs)	15 min	Channels, patterns, sharded pub/sub, and its at-most-once delivery.