Design an LLM chat platform

A chat assistant looks like a simple request and response, but each reply is a long, expensive computation on scarce GPUs, streamed a few words at a time for ten or twenty seconds. The design problems are unusual: GPU memory, not CPU, limits how many people you can serve; idle GPUs burn money; and keeping the first word fast matters more than total speed. This page has material for a full 60–90 minute interview.

1. The prompt

"Design a chat assistant like ChatGPT: people type a message, the model's reply streams back word by word, and their conversations are saved."

The surrounding product (accounts, conversation lists) is ordinary. The heart is the inference tier: how requests share GPUs, how much memory each conversation holds while it generates, how to keep the first token fast under load, and how to fail gracefully when GPUs run out. Say early that GPU cost dominates everything; it frames every trade-off that follows.

2. Frame the problem

Ask these in the first five minutes. Guess each answer and its effect before opening it.

3. What it must do

1. A user can send a message and watch the reply stream in, and stop it at any time.
2. Conversations are saved: the user can list them, reopen one and continue it.
3. The user can regenerate the last reply or switch between available models.

Not now (with a reason):

• Tools, browsing and uploads: the agent loop and retrieval (Module 29), layered on top later.
• Voice and images: other model types, similar serving ideas.
• Fine-tuning per customer: a training pipeline, not a chat feature.
• Sharing conversations: an easy read-only link later.

4. How well it must do it

5. Numbers that change the design

The per-replica throughput is an illustrative planning number. Real figures depend on the model, GPUs, quantization and prompt lengths, so say you'd benchmark it. The shape of the conclusion doesn't change: GPUs dominate, and memory per conversation limits batching.

6. The nouns

• User (with tier).
• Conversation.
• Message: user or assistant text, with token counts and model.
• Generation: one in-flight reply, from queued to streaming to done or cancelled.
• Model pool: the replicas of one model.
• Usage record: tokens used, for limits and billing.

conversations  conversation_id, user_id, title, model, updated_at          (by user_id, updated_at desc)
messages       conversation_id, message_id (time-ordered), role, content, tokens_in, tokens_out, model, parent_id
generations    generation_id, conversation_id, status, model_pool, replica, queued_at, first_token_at, finished_at
usage          user_id, day, tokens_in, tokens_out                          (token limits per tier)

parent_id lets a regenerated reply become a sibling of the old one, so the user can flip between versions.

7. The contract

REST for conversations; the reply streams as Server-Sent Events on the response to the POST, because it's one-way and long. Many model APIs stream this way. Internally, the chat service calls the inference tier over gRPC with a streaming response. The user comes from the session token; the tier and limits come from the account, never from the client.

POST /v1/conversations                        → 201 {conversation_id}
POST /v1/conversations/{id}/messages          {content, model, client_message_id}
     ← event: message_start  data: {message_id, generation_id, model}
     ← event: delta          data: {text: "Sure"}           (many)
     ← event: message_end    data: {stop_reason: "end", usage: {in: 1840, out: 412}}
POST /v1/generations/{generation_id}/cancel   → stop generating, free the GPU slot
GET  /v1/generations/{generation_id}/stream   Last-Event-ID: 57    → resume a dropped stream
GET  /v1/conversations?cursor=…               → list, newest first
GET  /v1/conversations/{id}/messages?cursor=… → history

// internal (gRPC)
Generate(stream) {model, prompt_tokens, max_tokens, priority, cache_key} → stream of tokens

Data flow: (1) check limits → (2) load the conversation and build the prompt (system prompt + history that fits the context) → (3) route to a model pool and queue → (4) a replica runs prefill (reads the whole prompt), then decodes one token at a time → (5) tokens stream back through the chat service to the browser → (6) the finished message and usage are saved.

8. First working version

Start simple: a chat service, a database, and a few GPU servers each running a model server that handles one request at a time. Get the three requests working.

8.1 Sending a message

The chat service saves the user's message, loads earlier messages, and builds the prompt. If the history is longer than the context window, it drops the oldest turns (we improve this in 9.2). It then calls a model server.

8.2 Streaming the reply

The model server produces tokens one by one; the chat service forwards each as an SSE delta event, and on the last token saves the full assistant message with its token counts. The browser renders text as it arrives.

8.3 Listing and reopening conversations

The sidebar lists conversations by (user_id, updated_at desc) with a cursor. Opening one pages through its messages. A short title is generated after the first exchange by a cheap model call, in the background.

9. Make it hold up

9.1 Sharing GPUs: continuous batching cost · first token fast

Generating a token for one conversation barely uses a GPU's compute: the time goes into reading the model's weights from memory. Reading them once and producing the next token for many conversations at the same time costs little more. Batching is therefore the biggest lever in the whole design.

Two phases share the GPU. Prefill reads the whole prompt at once (compute-heavy, it sets TTFT). Decode produces one token per step (memory-heavy, it sets tokens per second). Engines interleave them; large deployments sometimes run prefill and decode on separate GPU pools, so long prompts don't stall everyone's streaming. Try the three modes in the lab in section 13.

9.2 Memory is the limit: KV cache and context cost · first token fast

• Why memory: while a reply is generated, the model keeps keys and values for every token so far (the KV cache), about 320 KiB per token for a 70B model with grouped-query attention. A batch can only hold as many conversations as fit in the memory left after the weights.
• Paged KV cache: allocating each conversation's maximum length up front wastes most of the memory. vLLM's PagedAttention stores the cache in small fixed-size blocks, like virtual memory pages, allocated as the reply grows. Less waste means bigger batches.
• Prefix caching: every message in a conversation starts with the same system prompt and history. Keeping the KV blocks of that shared prefix lets the next message skip most of its prefill. vLLM calls this automatic prefix caching. It only works if the conversation comes back to a replica that still has the blocks (see 9.3).
• Managing long conversations: when history exceeds a budget (say 16K tokens), summarise the oldest turns into a short note (a cheap model call in the background) and keep the recent turns verbatim. Cap max_tokens per reply by tier.
• Smaller numbers: 8-bit weights halve memory against 16-bit with small quality loss; a quantized KV cache stretches further. Measure quality before switching.

9.3 Routing, priorities and admission control degrade, don't fail · first token fast

Scaling GPUs is slow: a new replica needs a GPU machine, then minutes to load 70 GB of weights. Autoscale on queue depth and forecast (the daily peak is predictable), keep warm headroom, and use the small model as the pressure valve for sudden spikes.

9.4 Streams that survive bad connections degrade, don't fail

• Decouple generation from the connection: the inference tier writes tokens into a short-lived per-generation buffer (for example a Redis stream with a TTL of a few minutes), and the chat service relays from it. A dropped browser reconnects to /v1/generations/{id}/stream with Last-Event-ID and continues where it stopped.
• Cancel really cancels: the Stop button, or no reader for, say, 60 s, sends a cancel to the replica, which removes the sequence from the batch and frees its KV memory at the next step.
• No double generations: client_message_id makes a retried POST attach to the existing generation instead of starting a second one, which would cost GPU time twice.
• Save once: the final message is written when the generation ends (or is cancelled, keeping what was produced), with its usage for limits and billing.

9.5 Storing conversations, and deleting them kept and private

• Shape: messages partitioned by conversation_id and sorted by time-ordered id (a wide-column store or a sharded relational database); conversation lists indexed by (user_id, updated_at).
• Writes are small: one user message and one assistant message per exchange. The database is not the bottleneck; GPUs are.
• Deletion: deleting a conversation marks it deleted at once (hidden everywhere), and a background job removes messages, cached summaries, generation buffers and search indexes, then confirms. Backups expire on a fixed schedule, so deleted data ages out of them too.
• No-retention tenants: skip storage entirely and keep content out of logs. Logs contain ids and token counts, never message text. Encrypt stored content with per-tenant keys where contracts require it.

9.6 Safety on a stream safe

• Input check: a fast classifier on the user's message before it reaches the big model (tens of milliseconds), plus per-user rate limits and abuse detection.
• Output check while streaming: run a classifier over the reply in chunks as it streams. If a chunk is flagged, stop the generation, replace the visible text with a notice and record why. A small buffer (a sentence or so) before display trades a little latency for not showing flagged text.
• Separation of duties: safety models and rules are versioned and deployed separately from the chat model, so they can be updated quickly.

9.7 The whole design after the deep dives

One message through every part: limits, prompt building with summaries, cache-aware routing, a continuously batched replica, a resumable token buffer and the saved conversation.

10. Interviewer follow-ups

11. When things break

12. Wrap-up and the clock

Trade-offs we made:

• Continuous batching and big batches (throughput) balanced against TTFT.
• Cache-aware routing against perfectly even load.
• Summarising old turns (cheaper, slightly lossy) instead of resending everything.
• Degrading free traffic instead of failing everyone.
• A short output buffer for safety, at the cost of a few hundred milliseconds.

What we'd watch:

• TTFT p95 and tokens per second per tier.
• Queue time.
• GPU utilisation and KV-memory use.
• Prefix-cache hit rate.
• Fallback rate.
• Cost per 1,000 replies.
• Safety block rate.

At 10×:

• Separate prefill and decode pools.
• More regions with GPU capacity.
• Aggressive routing of easy requests to small models.
• Speculative decoding.
• Capacity planned from forecasts weeks ahead.

Common mistakes:

• Treating inference like a stateless web call.
• Ignoring KV-cache memory.
• One request per GPU.
• Autoscaling GPUs as if they started in seconds.
• No plan for overload.
• Losing the reply when the connection drops.
• Generating for users who already left.
• Logging message content.

13. Try it, check yourself, read more

GPU batching lab. One model replica for 60 seconds. Requests arrive at random; reply lengths vary from a sentence to a page. Each row is one request: grey while it waits in the queue, coloured while it generates (blue for long replies), red if it was still unfinished when we stopped watching.

Model of one replica: one decode step every ~30 ms (slightly slower with bigger batches), up to 16 sequences in memory at once (8 for static batches). Illustrative, not a benchmark.