Designing AI systems

A product built on a large language model is still a distributed system: requests, queues, caches, databases, rate limits, failures. But one component behaves differently from anything before it: it's slow, expensive per call, priced and limited in tokens, non-deterministic, and can be steered by its own input. This module applies the whole course to AI features: the request path, cost and capacity, serving models yourself, retrieval, agents, and making it safe and measurable. For the AI concepts themselves, this site's AI engineering track goes deeper.

1. The idea

• Tokens are the unit of everything. Price, rate limits, context size and latency are all counted in tokens (roughly ¾ of an English word). Input and output tokens are priced separately; output is slower and usually costs more.
• Latency has two parts: time to first token (TTFT: reading the prompt) and output speed (tokens per second). A 600-token answer at 60 tokens a second takes 10 seconds, so stream it (SSE, Module 19).
• Calls fail and get throttled. Providers enforce limits in requests and tokens per minute and return overload errors under load. Treat a model API like any flaky dependency: timeouts, retries with backoff, fallbacks (Module 24).
• Outputs vary. The same input can give different answers. You test with evals over many examples, not one run.
• Inputs can attack. Text the model reads (a web page, an email, a document) can contain instructions: prompt injection. Design permissions assuming the model can be talked into anything.

2. How it works

One question to a document assistant

Press Play: a user asks an internal assistant a question about company policy.

Making it cheaper and faster

• Prompt caching: providers can reuse the processed form of a repeated prompt prefix (system instructions, tool definitions, a long document). Put stable content first and variable content last. Cached input is cheaper and faster to process; the exact discount and rules differ by provider, so check their docs.
• Model routing: send easy requests (classification, short extraction) to a small, cheap model and hard ones to a large one. A classifier or simple rules decide.
• Response caching: cache full answers to identical requests; "semantic caching" by embedding similarity is riskier, because similar questions can need different answers.
• Batch APIs: for work that can wait (nightly summarisation, evals), providers offer asynchronous batch processing at lower prices.
• Trim the context: retrieve fewer, better chunks; summarise long histories. Every token in the prompt is paid for on every call.

Serving models yourself

Self-hosting an open-weight model trades a per-token bill for a GPU bill and operational work. The dominant constraint is GPU memory: model weights (parameters × bytes per parameter: a 70-billion-parameter model at 16-bit precision is 140 GB) plus the KV cache, which grows with context length and the number of requests being served at once. Inference servers such as vLLM use PagedAttention to manage that cache efficiently and continuous batching to add new requests to a running batch, which raises GPU utilisation dramatically. Quantisation (8-bit or 4-bit weights) cuts memory at some cost in quality. Try the sizing calculator in section 3.

Agents: models that act

An agent is a loop: the model decides on a tool call, your code runs it, the result goes back to the model, repeat until done (AI track 07). As a system, that means: bound the loop (maximum steps, tokens, time and money), run long tasks as durable workflows so a crash doesn't restart them (Module 22), make tool calls idempotent, and give the agent only the permissions the task needs, with a human approval step for irreversible actions. The Model Context Protocol (MCP) standardises how agents connect to tools and data.

Security: the lethal trifecta

An AI system becomes dangerous when it combines three things: access to private data, exposure to untrusted content (which may carry injected instructions), and the ability to communicate externally (send email, call URLs, render links or images). With all three, an attacker's hidden instruction can make it leak the data. Remove at least one leg for any flow, and enforce permissions in your code, never in the prompt (AI track 18).

3. Try it yourself

Cost and capacity of an AI feature

The prices are example values, not any provider's current list: type in the real ones from your provider's pricing page.

How many GPUs to hold the model?

Rough sizing: weights = parameters × bytes per parameter; the KV cache and activations need extra memory that grows with context length and concurrent requests. Real deployments benchmark with their own traffic.

4. Trade-offs

5. In real systems

• Providers (Anthropic, OpenAI, Google and others) offer prompt caching, streaming, batch APIs and structured outputs; details and prices change often, so the AI track's model API module and each provider's docs are the place to check.
• Inference servers: vLLM, SGLang and NVIDIA's TensorRT-LLM / Triton serve open-weight models with continuous batching; managed platforms (cloud AI services, hosted inference providers) run them for you.
• GPU memory: NVIDIA's H100 has 80 GB (94 GB in the NVL version); newer data-centre GPUs carry more. Large models are split across several GPUs with tensor or pipeline parallelism.
• Vector search: pgvector inside PostgreSQL covers many products; dedicated vector databases and search engines with vector support (OpenSearch, Elasticsearch) scale further. Combine vector and keyword (hybrid) search, then rerank.
• Observability: OpenTelemetry's generative-AI conventions name fields such as the model and token counts; dedicated LLM tracing tools capture prompts, tool calls and costs per request, with care for sensitive data.

In an interview (design ChatGPT-like chat, an AI support agent, a RAG search), cover: streaming over SSE; an AI gateway with per-user token limits and model fallbacks; conversation storage; retrieval with permission filters; a queue for long agent tasks; cost per request estimated in tokens; evals and tracing; and prompt-injection boundaries. Design D23 works one through.

The technologies up close

6. Remember it

7. Common mistakes

8. Design exercise (40 minutes)

1. Design a document Q&A assistant for a company with 50,000 employees and 2 million documents with per-team permissions. Draw ingestion, retrieval, generation, and where permissions are enforced.
2. Estimate cost: 20,000 questions a day, 3,000 input and 500 output tokens each, 60% of input cacheable. Use the calculator with your provider's current prices.
3. Plan capacity: at peak, 50 questions a second. What token-per-minute limit do you need from the provider? What happens when you hit it?
4. Close the trifecta: the assistant can also send email summaries. Which leg would you remove or restrict, and how?
5. Write it down: the gateway's responsibilities, and the four words of the memory hook.

9. Self-check quiz

10. Go deeper

11. Under the hood (optional)

Not needed for most interviews; open it when you want to know how the products really do it. All the deep dives are listed on the Under the hood page.