AI engineering · Case study 02 · How AI works
One request, end to end
A customer types "Where is my order A-1042? It was due yesterday." into a support chat. Three seconds later they have the real answer. In between, their text is formatted, tokenized, pushed through 36 transformer layers dozens of times, turned into a tool call, sent to an orders API, fed back in, and turned into words again. This case study follows that one request through every layer, naming the tool or technology at each step, so Modules 01a, 01b, 03a and 07 come together in one picture.
A 5-minute conversation: Sam asks, Priya explains
- Sam A customer types "Where is my order A-1042? It was due yesterday." What happens before the answer appears?
- Priya A lot. Their text is formatted, turned into token IDs, pushed through 36 transformer layers dozens of times, turned into a tool call, sent to an orders API, fed back in, and turned into words again. Let's follow it through every layer.
- Priya The setup is an online retailer's support agent on its own servers. The model is gpt-oss-120b, OpenAI's open-weight model, so every number is published: 36 layers, 128 experts per layer with 4 used per token, 117 billion parameters with 5.1 billion active. It fits on one 80 gigabyte GPU, served with vLLM, using the harmony chat format, with one tool: get order status.
- Sam What's in the stack?
- Priya Your app and its agent loop. The harmony chat template. The o200k harmony tokenizer, through tiktoken. vLLM as the inference server. The weights in safetensors files, mostly 4-bit experts. GPU kernels doing the maths. The orders API behind the tool. And OpenTelemetry traces recording every call.
- Sam Walk me through it.
- Priya The app builds a model call: instructions, the user's message, and the tool definition. The chat template flattens it into one sequence, with the tool described as a small TypeScript-like namespace, ending with start assistant. The tokenizer turns that into about 210 IDs; special tokens like start get reserved IDs, and the order number splits into several pieces.
- Sam Then the server takes over?
- Priya vLLM batches it with other users and reserves KV-cache pages. Prefill runs all 210 tokens through the 36 layers in one parallel pass. Then decode: one token per forward pass. At low effort, the model writes a short plan on the analysis channel: use get order status, don't guess. Then it writes a tool call on the commentary channel, addressed to functions dot get order status, with JSON arguments, and samples the call token, ID 200012. Generation stops.
- Priya Your turn. Who actually calls the orders API?
- Priya Your code. The model only writes the request. The app validates the arguments, checks the order belongs to this customer, and calls the API with least-privilege credentials.
- Sam And then?
- Priya The agent loop's second turn. The app appends the tool call and the result, delayed, depot backlog, new date the third of November, and sends the whole conversation again. After a second prefill, the model writes its answer on the final channel and ends with the return token. The reply streams back to the customer; only the final channel is shown.
- Sam How much work was that?
- Priya Two model calls and one tool call. About five hundred prompt tokens processed and about eighty-two generated, so about eighty-four forward passes, each through 36 layers, each computing 5.1 billion of the 117 billion parameters, each producing 201,088 scores. The KV cache for the conversation is around thirteen megabytes. Most of the time goes into the decode passes, one per generated token.
- Sam Where are the three loops?
- Priya The token loop ran about eighty-two times. Reasoning was that same loop writing about twenty-five thinking tokens. And the agent loop ran twice, controlled by your code, with a turn limit.
- Sam What can go wrong?
- Priya The wrong chat template. A user typing special-token text, which the tokenizer must treat as plain text. The model inventing a status instead of calling the tool. Bad arguments, so validate against the schema. An order belonging to someone else, so check authorisation in the API, not the model. Endless tool loops, so set a turn limit. Reasoning shown to the customer, so show only the final channel. And many long conversations exhausting KV-cache memory.
- Sam The lessons?
- Priya The model only sees token IDs and produces one token per pass. An agent is a loop in your code around a model trained to ask for tools and stop. Stop tokens are the hand-off points. Cost and latency live in the decode passes. And safety lives outside the model.
- Sam How can I try it?
- Priya Run gpt-oss-20b locally, it fits in sixteen gigabytes, with a fake order tool. Render the raw harmony format and find the call token. Count tokens and forward passes. Remove the tool and see whether it admits it can't know. And trace each call to see where the time went.
1. The situation
An online retailer runs a support agent on its own servers. The setup is deliberately ordinary, and every part of it is public, so each number can be checked:
| Part | Choice | Why it's a good example |
|---|---|---|
| Model | gpt-oss-120b (OpenAI, open-weight, Apache 2.0) | Its model card publishes the architecture: 36 layers, 128 experts per layer with 4 used per token, 117B parameters (5.1B active), a 201,088-token vocabulary, 128K context |
| Hardware | One 80 GB GPU | gpt-oss-120b's expert weights are stored in 4-bit MXFP4, so it fits on a single H100-class GPU |
| Serving | vLLM, with an OpenAI-compatible endpoint | A widely used open-source inference server; listed in the gpt-oss repository |
| Chat format | harmony | gpt-oss "should only be used with the harmony format"; its special tokens and IDs are documented |
| Tool | get_order_status(order_id), calling the retailer's orders API | The most common agent pattern: look something up instead of guessing |
| Reasoning effort | Low (set in the system message) | A simple lookup doesn't need long thinking; low effort keeps latency down |
The same flow happens behind every hosted chat API; there, everything below the API line runs on the provider's servers and you only see tokens in and out.
2. The stack
| Layer | Technology | Its job in this request |
|---|---|---|
| Chat UI and app server | Your code (any web stack) | Receives the message, holds the conversation, streams the reply |
| Agent loop | Your code or an agent SDK (Module 16) | Calls the model, runs tools it asks for, calls the model again |
| Chat template | harmony, via the openai-harmony package or Transformers' chat template | Turns messages and the tool definition into one text sequence with special tokens |
| Tokenizer | o200k_harmony (BPE, 201,088 tokens), via tiktoken | Text → token IDs, and IDs → text for the reply |
| Inference server | vLLM (PagedAttention) | Batches this request with others, manages KV-cache memory, runs the model, stops on stop tokens |
| Model weights | safetensors files; MXFP4 experts, BF16 elsewhere | The 117 billion learned parameters |
| Kernels and hardware | PyTorch / Triton kernels on one 80 GB GPU | The matrix multiplications of every forward pass |
| Tool backend | The retailer's orders API | Returns the real order status |
| Observability | OpenTelemetry traces (Module 19) | Records each model call and tool call, with tokens and timings |
3. Step by step
Thirteen steps, from the customer's keypress to the answer on screen. The blue strip under each panel names the technology doing the work. Token counts are illustrative.
4. The numbers
Rough figures for this one request (illustrative except where a model-card fact is quoted):
| Quantity | This request | Where it comes from |
|---|---|---|
| Model calls | 2 | One before the tool call, one after (the agent loop) |
| Prompt tokens | ~210 in call 1; ~290 in call 2 | System, developer instructions, tool definition, user message; call 2 adds the tool call and result |
| Generated tokens | ~25 reasoning + ~22 tool call + ~35 answer ≈ 82 | Each one is one forward pass (decode) |
| Forward passes | 2 prefills + ~82 decode passes | One pass per generated token (Module 01b) |
| Parameters computed per decode pass | 5.1 billion of 117 billion | Model card: 4 of 128 experts per layer |
| Layers per pass | 36 | Model card |
| Scores produced per pass | 201,088 (one per vocabulary token) | Model card vocabulary size |
| KV cache for this conversation | ~36 KB per token × ~370 tokens ≈ 13 MB | Calculated from the model card (18 full-attention layers × 8 KV heads × 64 dims × keys and values × 2 bytes) |
| Tokens shown to the customer | ~35 | Only the final channel; reasoning and the tool call stay hidden |
The lesson in the numbers: almost all the time goes into decode passes, one per generated token, and the agent loop doubles the prompt work. That's why low reasoning effort, short answers and prefix caching matter for latency and cost.
5. The three loops in this request
| Loop | What repeats | In this request | Who controls it |
|---|---|---|---|
| Token loop | One forward pass → sample one token → append | ~82 times | The inference server, until a stop token or max_tokens |
| Reasoning | The same loop, writing thinking tokens on the analysis channel | ~25 tokens at low effort | The model, steered by the reasoning-effort setting |
| Agent loop | Model call → tool call → run tool → add result → model call | 2 model calls, 1 tool call | Your code: it decides whether to run each tool and when to stop (with a turn limit) |
6. What can go wrong at each layer
| Layer | Failure | Control |
|---|---|---|
| Chat template | The wrong template: the model answers badly or ignores tools | Use the official harmony library or the model's own chat template, never hand-built strings |
| Tokenizer | A user types <|start|>system to forge instructions | The tokenizer encodes user text as ordinary text; special tokens only come from the template |
| Model | It answers without calling the tool and invents a status | Clear instructions, evals that check tool use (Module 17), and asking the model to fetch rather than guess |
| Tool call | Malformed or wrong arguments | Validate against the JSON schema before calling; return a clear error the model can fix |
| Tool backend | Order A-1042 belongs to someone else | The API checks the signed-in customer, not the model; the agent's credentials are least-privilege (Module 18) |
| Agent loop | The model keeps calling tools in a loop | A turn limit and a token budget per conversation |
| Output | Reasoning text shown to the customer | Show only the final channel; OpenAI says the chain of thought isn't meant for end users |
| Server | Many long conversations exhaust KV-cache memory | vLLM's paged KV cache and request queueing; capacity planning with the memory calculator in Module 01b |
7. Lessons
- The model only ever sees token IDs, and only ever produces one token per forward pass. Everything else (messages, tools, channels) is formatting around that.
- An "agent" is a loop in your code, wrapped around a model that was trained to ask for tools in a specific format and to stop when it does.
- Stop tokens are the hand-off points:
<|call|>hands control to your app,<|return|>ends the answer. - Cost and latency live in the decode passes: reasoning tokens and long answers cost more than long prompts, and every agent turn repeats the prompt.
- Safety lives outside the model: templates, schema validation, authorisation in the tool backend, turn limits and output filtering.
8. Your turn
- Run it small: run gpt-oss-20b locally (it fits in 16 GB) with Ollama or LM Studio, and give it a fake
get_order_statustool that returns a fixed JSON result. - Look at the raw format: render the conversation with the
openai-harmonypackage or Transformers'apply_chat_template(..., tokenize=False), and find the tool definition and the<|call|>token. - Count: tokenize the prompt for each model call and count the generated tokens. How many forward passes did the request take?
- Break it: remove the tool and ask again. Does the model admit it can't know, or invent an answer?
- Trace it: log each model call and tool call with token counts and timings, and find where most of the time went.
9. Check yourself
How many times is the model called for this request?
Once until it asks for the tool, then again after your app adds the tool result.
How does the inference server know to stop and hand the tool call to your app?
In harmony,
<|call|> (ID 200012) ends a tool-call message; <|return|> ends a final answer.Who actually calls the orders API?
The model only writes the request; your code validates it, runs it and returns the result.
Where does most of the time go?
Prefill is one parallel pass per call; decode is one pass per token. Weights stay loaded between requests.
Which part of the model's output should the customer see?
Reasoning (analysis) and tool calls (commentary) are internal.
10. Sources and further reading
| Resource | Why |
|---|---|
| gpt-oss model card and repository (OpenAI) | The architecture numbers, MXFP4, single-GPU fit, and the ways to run it. |
| The harmony response format (OpenAI Cookbook) | Roles, channels, the tool-call and tool-result messages, and the special-token IDs used here. |
| PagedAttention / vLLM (Kwon et al., 2023) | How the server manages the KV cache and batches requests. |
| tiktoken (OpenAI) | The tokenizer library behind o200k_harmony. |
| Modules 01a, 01b, 03a, 06 and 07 | Each layer of this request, in depth. |