AI engineering · Module 03a · Level 1: How AI works
From base model to agent
Module 03 explained where a model's knowledge comes from. This module follows the whole production line: how training adjusts the weights, why a freshly pretrained "base model" can't hold a conversation, and the stages that turn it into an assistant, then a reasoning model, then an agent that calls tools. Each stage uses a different kind of data and a different way of rewarding the model.
A 4-minute conversation: Sam asks, Priya explains
- Sam How does a model go from reading the internet to calling tools for me?
- Priya Through a production line. Pretraining makes a base model that knows a lot but only continues text. Then post-training stages teach it to answer as an assistant, to prefer honest helpful answers, to reason, and finally to use tools.
- Priya Think of training a doctor. Years of reading every textbook: that's pretraining. Residency, copying how senior doctors handle patients: supervised fine-tuning. Feedback on which of two approaches was better: preference tuning. Practice cases with a right answer, until the reasoning is reliable: reinforcement learning. And learning to order tests instead of guessing: tool use.
- Sam And all of that changes the same weights?
- Priya Yes, with the same machinery. Show the model text, measure how wrong its prediction was with a loss, and nudge every weight slightly in the direction that lowers the loss. That's gradient descent; backpropagation works out the direction for billions of weights at once. Repeat for trillions of tokens.
- Sam Show me the stages on a real question.
- Priya Take "Where is my order A-1042? It was due yesterday." The base model continues the text, writing more complaints. After supervised fine-tuning, it answers like an assistant, but invents a delivery date. After preference tuning, it honestly says it can't see order details. After reasoning training, it plans first on an analysis channel. And after tool-use training, it calls get order status, stops, reads the real result, and answers: delayed at the depot, now due on the third of November.
- Sam How is reasoning trained?
- Priya DeepSeek's R1 paper showed it can be taught with reinforcement learning on problems with checkable answers. For each problem, sample a group of answers, reward the ones whose final answer is right, and push the model towards the above-average ones. That's GRPO. They avoided learned reward models because the model learned to game them. Reasoning emerged on its own: answers got longer, the model started saying "wait, let me re-check", and maths accuracy rose from about sixteen to seventy-eight percent.
- Priya Your turn. Can a smaller, well-tuned model beat a bigger raw one?
- Priya Yes. In OpenAI's InstructGPT paper, people preferred a 1.3 billion parameter tuned model over the 175 billion parameter GPT-3.
- Sam What can I try?
- Priya One real training step on a toy model. The right next word is "tomorrow". Press train, and the scores move exactly as gradient descent moves them; watch the loss fall. Then a sorter: which training stage teaches each skill.
- Sam What does this mean for me?
- Priya Knowledge stops at a cutoff, so recent facts need retrieval or tools. Don't choose models by size alone. Reasoning models shine where answers can be checked. Your evals should check real outcomes, because rewards can be gamed. Use each open model's own tool format. And if you fine-tune, that's supervised or preference tuning on top of an existing model; try prompting and retrieval first.
- Sam What methods will I hear about?
- Priya Supervised fine-tuning on example conversations. RLHF with a reward model and PPO, from InstructGPT. DPO, which trains on preference pairs directly with no reward model. Rejection sampling, keeping the best of many answers. GRPO for reasoning. And distillation, training a smaller model on a bigger one's outputs.
- Sam The memory hook?
- Priya Read everything, copy the best, learn what people prefer, practise until right, then use tools.
- Sam Common mistakes?
- Priya Fine-tuning to add facts. Chatting with a base model. Choosing by parameter count. Ignoring the format a model was trained on. And trusting reasoning everywhere, not just where answers can be checked.
- Sam And the lab?
- Priya Give a base model and its instruct version the same question. Read the gpt-oss model card for its data, cutoff and post-training. Compare a reasoning model at low and high effort. Then write five example conversations for your use case, and a worse answer for two of them.
1. The idea
Training a doctor. First, years of reading every textbook and journal: broad knowledge, but no bedside manner (pretraining). Then supervised residency: watching and copying how senior doctors handle real patients (supervised fine-tuning). Then feedback from patients and seniors on which of two approaches was better (preference tuning). Then practice cases with a right answer, solved again and again until the reasoning is reliable (reinforcement learning). Finally, learning to order tests and read the results instead of guessing (tool use).
| Stage | Data | What the model learns | Result |
|---|---|---|---|
| Pretraining | Trillions of tokens of text and code (Llama 3: over 15 trillion) | Predict the next token | Base model: knows a lot, continues any text, doesn't follow instructions |
| Supervised fine-tuning (SFT) | Thousands to millions of example conversations, written or checked by people, in the chat template | Answer in the assistant's role and format | Instruction-following model |
| Preference tuning | Pairs of answers where people (or a model) picked the better one | Prefer helpful, honest, safe answers | Assistant that people prefer |
| Reinforcement learning with verifiable rewards | Problems with checkable answers: maths with known results, code with tests | Think step by step until the answer is right | Reasoning model |
| Tool-use and agentic training | Conversations with tool definitions, tool calls and results; multi-step tasks | When and how to call tools, and use their results | Agentic model |
Everything after pretraining is called post-training. It uses a tiny fraction of the compute, but it decides how the model behaves.
2. How it works
How training changes the weights
Every stage uses the same machinery. Show the model some text, let it predict, measure how wrong it was with a loss (for next-token prediction, how low a probability it gave the real next token), then nudge every weight slightly in the direction that would have made the loss smaller. That nudge is gradient descent, and working out the direction for billions of weights at once is backpropagation. Repeat for trillions of tokens.
The same prompt, through each stage
Watch the support question from Module 01b at each stage of the production line:
Reinforcement learning for reasoning, in more detail
DeepSeek's R1 paper (published in Nature in 2025) showed that reasoning can be taught with reinforcement learning on problems whose answers can be checked automatically, without people writing out reasoning steps:
3. Try it yourself
One training step, for real
A toy model has learned scores for the next word after "The order will arrive". The training example says the right word is tomorrow. Press Train one step: the scores move exactly as gradient descent on the cross-entropy loss moves them (score += learning rate × (target − probability)). Watch the loss fall.
Which stage teaches this?
4. Why it matters at work
| Fact about training | What it means for you |
|---|---|
| Knowledge comes from pretraining and stops at a cutoff | gpt-oss's knowledge cutoff is June 2024. Recent facts must come from your prompt, retrieval or tools (Modules 03, 15). |
| Post-training shapes behaviour, and a smaller well-tuned model can beat a bigger raw one | In InstructGPT, people preferred a 1.3B-parameter tuned model over the 175B-parameter GPT-3. Don't choose models by size alone. |
| Reasoning is trained with checkable rewards | Reasoning models shine on maths, code and logic, where answers can be verified; gains are smaller on open-ended writing. |
| Reward models can be gamed | DeepSeek avoided learned reward models for reasoning because of reward hacking. Your own evals (Module 17) must check real outcomes, not proxies. |
| Agentic ability is trained in a specific format | Open models expect their own chat and tool format (gpt-oss: harmony). Use the official template, or tool calling degrades. |
| You can do the later stages yourself | Fine-tuning on your data is SFT or preference tuning on top of an existing model; gpt-oss-120b can be fine-tuned on a single H100 node. Try prompting and retrieval first: they're cheaper and easier to change. |
The training stack
- Data pipelines collect, filter and deduplicate text, then tokenize it (Module 01a) into training sequences.
- A framework such as PyTorch defines the network and computes gradients automatically.
- Thousands of GPUs train together: the data, the layers and even single matrices are split across machines (data, pipeline and tensor parallelism), with mixed precision to save memory.
- Checkpoints save the weights regularly, as safetensors files, so a crash doesn't lose weeks of work.
- Post-training reuses the same stack with smaller datasets: demonstrations for SFT, preference pairs for DPO, and for RL a loop that samples answers, scores them and updates the weights.
The honest caveatsFrontier labs publish few details of their training data and post-training recipes; open-weight models (gpt-oss, Llama, DeepSeek) are the best public evidence, and even they leave much out. The stage list here is a simplification: real pipelines mix stages, repeat them and add safety training throughout. And "agentic training" is the least documented stage of all.
5. Training methods you'll hear about
Checked 1 November 2026.
| Method | How it works | Known from |
|---|---|---|
| SFT (supervised fine-tuning) | Train on example conversations in the chat template, with the same next-token loss as pretraining | Every assistant model |
| RLHF with a reward model (PPO) | Collect rankings of answers, train a reward model on them, then optimise the model with reinforcement learning to score highly | InstructGPT (OpenAI, 2022) |
| DPO (direct preference optimization) | Train directly on preference pairs with a classification-style loss: no reward model and no RL loop | Rafailov et al., 2023; used alongside PPO in Llama 3 |
| Rejection sampling | Generate many answers, keep the best (by a checker or reward model), fine-tune on those | Llama 3; DeepSeek-R1's third stage |
| GRPO | Sample a group of answers per question, score each, and push the model towards the above-average ones; no separate critic model | DeepSeek-R1 (2025) |
| Chain-of-thought RL | Reward correct final answers so the model learns to reason first; reasoning effort becomes adjustable | gpt-oss ("similar to OpenAI o3"), DeepSeek-R1 |
| Distillation | Train a smaller model on a larger model's outputs, including its reasoning | DeepSeek-R1's smaller distilled models |
6. Remember it
Memory hook: "Read everything, copy the best, learn what people prefer, practise until right, then use tools"Pretraining, SFT, preference tuning, reinforcement learning, tool use.
How this hook works: what the five phrases mean
What the slogan means, phrase by phrase
- “Read everything”
- Pretraining on trillions of tokens: the base model.
- “copy the best”
- SFT on example conversations.
- “learn what people prefer”
- Preference tuning: RLHF or DPO.
- “practise until right”
- RL with verifiable rewards: reasoning.
- “then use tools”
- Training on tool calls and multi-step tasks: an agent.
Why this picture fits: it's the order a professional is trained in, and the order labs train models in.
The story → the concept
| Doctor's training | Model training |
|---|---|
| Medical school reading | Pretraining |
| Residency | Supervised fine-tuning |
| Feedback on bedside manner | Preference tuning |
| Practice cases with answers | RL with verifiable rewards |
| Ordering tests | Tool use |
The trap it saves you from
Thinking an assistant is "just" a big pretrained model. Most of what makes it useful, safe and agentic comes from post-training.
7. Common mistakes
Fine-tuning to add factsFine-tuning is good at behaviour and format, poor at adding reliable knowledge. Use retrieval for facts that change.
Chatting with a base modelA base model continues text; it may write the next question instead of answering. Use an instruction-tuned model.
Choosing by parameter countPost-training matters as much as size; compare models on your own tasks (Module 20).
Ignoring the format a model was trained onTool calls and roles only work reliably in the model's own template.
Trusting reasoning everywhereReasoning training targets checkable problems; for judgement calls, still check the output.
8. Hands-on lab (roadmap week 3)
- Base versus instruct: on Hugging Face, find a base model and its instruct version (for example a Llama or Qwen pair). Give both the same question and compare.
- Read a model card: in the gpt-oss model card, find the pretraining data, knowledge cutoff and post-training method.
- Feel reasoning effort: ask a reasoning model the same logic puzzle at low and high effort; compare thinking length, time and accuracy.
- Write SFT data: write five example conversations for your support use case in a chat format. What would a sixth need to cover?
- Write preference pairs: for two of them, write a worse answer too. What makes the better one better?
9. Self-check quiz
What does a base model do with "Where is my order A-1042?"
Following instructions comes from SFT and preference tuning.
What does one gradient-descent step do?
Backpropagation computes that direction for all weights at once.
What was InstructGPT's headline result?
Post-training can matter more than size.
How does DPO differ from classic RLHF?
A simple classification-style loss replaces the reward model and PPO.
What did DeepSeek-R1's reinforcement learning reward?
Rule-based accuracy and format rewards; learned reward models were avoided because of reward hacking.
Your model must know this week's prices. Best approach?
Fine-tuning teaches behaviour; changing facts belong in the prompt via retrieval or tools.
10. Go deeper
| Resource | Time | Why |
|---|---|---|
| Training language models to follow instructions with human feedback (InstructGPT, OpenAI, 2022) | 45 min | The SFT → reward model → PPO recipe behind the first chat assistants. |
| Direct Preference Optimization (Rafailov et al., 2023) | 40 min | Preference tuning without a reward model or RL. |
| DeepSeek-R1 (DeepSeek-AI, 2025; Nature) | 60 min | Reasoning from reinforcement learning: GRPO, rewards, the "aha moment", the four-stage pipeline. |
| gpt-oss model card (OpenAI) | 30 min | Pretraining data, cutoff, chain-of-thought RL and the harmony format for an open agentic model. |
| Introducing Meta Llama 3 (Meta) | 15 min | 15 trillion pretraining tokens, and post-training with SFT, rejection sampling, PPO and DPO. |
| Chat templates (Hugging Face) | 15 min | How SFT data is formatted, and why base models just continue text. |