AI engineering · Module 03 of 21 · Level 1: How AI works

Training, knowledge cutoff and hallucination

Where does a model's knowledge come from, why does it stop at a date, and why does it sometimes state made-up things with total confidence? Knowing how models are made tells you exactly when to trust them, and how to make them trustworthy.

1. The idea

A student spends years reading an entire library, then months practising how to answer questions politely and step by step. Then they sit an exam with no books allowed, where every blank answer scores zero. They answer everything from memory, and when memory runs out, they write something that sounds right.

A model is made in stages, and each stage explains a behaviour you'll see every day:

Pretraining: reading the library. The model learns to predict the next token across an enormous collection of text and code. Along the way it absorbs grammar, facts, reasoning patterns and coding styles. The result is a base model: a superb autocomplete, but not yet a helpful assistant.
Post-training: learning to be an assistant. It's trained on example conversations (how a good assistant answers), on human and AI rankings of better and worse answers (RLHF and similar methods), and with reinforcement learning on tasks whose answers can be checked, like maths and passing code tests. That last step is what taught models to reason step by step.
Frozen and shipped. Training stops, the weights (the billions of numbers that hold all this) are frozen, and the model is released. Its knowledge ends at its knowledge cutoff; for example, Anthropic lists June 2026 for its current Claude models, and OpenAI lists April and May 2026 for its GPT-6 models.

Two things follow. Your chats don't teach the model: its weights don't change while you use it (vendors may use data for future training depending on your plan and settings, which is a privacy question, covered below). And it can't know anything after its cutoff, or anything private to your company, unless you put it in the context.

Why it hallucinates. A hallucination is a fluent, confident answer that's wrong. Two causes:

It writes what's plausible, not what's checked. Facts seen thousands of times are recalled reliably; facts seen once, or never, get filled in with something that fits the pattern.
Training rewarded guessing. OpenAI's 2025 research showed that when tests only count correct answers, guessing scores better than saying "I don't know", so models learn to guess. Newer models are trained to abstain more, but no model has eliminated this.

2. How it works

The model's built-in knowledge is frozen at the cutoff. Anything newer, or private, reaches it only through the context: your files, search results and tool output. That's called grounding.

Step through the same question answered two ways: from memory, and from a source.

3. Try it yourself

Why models learned to guess

A model faces 100 questions. It truly knows some answers; for the rest, a guess is right 1 time in 4. Compare two strategies: always guess, or say "I don't know" when unsure. Then change how a wrong answer is scored.

With no penalty, guessing always wins, so training on scores like that produces a model that guesses. Once a wrong answer costs more than a third of a point, honesty wins. That's the change the 2025 research argues for: grade models in a way that rewards "I don't know" over confident errors.

4. Why it matters at work

You don't need to distrust everything. Match your checking to the kind of claim:

Usually reliable	Always check
Explaining well-known concepts	Exact version numbers, release dates, prices
Common code patterns and idioms	Function and API signatures, especially in newer or less popular libraries
Summarising text that is in the context	Quotes, citations and links: open them
Rewording, drafting, brainstorming	Anything after the knowledge cutoff, or about your internal systems
Reading code it has actually opened	Legal, medical, financial or security claims

How to ground a model, from cheapest to strongest:

Give it the source. Point the agent at the file, attach the doc, or paste the relevant section.
Ask it to quote. "Quote the lines you rely on" makes invented details easy to spot.
Allow "I don't know". "If the sources don't say, answer not stated." This gives it permission to abstain.
Let it search. Web search or a documentation tool for anything recent.
Verify by running. For code, tests, type checks and actually executing it beat any amount of reading. Agents can do this themselves.

Privacy is part of thisWhat you put in a prompt goes to the vendor. Whether it's stored or used for training depends on your plan and settings: business and enterprise plans typically don't train on your data, while consumer settings vary. Never paste secrets or customer data, and know which plan you're using (Module 18).

5. In your five tools

Every tool can ground the model in your code, and most can reach the web. Checked October 2026.

Tool	Ground in your code	Ground in the web or docs
Claude Code	Reads, searches and runs anything in the repo	Built-in web search and web fetch tools; documentation MCP servers
Antigravity CLI	Code search, file reads and terminal commands in the workspace	Web search, used by its agents and subagents
GitHub Copilot (VS Code)	`#codebase`, or `#`-mention files and symbols	`#fetch` for a web page; web search tools
Cursor	Codebase indexing; `@`-mention files and folders	`@Docs` for indexed documentation, `@Web` for the web
Codex CLI	Reads, searches and runs anything in the repo	Web search: cached results by default; `--search` or `web_search = "live"` for live results

6. Remember it

Memory hook: "Closed book guesses, open book cites"From memory alone, a model fills gaps with plausible guesses. Give it the source and ask it to cite, and you can check every claim.

How this hook works: what “Closed book guesses, open book cites” means

What the slogan means, phrase by phrase

“Closed book”: Answering from training alone: the model's frozen memory of what it read before the cutoff. Nothing private, nothing recent.
“guesses”: When memory is thin, it still answers, because it produces plausible text and was rewarded for answering. That's hallucination.
“open book”: Grounding: the relevant source is in the context, from your files, a search, a tool or a document you attached.
“cites”: Ask it to quote or link what it relied on. Then you're checking evidence, not trusting confidence.

Why this picture fits: everyone knows a closed-book exam invites guessing and an open-book exam invites looking things up. It turns "AI hallucinates" from a vague worry into one concrete action: open the book.

The story → the concept

In the story	In the model
Years of reading the library	Pretraining on huge amounts of text and code
Practice in answering well	Post-training: examples, rankings (RLHF), reinforcement learning
The day the reading stopped	The knowledge cutoff
Exam with no books, blanks score zero	Answering from weights alone, trained by scores that rewarded guessing
Being allowed to bring the book	Grounding: files, search, tools, RAG in the context
"Show your working and page numbers"	Asking for quotes and citations

Watch it on an example

You ask: "What's the default timeout in our HTTP client wrapper?"

Closed book: the model has never seen your wrapper. It answers "30 seconds", the most common default it has read about. Plausible, and wrong.
Open book: you say "read lib/http/client.ts first". The agent opens the file.
Cites: "Line 12: timeoutMs = 10_000, so 10 seconds." You can check line 12 in two seconds.

The trap it saves you from

Treating confidence as evidence. A model sounds equally sure when it's right and when it's guessing. The only reliable signal is a source you can check.

It still works when…

You build AI features. RAG (Module 15) is "open book" automated: search your documents, put the best passages in the context, and require citations. Evals (Module 17) then measure how often the book was opened correctly.

7. Common mistakes

Asking for citations and not opening themModels can produce real-looking links, titles and quotes that don't exist. A citation is only useful once you've checked it.

Asking "are you sure?"The model may change a right answer or defend a wrong one: neither is evidence. Ask for the source or a test instead.

Using a model for recent facts without searchLibrary versions, APIs and prices after the cutoff are invisible to it. Turn on search or give it the docs.

Believing the model learns from your correctionsWithin a session it uses your correction because it's in the context. Next session, it's gone, unless you wrote it into a file the tool loads.

8. Hands-on lab (20 minutes)

Find the cutoff. Look up your main model's knowledge cutoff on the vendor's models page. Ask the model about something that happened after it, first without search, then with search on.
Closed vs open book on your code. In a coding agent, ask a specific question about your repo ("what does function X return when Y?") and tell it not to open files. Then ask again, letting it read the code and quote lines. Compare.
Permission to abstain. Ask about a made-up library function twice: once normally, once adding "If you're not sure it exists, say so." Note the difference.
Check three citations. Ask for sources on a technical claim and open every link. Note how many are real and actually support the claim.
Log it: one sentence in your learning log on when you'll ground the model from now on.

9. Self-check quiz

Does a model learn from your conversations as you use it?

The weights are frozen at release. It "remembers" within a session only because the conversation is in the context.

What most reliably reduces hallucination about your own codebase?

Grounding plus quotes turns an answer into something you can check. Bigger models still guess about things they've never seen.

Why do models guess instead of saying "I don't know"?

If blanks score zero and guesses sometimes score, guessing wins on average, so training on such scores teaches guessing.

Which stage turns a text predictor into a helpful assistant that reasons step by step?

Pretraining gives knowledge; post-training (examples, preference ranking, reinforcement learning) gives the assistant behaviour and reasoning.

10. Go deeper

Resource	Time	Why
Deep Dive into LLMs like ChatGPT (Karpathy)	3.5 h	Pretraining, post-training, RL and hallucinations explained by one of the people who built them.
Why language models hallucinate (OpenAI, 2025; paper)	15 min	The incentive explanation behind the widget above.
Anthropic Transparency Hub	10 min	Training data, cutoffs and data-use policies for a real model family.
How I use LLMs (Karpathy)	2 h	Practical habits for when to trust, when to search, and when to check.