AI engineering · Module 17 of 21 · Level 5: Production
Evals: how you know it works
Normal code has unit tests. AI features need evals: a set of real inputs, plus checks on the outputs, run every time you change the prompt, the model or the retrieval. Without them, every change is a guess, and "it looked fine when I tried it" is the only evidence you have.
1. The idea
A restaurant doesn't guess whether the food is good. The head chef tastes plates and writes down exactly what's wrong ("too salty", "cold in the middle"), then checks those things on every plate before it goes out. Evals are the same: taste first, name what's wrong, then check it every time.
The order matters:
- Read real outputs (error analysis). Look at 50–100 actual inputs and outputs and write a short note on each one that's wrong.
- Name the failures. Group the notes into a few failure types, and count them. Now you know what actually goes wrong, and how often.
- Write a check for each failure type: code where possible (valid JSON? cited a source?), a model as judge where it needs judgement (was the tone rude?). Each check answers pass or fail.
- Run them on every change, and keep adding the failures you find in production.
Public benchmarks (Module 04) tell you which models to shortlist. Your own evals tell you whether your feature works.
2. How it works
Three kinds of grader
| Grader | Good for | Watch out for |
|---|---|---|
| Code | Format, schema, exact values, "did the tests pass?", "is a source cited?"; fast, cheap, repeatable | Too strict on valid variations |
| Model (LLM-as-judge) | Tone, helpfulness, "is every claim supported by the sources?" | Its own mistakes. Check it against human labels before trusting it |
| Human | The gold standard; labelling, and checking the other graders | Slow and expensive; use for calibration, not every run |
3. Try it yourself
Error analysis: label eight real answers
A support bot answers questions from a help centre. Mark each answer Pass, or pick what's wrong with it. Then see which failure is most common: that's what you fix and test first.
Can you trust the judge?
An LLM judge marks answers as failing. Twenty answers have human labels (green = good, red = bad). Move the judge's strictness and watch two numbers: how many real failures it catches, and how many good answers it wrongly fails.
4. Why it matters at work
- Start small, from real failures. 20–50 cases taken from real bugs and complaints beat 1,000 made-up ones. Add every new production failure as a case.
- Pass/fail, not 1–5 scores. "Is every claim supported by a cited source? yes/no" gives consistent labels; a 1–5 "quality" score doesn't.
- Avoid generic metrics. Off-the-shelf "helpfulness" or "coherence" scores create false confidence. Measure the failures you actually found.
- Check the judge. Before trusting an LLM judge, compare it with human labels on a held-out set: how many real failures it catches (true positive rate) and how many good answers it passes (true negative rate).
- Two kinds of suite. Regression evals should stay near 100% and run in CI on every prompt or model change. Capability evals start low and show progress on hard tasks.
- Agents vary between runs. Run each task several times. pass@k = at least one of k tries works (fine when a human picks the best). pass^k = all k tries work (what users need from a product).
- Read the transcripts. A rising score means nothing if the grader is wrong. Look at what passed and failed.
Evals for your coding setup tooThe same idea works for your agent configuration: keep a handful of real tasks from your repo, and re-run them when you change AGENTS.md, skills or the model. Did the agent still run the right tests and follow the rules?
5. Eval tools
You can start with a spreadsheet and a script. Tools help once you run evals often. Checked October 2026.
| Tool | What it is |
|---|---|
| Claude Console evaluation tool | Run a prompt over many test cases in the browser and compare outputs side by side |
| OpenAI Evals | Define test data and graders through the API or dashboard, then run them against models and prompts |
| Vertex AI Gen AI evaluation | Google Cloud's evaluation service, with code-based and model-based metrics |
| Inspect | Open-source eval framework from the UK AI Security Institute, strong for agent tasks; works with many providers |
| promptfoo | Open-source command-line tool: test cases and assertions in a config file, runs in CI, also does red-teaming |
Observability platforms (Module 19) often include evals too, so you can turn a bad production trace into a test case in one click.
6. Remember it
Memory hook: "Read, name, check"Read real outputs. Name the failures you see. Check for each one, pass or fail, on every change.
How this hook works: what “Read, name, check” means
What the slogan means, word by word
- “Read”
- Error analysis: look at real inputs and outputs, one by one, with a short note on each problem. Nothing replaces this.
- “name”
- Group the notes into a few failure types and count them. The biggest groups are what you fix and test first.
- “check”
- One pass/fail check per failure type (code if possible, a validated judge if not), run on every prompt, model or retrieval change.
Why this picture fits: it puts reading first. Most bad eval suites start at "check", with generic scores that measure nothing in particular.
The story → the concept
| The kitchen | Evals |
|---|---|
| Tasting plates | Error analysis on real outputs |
| "Too salty", "cold in the middle" | Named failure types |
| A thermometer in every plate | Code-based check |
| A trained taster | LLM-as-judge, checked against the chef |
| The head chef | Human labels, the gold standard |
| Checking again after changing the recipe | Regression evals in CI |
Watch it on an example
- Read: 60 support answers; 14 have problems.
- Name: 6 "answered from memory, not the docs", 4 "wrong order number format", 3 "too long", 1 "rude".
- Check: a judge for "every claim cited", a regular expression for order numbers, a length limit in code. Run all three on every prompt change.
The trap it saves you from
A dashboard full of generic scores that go up while users still complain.
It still works when…
You evaluate agents: read the transcripts, name where runs go wrong (wrong tool, gave up early, broke a rule), and check the outcome, not just the final message.
7. Common mistakes
Trusting an unchecked judgeAn LLM judge is a model too. Measure how well it agrees with human labels before using its numbers.
Starting with metrics instead of readingYou can't choose good checks until you know how your feature fails.
1–5 scoresNobody can say what separates a 3 from a 4. Use pass/fail with a written definition.
Only happy-path casesInclude hard, ambiguous, adversarial and "should refuse" cases, and questions with no answer.
A test set that never changesAdd every production failure. A frozen set slowly stops matching reality.
8. Hands-on lab (roadmap weeks 20–21)
- Read 50 outputs from your Module 14–15 feature (or any AI tool you use). Write one line on each problem.
- Name the failures: group the notes into 3–6 types and count them.
- Build a 30-case set with expected behaviour for each, including hard and "should refuse" cases.
- Write the checks: code where possible, one LLM judge with a pass/fail rubric. Label 20 outputs yourself and measure the judge's agreement.
- Compare two models (or two prompts) on the set and report pass rates. Run the agent tasks 3 times each and note pass^3.
9. Self-check quiz
What's the first step in building evals?
Error analysis tells you which checks are worth writing.
Which grading scale gives the most consistent labels?
A binary question forces a clear definition of "good".
How do you know an LLM judge is trustworthy?
Measure how many real failures it catches and how many good answers it passes.
A customer-facing agent must succeed every time. Which metric matters?
pass^k: all k attempts succeed. pass@k only needs one.
10. Go deeper
| Resource | Time | Why |
|---|---|---|
| AI evals FAQ (Hamel Husain and Shreya Shankar) | 1 h | The most practical guide: error analysis, binary judges, validating judges. |
| Demystifying evals for AI agents (Anthropic) | 30 min | Tasks, trials, graders, pass@k vs pass^k. |
| Define success and build evaluations (Claude docs) | 20 min | Writing success criteria and graders, with examples. |
| Inspect documentation | 1 h | An open-source framework to run real eval suites, including agents. |