AI engineering · Module 20 of 21 · Level 5: Production
Choosing tools and staying current
New models and tool features ship every week, and every launch claims to change everything. You don't need to chase them. Pick tools by testing them on your own work, and keep up with a small routine built on primary sources. The concepts from this course change slowly; the buttons change monthly.
1. The idea
A professional cyclist doesn't buy a bike because of an advert. They ride the candidates on their usual routes, timing the same climbs, and note how each one feels after three hours. Once a season they test again. In between, they read the maker's spec sheets, not the forum rumours.
- Choose with a trial on your own tasks. Run the same handful of real tasks in each candidate tool, under the same conditions, and score the results against criteria you weighted in advance.
- Keep up from primary sources. Vendor changelogs, documentation and standards sites; a few practitioners whose claims you can check. Skim weekly, try one thing monthly, re-test quarterly.
- Invest in what transfers. AGENTS.md, Agent Skills, MCP servers, evals and good habits work across all five tools (Modules 09–11). Switching tools then costs days, not months.
2. How it works
3. Try it yourself
Your tool scorecard
Set how much each criterion matters to you, then score each tool from 1 to 5 after running the same tasks in each. Everything starts at 3: these are your scores, not ours. They're saved in this browser only.
Should you act on this news?
Pick a headline, or answer the three questions for one you've just seen.
4. Why it matters at work
Run a fair trial
- Pick 4–6 real tasks you do often: build a small feature, fix a bug, refactor, write tests, explain unfamiliar code, review a PR.
- Same starting point: the same commit (a fresh branch or git worktree per tool), the same AGENTS.md, the same prompt, a similar model tier.
- Record facts per task: finished or not, how many corrections you made, time to an accepted change, cost or quota used, anything unsafe it tried.
- Score against weighted criteria, decided before you start, so the flashiest demo doesn't win by default.
- Check the non-functional things: data and privacy terms for your plan, admin and security controls (Module 18), price predictability, and whether it works in your editor, OS and CI.
Choosing models for a feature works the same way: shortlist with public leaderboards and independent comparisons, then decide with your own eval set (Module 17) and cost per successful task (Module 19).
Keep switching cheapPut instructions in AGENTS.md, know-how in skills, integrations in MCP servers, and model calls behind a small interface in your code. Then a better tool or model is a configuration change, not a rewrite.
5. Where to follow your tools
Primary sources only. Checked October 2026.
| Tool | Changelog |
|---|---|
| Claude Code | code.claude.com/docs/en/changelog |
| Antigravity CLI | antigravity.google/docs/changelog |
| GitHub Copilot | github.blog/changelog (Copilot label) |
| Cursor | cursor.com/changelog |
| Codex | developers.openai.com/codex/changelog |
| Also worth following | Why |
|---|---|
| agents.md, agentskills.io, modelcontextprotocol.io and the MCP registry | The open standards behind instructions, skills and integrations in every tool |
| Simon Willison and Hamel Husain | Practitioners who test claims and show their work (security, evals) |
| SWE-bench and Artificial Analysis | Public benchmarks and independent price/speed comparisons, for shortlisting only |
| Each model vendor's news page and API changelog | New models, deprecations and price changes, with dates |
6. Remember it
Memory hook: "Own tasks, own scores, primary sources"Judge tools on your own work, with criteria you set in advance, and learn about changes from the people who ship them.
How this hook works: what “Own tasks, own scores, primary sources” means
What the slogan means, phrase by phrase
- “Own tasks”
- Real tasks from your work, from the same starting point in every tool. Demos and benchmarks use someone else's tasks.
- “own scores”
- Criteria and weights you choose before testing, scored from what actually happened: finished, corrections needed, time, cost, safety.
- “primary sources”
- Changelogs, docs and standards sites, plus a few practitioners who show evidence. Not "this changes everything" posts.
Why this picture fits: all three phrases are about evidence you control. That's what lets you ignore most launches calmly and spot the few that matter to you.
The story → the concept
| The cyclist | You |
|---|---|
| Their usual routes | Your own tasks |
| Timing the same climbs | Same commit, same AGENTS.md, same prompt |
| What matters on a long ride | Your weighted criteria |
| The maker's spec sheet | Vendor changelogs and docs |
| Testing again each season | Quarterly re-test of tools and evals |
| Parts that fit any frame | AGENTS.md, skills, MCP: portable setup |
Watch it on an example
- A headline: "New model tops a coding leaderboard."
- Primary source: the vendor's changelog says it's available in two of your tools, at a higher price.
- Own tasks, own scores: you run your five scorecard tasks with it. It finishes one more task but costs 40% more; you use it for hard bugs only.
The trap it saves you from
Switching tools every month after each launch, and never getting good at any of them.
It still works when…
The tools in this course are replaced. The method (real tasks, set criteria, primary sources, portable setup) doesn't depend on any one of them.
7. Common mistakes
Choosing by leaderboardBenchmarks measure someone else's tasks. Use them to shortlist, then test on yours.
Unfair trialsDifferent prompts, a stronger model in one tool, or more context in another. Hold everything equal except the tool.
Learning from hype channelsMany "best AI tools" lists sell a course or a community. Prefer vendors' own docs, universities and practitioners who show their work.
Lock-in by habitTool-specific instruction files and custom glue make switching painful. Keep the shared parts in open formats.
Never re-testingYour scorecard ages fast. Re-run it each quarter, or when a major model ships.
8. Hands-on lab (roadmap weeks 25–26)
- Pick five real tasks from your recent work and write each as a prompt.
- Run them in two or more tools from the same commit and the same AGENTS.md. Record finished, corrections, time and cost.
- Fill in the scorecard above with weights you chose before testing. Note what surprised you.
- Set up your routine: bookmark the changelogs, block 15 minutes a week, and put a quarterly re-test in your calendar.
- Capstone: one repository with AGENTS.md, a skill, an MCP server, an evaluated AI feature, security controls and a cost estimate, all from this course. Write up what you'd do differently.
9. Self-check quiz
What's the fairest way to compare two coding tools?
Same tasks, same commit, same instructions: only the tool changes.
When should you decide the scorecard weights?
Setting them afterwards lets you rationalise whichever tool you already liked.
The most reliable place to learn what changed in a tool?
Primary source, dated, and specific about what actually changed.
What keeps switching tools cheap?
Open formats that all five tools read (Modules 09–11).
10. Go deeper
| Resource | Time | Why |
|---|---|---|
| The 26-week roadmap | 5 min | See where each module fits, and plan your next pass through the course. |
| Module 09, 10 and 11 | Review | The portable setup that makes switching cheap. |
| Module 17: Evals | Review | The same method, applied to models and prompts. |
| simonwillison.net | Weekly | A practitioner who tries new releases in public and writes up what works. |