InterviewKata
AI engineering · Module 01 of 21 · Level 1: How AI works

Tokens and next-token prediction: how an LLM writes

Every AI tool you'll use (chat, coding agents, AI features in products) is built on one simple loop: look at the text so far, guess the next small chunk, add it, repeat. Understand that loop and most of AI's strengths, weaknesses and costs stop being mysterious.

1. The idea

Your phone keyboard suggests the next word as you type. Now imagine a keyboard that has read a large part of everything ever written, suggests the next piece of text with uncanny skill, and is allowed to keep accepting its own suggestion. That's a large language model.

A large language model (LLM) is a program that does one thing: given some text, it predicts what comes next. It doesn't look up answers in a database. It writes the most likely continuation, one small piece at a time. Those small pieces are called tokens.

  1. Text becomes tokens. A tokenizer chops your text into tokens: whole common words, parts of rarer words, punctuation, spaces. Each token has a number (its ID).
  2. The model scores every possible next token. It looks at all the tokens so far and gives every token in its vocabulary (tens or hundreds of thousands of them) a probability of coming next.
  3. One token is picked. Usually a likely one, with a little randomness (more on that in section 3).
  4. It's added to the text, and the loop repeats until the model produces a special "end" token or hits a length limit.

That's genuinely all that happens when an AI "answers" you. The intelligence lives in how good those predictions are, which comes from training (Module 03).

2. How it works

The generation loop. The model runs once per new token, so a 500-token answer means about 500 trips around this loop.

Press Play to watch a model finish the sentence "The capital of France is", one token at a time. The probabilities are illustrative, but the steps are exactly what happens inside every LLM.

Why answers appear word by wordEach token needs its own trip through the model, so the app shows tokens as they arrive. That's called streaming. It also explains why a long answer takes longer than a short one: output is produced one token at a time, while your input is read all at once.

3. Try it yourself

See text turn into tokens

Type anything, or pick an example. Notice how common English words are one token each, while rare words, numbers, code symbols and non-English scripts break into more pieces.

This playground is simplifiedReal tokenizers learn their pieces from data (a method called byte-pair encoding), and every model family has its own, so the same text gives a different count in Claude, Gemini and GPT. For real numbers use the vendors' counters: Anthropic's token counting API, Gemini's countTokens, and OpenAI's tokenizer page or the tiktoken library.

Rule of thumb: in English, one token is roughly half to three-quarters of a word, depending on the tokenizer. Anthropic's current tokenizer fits about 555,000 English words into 1 million tokens; older tokenizers fitted about 750,000. Code, numbers and most non-English languages use noticeably more tokens for the same meaning.

Temperature: how adventurous the next pick is

The model gives probabilities; something still has to pick. Always taking the top token makes text repetitive, so apps sample: likely tokens are picked most often, less likely ones sometimes. Temperature controls how much: near 0 almost always takes the top token, higher values flatten the odds. Drag the slider, then sample a few times.

1.0

Prompt: My favourite fruit is → next token?

This is why the same question can give different answers on different runs, and why you can't judge an AI feature from one try (Module 17 covers testing properly). Many current reasoning models fix the temperature for you; there you steer behaviour with the prompt and the effort setting instead.

4. Why it matters at work

Everything is counted in tokens

Prices are per million tokens, limits are in tokens, and speed is tokens per second. Output tokens usually cost several times more than input tokens, because each one needs its own pass through the model.

Plausible isn't the same as true

The model writes the most likely continuation. For well-known facts that's usually right; for rare facts, exact numbers, versions and citations it can be confidently wrong. Module 03 shows how to protect yourself.

The same prompt, different answers

Sampling makes outputs vary. Judge a prompt or feature on a set of examples (an eval), never on one lucky or unlucky run.

It sees tokens, not letters

Character-level tasks (counting letters, exact spelling, reversing strings) used to trip models up because "strawberry" might be only two or three tokens. Reasoning models handle these better, but for exact text work, ask the model to use a tool or write code.

5. In your five tools

Each tool hides the token loop, but each lets you choose the model and see how many tokens you've used. Commands checked against each tool's documentation, October 2026.

ToolChoose the modelSee token usage
Claude Code/model/cost (also /usage) for the session; /context for what's loaded
Antigravity CLI/model/usage (quota used) and /credits; /context for what's loaded
GitHub Copilot (VS Code)Model picker in the chat inputThe context-size indicator next to the chat input; click it for a token breakdown
CursorModel picker (Auto, or a specific model)The context percentage in the chat; hover it for the token count
Codex CLI/model (then pick the reasoning effort)/status (session settings and token usage)

Building your own feature? The APIs return exact token counts with every response (usage fields), and each vendor has a token-counting endpoint for checking a prompt before you send it.

6. Remember it

Memory hook: "A very well-read autocomplete"It predicts the next token, adds it, and predicts again. Brilliant at plausible, not a database of truth.
How this hook works: what “A very well-read autocomplete” means

What the slogan means, phrase by phrase

“autocomplete”
Like your phone keyboard, the model only ever continues text. Chat, code and agents are all continuation with a clever setup around it.
“very well-read”
It learned from an enormous amount of text, so its guesses about what comes next are remarkably good: good enough to write code, explain ideas and follow instructions.
“A … autocomplete” (one guess at a time)
It writes one token, then decides the next given everything so far. It never plans the whole answer up front unless it writes the plan out (that's what reasoning models do, Module 04).

Why this picture fits: it explains the good (fluent, fast, knows a lot) and the bad (can be confidently wrong, varies between runs, priced per token) with one everyday object everyone already understands.

The story → the concept

In the storyIn the model
The keyboard's next-word suggestionsProbabilities for every possible next token
Tapping a suggestionSampling one token (temperature decides how adventurous)
What it has "read"Training data, stored in the model's weights
Words and word pieces it suggestsTokens from its tokenizer's vocabulary
Tapping until the sentence is doneThe loop that runs until an end token or a length limit

Watch it on an example

You ask: "Which version of our internal billing library added refunds?"

  1. The autocomplete has never read your internal library: it isn't in its training data.
  2. It still continues the text in the most plausible way: "Refunds were added in version 2.3, in the refund() method…"
  3. That sentence is fluent, specific and invented. An autocomplete doesn't know what it doesn't know.
  4. Give it the changelog (paste it, or let an agent read the repo) and the same autocomplete continues from real text, and gets it right.

The trap it saves you from

Treating fluent output as looked-up fact. "Well-read autocomplete" reminds you to ask: has it actually seen the information this answer needs? If not, provide it.

It still works when…

You use agents (Module 07). An agent is the same autocomplete whose continuation happens to be "call this tool with these arguments". The program runs the tool and pastes the result back, and the autocomplete continues from there.

7. Common mistakes

Trusting specific facts without a sourceVersions, API names, numbers, quotes and citations are where plausible-but-wrong shows up most. Give the model the source, or verify.
Counting words instead of tokensBudgets, limits and prices are in tokens. Code, logs, JSON and non-English text can use far more tokens than their word count suggests.
Judging from one runOne great (or terrible) answer tells you little. Try a handful of inputs, several times, before deciding a prompt works.
Comparing vendors by price per token aloneDifferent tokenizers split the same text into different numbers of tokens. Compare the cost of the same task, not the price list.

8. Hands-on lab (20 minutes)

  1. Count real tokens. Paste one paragraph of English and the same paragraph in another language you know into OpenAI's tokenizer page or Anthropic's token counter. Note both counts.
  2. Same question, three runs, two tools. Ask "Suggest a name for an inventory app" three times in any two of the five tools. Write down how much the answers vary.
  3. Catch plausible invention. Ask one tool about a function that doesn't exist, e.g. "How do I use list.fastsort() in Python?" Note whether it invents an answer or says it doesn't exist.
  4. Watch your usage. In a CLI tool run its usage command (/cost, /usage or /status) after a few questions. Which used more tokens, your questions or the answers?
  5. Log it. Three bullets in your learning log: tokens, sampling, plausible-not-true.

9. Self-check quiz

What does an LLM actually do when it answers?

Everything is generated one token at a time, each based on all the tokens before it.

The same sentence in Telugu versus English: which usually uses more tokens?

Tokenizers are built mostly from English-heavy data, so other scripts break into more, smaller pieces. That means higher cost and less fits in the window.

Why can the same prompt give different answers?

Your chats don't retrain the model. The variation comes from sampling the next token from probabilities.

Which are usually more expensive per token?

Input is processed in one pass; every output token needs its own pass through the model, so output is priced several times higher.

10. Go deeper

ResourceTimeWhy
Large Language Models explained briefly (3Blue1Brown)8 minThe best short visual introduction.
Intro to Large Language Models (Andrej Karpathy)1 hourThe classic mental model of what an LLM is, from one of its builders.
Deep Dive into LLMs like ChatGPT (Karpathy), tokenization section20 min of 3.5 hWatches real tokenizers at work.
Transformers, the tech behind LLMs (3Blue1Brown)27 minOptional: what happens inside the "model" box.