Tokenizers: how text becomes numbers

A language model never sees letters or words. Before your prompt reaches it, a tokenizer cuts the text into pieces called tokens and swaps each piece for a number. Module 01 showed what tokens are; this module shows how a tokenizer is built, what it stores, how it turns your prompt into numbers, and the special tokens that turn a conversation into one long sequence.

1. The idea

The whole journey from text to the model, in four steps:

1. Normalise and split. The text is split into rough chunks (words, spaces, punctuation, digits) by a fixed pattern, so pieces never cross, say, a word and the punctuation after it.
2. Apply the merges. Each chunk starts as single bytes; the tokenizer joins neighbouring pieces using merge rules it learned in training, in the order it learned them.
3. Look up the IDs. Each final piece is replaced by its number in the vocabulary, for example " order" → one ID.
4. Hand over to the model. The model turns each ID into a vector of numbers by looking up one row of its embedding table (Module 01b).

2. How it works

Training a BPE tokenizer

Training happens once, before the model itself is trained, on a large sample of the text the model will learn from. This tiny version uses the Hugging Face course's example corpus: five words and how often each appears.

Real tokenizers do exactly this on billions of words, and stop when the vocabulary reaches its target size: about 200,000 for recent OpenAI models, 128K for Llama 3. Most modern tokenizers use byte-level BPE: the starting vocabulary is the 256 possible byte values, not letters, so any text in any language, emoji or file can be encoded and nothing is ever "unknown". Frequent English words end up as one token; rare words, names and many non-English words take several.

Tokenizing your text

When you send a prompt, nothing is learned: the tokenizer splits the text, then applies the stored merges in the order they were learned.

What a tokenizer looks like on disk

A tokenizer is a small set of files that ships with the model weights. The model's weights only make sense with the exact tokenizer they were trained with.

Special tokens and chat templates

A chat is not sent as a list of messages. It is flattened into one sequence by a chat template, which wraps each message in special tokens. Here is the real format of OpenAI's open-weight gpt-oss models, called harmony, with the documented token IDs:

3. Try it yourself: train a tokenizer

Type a few words with how often they appear (word:count) or just a sentence. Press Merge next pair to run one training step, or Train to a vocabulary size. Then type a word to tokenize with the merges you've learned.

4. Why it matters at work

See it in code

import tiktoken                       # OpenAI's open-source BPE tokenizer

enc = tiktoken.get_encoding("o200k_base")
ids = enc.encode("Where is my order A-1042?")
print(ids)                            # a short list of integers
print([enc.decode([i]) for i in ids]) # the text of each token
assert enc.decode(ids) == "Where is my order A-1042?"   # lossless round trip

enc = tiktoken.encoding_for_model("gpt-4o")   # the encoding a model uses

5. Tokenizers you'll meet

Checked 1 November 2026.

6. Remember it

7. Common mistakes

8. Hands-on lab (roadmap week 1)

1. Count tokens for one paragraph with tiktoken's cl100k_base and o200k_base. Which is smaller, and why might that be?
2. Try other languages: translate the paragraph into two languages you know and compare token counts.
3. Inspect the pieces: print each token's text for a sentence with a long number, an email address and some code.
4. Train a tiny tokenizer with the Hugging Face tokenizers library on a few pages of your own documents, and look at its first 20 merges.
5. See a chat template: with Transformers, call apply_chat_template(messages, tokenize=False) for an open model and read the special tokens it adds.

9. Self-check quiz

10. Go deeper