Inside the transformer

The tokenizer hands the model a list of numbers (Module 01a). What happens next produces one new token, every time, and then starts again. This module opens the box: what parameters and weights are, how a token ID becomes a vector, how attention lets each token look at the others, how the model picks the next token, how many times it loops, and how the weights are stored and served. The example throughout is OpenAI's open-weight gpt-oss-120b, because its design is published, so every number here can be checked.

1. The idea

Parameters are those knobs: the numbers the model learned in training. People say weights for the same thing (strictly, weights are the multipliers inside the parameter matrices, and biases are the smaller rest). Hyperparameters are design choices fixed before training: how many layers, how wide each vector is, how many attention heads. gpt-oss-120b, from its model card:

The design is the transformer, introduced in 2017 in "Attention Is All You Need" by Vaswani and colleagues at Google: a network "based solely on attention mechanisms", which is what lets it learn from huge amounts of text in parallel.

2. How it works

Attention: each token looks back at the others

Attention is how a token gathers meaning from the tokens before it. To understand "it" in "Where is my order A-1042? It was due yesterday", the model has to connect "it" to "order".

Mixture of experts: only part of the model runs for each token

The generation loop: how far does it go?

A common belief is that the model searches through possible answers for the best one. It doesn't. It runs one forward pass per output token, picks one token, appends it, and runs again, until it produces a stop token or hits a length limit. A 300-token answer is 300 passes. Step through it:

3. Try it yourself

How the next token is picked

After the last layer, the model gives every token in the vocabulary a score (a logit). Softmax turns the scores into probabilities. Temperature sharpens or flattens them; top-p keeps only the most likely tokens that together reach probability p. These five scores are made up, for the text "Your order is".

How much memory does a model need?

Weights take parameters × bytes per parameter. On top of that, every token in every active conversation needs a KV cache entry in each layer (the keys and values attention keeps, so earlier tokens aren't recomputed). Pick a model and precision.

4. Why it matters at work

How weights are stored and served

1. On disk: weights are saved as named tensors (matrices) in safetensors files, a format designed to be safe to load (unlike Python pickle, it can't run code) and fast (zero-copy). gpt-oss stores its expert weights in MXFP4 and the rest in BF16 (16-bit).
2. Loaded onto accelerators: GPUs (or TPUs) hold the weights in their memory, because every token needs every active weight multiplied, and fast.
3. Served: an inference server batches many users' requests together and manages the KV cache. vLLM's PagedAttention stores the cache in pages, like an operating system's virtual memory, and its authors report 2–4× higher throughput than earlier systems.

The sampling step in code

import numpy as np

def sample_next(logits, temperature=1.0, top_p=1.0, rng=np.random.default_rng()):
    # Pick the next token ID from the model's scores for every token in the vocabulary.
    if temperature == 0:                       # greedy: always the highest score
        return int(np.argmax(logits))
    z = logits / temperature                   # <1 sharpens, >1 flattens
    p = np.exp(z - z.max()); p /= p.sum()      # softmax: scores -> probabilities
    order = np.argsort(p)[::-1]                # most likely first
    keep = np.cumsum(p[order]) - p[order] < top_p   # smallest set reaching top_p
    p_kept = np.zeros_like(p); p_kept[order[keep]] = p[order[keep]]
    return int(rng.choice(len(p), p=p_kept / p_kept.sum()))

5. The tools in this flow

Checked 1 November 2026. The gpt-oss repository lists the ways to run it.

6. Remember it

7. Common mistakes

8. Hands-on lab (roadmap week 1)

1. Read a real config: open config.json for an open model on Hugging Face and find the number of layers, hidden size, attention heads and vocabulary size.
2. Do the arithmetic: compute its embedding table size and its weight memory in BF16, and compare with the calculator above.
3. Run a model locally: run gpt-oss-20b (or another small open model) with Ollama or LM Studio. Measure time to the first token and tokens per second.
4. Feel prefill vs decode: send a long prompt with a short answer, then a short prompt with a long answer. Which takes longer?
5. Write the sampler: run the sample_next function above on made-up logits at temperatures 0.2, 1 and 1.5 and count the picks.

9. Self-check quiz

10. Go deeper