How an AI Picks Its Next Word: Top-k, Top-p and Temperature

Every word an AI writes is a weighted dice roll. This is the first 500x Explained, a one-minute explainer drawn entirely in code, and it shows the roll step by step: tokens, odds, top-k, top-p and temperature.

Tokens

The model reads your text as tokens: whole words and pieces of words. In OpenAI's tokenizers, "The cat sat on the" is five tokens: "The", " cat", " sat", " on", " the".

Odds

For the next token, the model gives a score to every token it knows. That's a big list: OpenAI's cl100k_base tokenizer has 100,277 tokens, GPT-4o's o200k_base has 200,019, and Meta says Llama 3 uses a vocabulary of 128K. The scores are turned into odds that add up to 100%.

In the video, "mat" is the favorite after "The cat sat on the", with "floor" and "couch" behind it. Those numbers are an illustration made for this video; a real model's odds will differ.

Top-k: a fixed-size shortlist

Before the roll, the list is shrunk. Top-k keeps a fixed number of the most likely tokens, say the top three, and drops everything else. It keeps three whether the model is confident or not.

Top-p: a shortlist sized by confidence

Top-p, also called nucleus sampling, keeps adding the most likely tokens until their odds add up to a set share, say 90%. When the model is sure, one or two tokens may already cover 90%, so the list is short. When the odds are spread out, it might take eight, or many more. That's the difference in one line: top-k always keeps the same number, and top-p keeps more when the model isn't sure.

Temperature

Temperature also reshapes the odds before the roll. Low temperature sharpens them, so the favorite wins almost every time. High temperature flattens them, so long shots get a real chance. Then one token is drawn, added to the text, and the whole loop runs again for the next one.

Use it

Writing code or looking up facts? Turn the temperature down. Brainstorming names or ideas? Turn it up. OpenAI's API reference backs up the direction: lower values make output more focused and deterministic, higher values more random. Most chat apps set these for you. Some APIs and playgrounds let you change them, and some models don't expose them.

So when an answer changes, that's the dice: same prompt, new roll.