At each step a language model produces a probability distribution over its whole vocabulary. Sampling parameters decide how a token gets chosen from it, and the three common ones work in genuinely different ways that get conflated constantly.
What changed in 2026
- Reasoning models restricted sampling. Several current models removed temperature and top-p entirely, replacing them with effort-style controls — passing them returns an error.
- Structured outputs displaced sampling tricks. Constrained decoding handles format reliability better than any sampling configuration.
- Defaults improved. Provider defaults became sensible enough that most applications should leave them alone.
- Repetition penalties fell out of favour. Recognised as treating a symptom whose cause is usually elsewhere.
The three mechanisms
Temperature rescales the distribution before sampling. Below one it sharpens — likely tokens become more likely, unlikely ones less. Above one it flattens, making unusual tokens more reachable. It changes the shape and never removes anything from consideration.
Top-k truncates to the k most likely tokens and samples among them. Everything else has zero probability. Fixed count regardless of how the probabilities are distributed.
Top-p (nucleus sampling) truncates to the smallest set of tokens whose probabilities sum to p. Adaptive: where the model is confident, that may be two tokens; where it is uncertain, it may be fifty.
|
Mechanism |
Adapts to confidence |
| Temperature |
Rescales all probabilities |
No |
| Top-k |
Keeps a fixed count |
No |
| Top-p |
Keeps a dynamic set |
Yes |
Top-p is generally preferred over top-k precisely because it adapts. A fixed k that is right when the model is confident is far too permissive when it is uncertain.
Do not stack them
Applying temperature, top-p and top-k together is common in copied configurations and produces behaviour nobody can reason about. Each transforms the distribution, and they compose in ways that are not intuitive — a top-p value means something different after temperature has reshaped the distribution it is measuring.
Pick one truncation method — top-p — and adjust temperature separately if needed. Leaving top-k unset is usually correct.
The practical guidance is narrower than the configuration surface suggests. Most applications should use provider defaults. Lower temperature suits tasks with one correct answer — classification, extraction, structured output. Higher temperature suits genuine creative variation. That is close to the whole decision.
Low temperature is not determinism
Setting temperature to zero makes token selection deterministic — always take the highest-probability option. It does not make the system reproducible, because the probabilities themselves vary slightly between runs due to floating-point non-associativity and batch composition.
This surprises people building tests on the assumption that temperature zero gives identical output. It does not, and building an exact-match test suite on that assumption produces persistent flakiness — see LLM determinism.
Repetition penalties treat symptoms
When a model loops — repeating a phrase or getting stuck in a cycle — the available fix looks like a repetition penalty that reduces the probability of already-used tokens.
It suppresses the symptom and distorts legitimate output. Text naturally repeats words, and penalising that produces awkward phrasing and, in structured output, can break syntax by penalising necessary repeated tokens like braces and quotes.
Persistent looping usually indicates something upstream: a prompt that does not clearly specify an endpoint, a context that has become confused, or a missing stop condition. Fixing the cause works better than penalising the output — see stop sequences for the endpoint side.
Common mistakes
- Stacking all three parameters. Opaque compound behaviour.
- Copying settings from an unrelated example. They are task-specific.
- Expecting reproducibility at temperature zero. Removes sampling randomness only.
- Repetition penalties on structured output. Can break syntax.
- Tuning sampling to fix prompt problems. Wrong layer.
- Passing sampling parameters to models that reject them. Some current models return an error.
- High temperature for factual tasks. Increases the chance of an unlikely, wrong token.
FAQ
What temperature should I use?
The default, unless you have a specific reason. Lower for single-correct-answer tasks, higher for genuine creative variation. Most applications never need to change it.
Does lower temperature reduce hallucination?
Somewhat and not reliably. It makes the model pick its most likely token, which is not the same as its most correct one. A confidently wrong model at temperature zero is confidently wrong every time.
Should I use top-p or top-k?
Top-p, because it adapts to model confidence. Leave top-k unset.
How do I get reliable JSON?
Structured outputs with schema enforcement, not sampling parameters. That guarantees shape in a way no temperature setting can — see structured outputs.
Where to go next
For guaranteeing output format properly, read structured outputs. For controlling where generation ends, stop sequences, and for the reproducibility question, LLM determinism.