Temperature and top-p are the two knobs most people reach for when an LLM output feels too random or too robotic — and most people misuse both. They are not intensity dials that go from boring to creative. They are two different ways of reshaping the probability distribution the model samples its next token from. Once you know what each one actually does, the difference between controllable generation and superstition-driven tweaking becomes obvious.
What changed in 2026
- Providers converged on saner defaults. Most major APIs now ship with a temperature near 0.7–1.0 and top-p at 1.0, and several document explicitly that you should adjust one, not both.
- Reasoning models changed the guidance. For models that emit long internal reasoning traces, vendors increasingly recommend leaving sampling near defaults, because aggressive truncation can derail the reasoning chain mid-thought.
- Alternative samplers went mainstream. Min-p and typical sampling are now exposed in many open-weight stacks, giving practitioners options beyond the classic temperature/top-p pair.
How the sampling loop actually works
At each step, the model produces a raw score (a logit) for every token in its vocabulary. A softmax turns those logits into a probability distribution. Sampling settings reshape that distribution before a token is drawn:
- Temperature scales the logits before the softmax. Below 1.0 sharpens the distribution so the top tokens get relatively more probable; above 1.0 flattens it so long-tail tokens get a real chance. At 0, it collapses to deterministic greedy selection of the single most likely token.
- Top-p (nucleus sampling) truncates the distribution. It keeps the smallest set of tokens whose cumulative probability reaches p, renormalizes, and samples only from that set. Top-p of 0.9 discards the unlikely tail entirely.
The core distinction: temperature reweights every token, while top-p removes tokens from consideration. They interact, which is exactly why changing both at once produces results you cannot reason about cleanly.
Temperature vs top-p at a glance
| Setting |
What it changes |
Lower value |
Higher value |
| Temperature |
Sharpness of the whole distribution |
More focused, repetitive |
More varied, less coherent |
| Top-p |
How much of the tail is kept |
Fewer candidate tokens |
More candidate tokens |
| Temperature 0 |
Deterministic greedy decoding |
Same output every run |
n/a |
| Top-p 1.0 |
No truncation at all |
n/a |
Full distribution in play |
When to change which
- Extraction, classification, structured JSON: temperature 0. You want the same answer every time.
- Factual Q&A and code: low temperature (0.2–0.4), top-p left at 1.0. Precision matters more than variety.
- General assistant chat: the provider defaults are usually fine — resist the urge to tune.
- Creative writing and brainstorming: raise temperature first (toward 0.9–1.1). Only reach for top-p if you want to keep diversity while cutting genuinely bad tokens.
If you are chasing consistency, a low temperature also tends to reduce fabricated details — see hallucination mitigation in 2026 for why sampling is only part of that story.
Common pitfalls
- Tuning both knobs simultaneously. If output changes, you will not know which setting caused it. Fix one at its default and move the other.
- Assuming temperature 0 is fully reproducible. Greedy decoding is deterministic in theory, but batching, hardware, and floating-point differences can still produce small run-to-run variation across providers.
- Using high temperature to fix a weak prompt. Randomness cannot substitute for clear instructions or good examples. Improve the prompt before touching the sampler.
- Ignoring evaluation. If you tune sampling by vibes, you will overfit to a handful of examples. Measure output quality on a real test set instead.
FAQ
Is a higher temperature the same as more creativity?
Not really. Higher temperature increases variability, which sometimes reads as creativity and sometimes reads as noise. Genuine creativity comes more from the prompt and the model than from flattening the distribution.
Should I ever set both temperature and top-p away from defaults?
Rarely, and only after measuring. Because they compound, most teams get more predictable behavior by keeping top-p at 1.0 and controlling variability with temperature alone.
What does top-p of 0.9 actually discard?
It drops the least likely 10% of cumulative probability mass at each step. Those tokens are usually off-topic or ungrammatical, so trimming them cleans up output without much loss of range.
Do these settings affect factual accuracy?
Indirectly. Lower temperature makes the model stick to high-probability tokens, which are often the more grounded ones, but it does not verify facts. Sampling is a stylistic and reliability lever, not a truth filter.
Where to go next