{"post":{"slug":"sampling-parameters-explained-2026","title":"Sampling Parameters: Temperature, Top-p and What They Do","excerpt":"Temperature, top-p and top-k all reshape the same probability distribution in different ways. Using several at once compounds their effects in ways that are hard to predict.","category":"ai","date":"January 9, 2026","author":"ByteLedger Team","content":"<span class=\"dropcap\">A</span>t each step a language model produces a probability distribution over its whole vocabulary. Sampling parameters decide how a token gets chosen from it, and the three common ones work in genuinely different ways that get conflated constantly.\n\n## What changed in 2026\n\n- **Reasoning models restricted sampling.** Several current models removed temperature and top-p entirely, replacing them with effort-style controls — passing them returns an error.\n- **Structured outputs displaced sampling tricks.** Constrained decoding handles format reliability better than any sampling configuration.\n- **Defaults improved.** Provider defaults became sensible enough that most applications should leave them alone.\n- **Repetition penalties fell out of favour.** Recognised as treating a symptom whose cause is usually elsewhere.\n\n## The three mechanisms\n\n**Temperature** rescales the distribution before sampling. Below one it sharpens — likely tokens become more likely, unlikely ones less. Above one it flattens, making unusual tokens more reachable. It changes the *shape* and never removes anything from consideration.\n\n**Top-k** truncates to the k most likely tokens and samples among them. Everything else has zero probability. Fixed count regardless of how the probabilities are distributed.\n\n**Top-p** (nucleus sampling) truncates to the smallest set of tokens whose probabilities sum to p. Adaptive: where the model is confident, that may be two tokens; where it is uncertain, it may be fifty.\n\n| | Mechanism | Adapts to confidence |\n| --- | --- | --- |\n| Temperature | Rescales all probabilities | No |\n| Top-k | Keeps a fixed count | No |\n| Top-p | Keeps a dynamic set | Yes |\n\nTop-p is generally preferred over top-k precisely because it adapts. A fixed k that is right when the model is confident is far too permissive when it is uncertain.\n\n## Do not stack them\n\nApplying temperature, top-p and top-k together is common in copied configurations and produces behaviour nobody can reason about. Each transforms the distribution, and they compose in ways that are not intuitive — a top-p value means something different after temperature has reshaped the distribution it is measuring.\n\nPick one truncation method — top-p — and adjust temperature separately if needed. Leaving top-k unset is usually correct.\n\nThe practical guidance is narrower than the configuration surface suggests. Most applications should use provider defaults. Lower temperature suits tasks with one correct answer — classification, extraction, structured output. Higher temperature suits genuine creative variation. That is close to the whole decision.\n\n## Low temperature is not determinism\n\nSetting temperature to zero makes token selection deterministic — always take the highest-probability option. It does not make the system reproducible, because the probabilities themselves vary slightly between runs due to floating-point non-associativity and batch composition.\n\nThis surprises people building tests on the assumption that temperature zero gives identical output. It does not, and building an exact-match test suite on that assumption produces persistent flakiness — see [LLM determinism](/blog/llm-determinism-explained-2026).\n\n## Repetition penalties treat symptoms\n\nWhen a model loops — repeating a phrase or getting stuck in a cycle — the available fix looks like a repetition penalty that reduces the probability of already-used tokens.\n\nIt suppresses the symptom and distorts legitimate output. Text naturally repeats words, and penalising that produces awkward phrasing and, in structured output, can break syntax by penalising necessary repeated tokens like braces and quotes.\n\nPersistent looping usually indicates something upstream: a prompt that does not clearly specify an endpoint, a context that has become confused, or a missing stop condition. Fixing the cause works better than penalising the output — see [stop sequences](/blog/stop-sequences-explained-2026) for the endpoint side.\n\n## Common mistakes\n\n- **Stacking all three parameters.** Opaque compound behaviour.\n- **Copying settings from an unrelated example.** They are task-specific.\n- **Expecting reproducibility at temperature zero.** Removes sampling randomness only.\n- **Repetition penalties on structured output.** Can break syntax.\n- **Tuning sampling to fix prompt problems.** Wrong layer.\n- **Passing sampling parameters to models that reject them.** Some current models return an error.\n- **High temperature for factual tasks.** Increases the chance of an unlikely, wrong token.\n\n## FAQ\n\n**What temperature should I use?**\nThe default, unless you have a specific reason. Lower for single-correct-answer tasks, higher for genuine creative variation. Most applications never need to change it.\n\n**Does lower temperature reduce hallucination?**\nSomewhat and not reliably. It makes the model pick its most likely token, which is not the same as its most correct one. A confidently wrong model at temperature zero is confidently wrong every time.\n\n**Should I use top-p or top-k?**\nTop-p, because it adapts to model confidence. Leave top-k unset.\n\n**How do I get reliable JSON?**\nStructured outputs with schema enforcement, not sampling parameters. That guarantees shape in a way no temperature setting can — see [structured outputs](/blog/structured-outputs-llm-2026).\n\n## Where to go next\n\nFor guaranteeing output format properly, read [structured outputs](/blog/structured-outputs-llm-2026). For controlling where generation ends, [stop sequences](/blog/stop-sequences-explained-2026), and for the reproducibility question, [LLM determinism](/blog/llm-determinism-explained-2026).","hue":199,"metaTitle":"Sampling Parameters: Temperature, Top-p Explained","metaDescription":"What temperature, top-p, and top-k actually do to token probabilities, why combining truncation methods is confusing, and when sampling is not your problem.","keywords":["sampling parameters","temperature llm","top-p nucleus sampling","top-k sampling","repetition penalty","decoding strategy","llm output control","generation settings 2026"],"tldr":["**Temperature reshapes the distribution; top-p and top-k truncate it** — different mechanisms, frequently confused.","**Pick one truncation method** — combining top-p and top-k makes the effective behaviour opaque.","**Low temperature is not determinism** — it removes sampling randomness, not all variation.","**Repetition penalties fight symptoms** — persistent looping usually means a prompt or context problem.","**Skip:** tuning sampling before fixing the prompt. Most output quality problems are not sampling problems."],"readTime":"4 min read","wordCount":811},"related":[{"slug":"temperature-vs-top-p-2026","title":"Temperature vs top-p in 2026: how LLM sampling settings work","excerpt":"Temperature and top-p both control how random an LLM sounds, but they reshape the probability distribution in different ways. Here is what each one does, when to change it, and the defaults worth trusting.","category":"ai","date":"June 30, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":320},{"slug":"constrained-decoding-explained-2026","title":"Constrained Decoding: Making Model Output Always Parse","excerpt":"Structured output modes guarantee valid JSON by restricting what tokens the model may emit. Here is how that works, what it costs, and where it quietly hurts quality.","category":"ai","date":"August 11, 2026","author":"ByteLedger Team","readTime":"3 min read","hue":213},{"slug":"logit-bias-explained-2026","title":"Logit Bias: Nudging or Banning Specific Tokens","excerpt":"You can push individual tokens up or down before sampling. It is a precise instrument for a narrow set of problems, and it operates on tokens rather than words, which is where it goes wrong.","category":"ai","date":"February 15, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":210}],"sameCategory":[{"slug":"agent-cost-attribution-2026","title":"Agent Cost Attribution: Finding Where the Tokens Went","excerpt":"A single monthly API bill tells you that agents are expensive. It does not tell you which feature, which customer, or which retry loop is responsible. That gap is fixable.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":220},{"slug":"human-in-the-loop-agents-2026","title":"Human-in-the-Loop Agents: Where to Put the Approval Gate","excerpt":"Approve everything and the agent is a slow autocomplete. Approve nothing and you have handed irreversible actions to a system that can be talked into things.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":197},{"slug":"reasoning-effort-controls-2026","title":"Reasoning Effort Controls: Tuning How Hard a Model Thinks","excerpt":"Fixed thinking budgets gave way to effort levels, and the default is rarely the right setting. Here is how the dial behaves and where each level actually belongs.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":210},{"slug":"retrieval-metrics-explained-2026","title":"Retrieval Metrics: Measuring the Half of RAG That Fails","excerpt":"When a RAG system answers badly, the retrieval usually failed before the model ever spoke. Measuring generation quality alone hides that entirely.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":215}],"newer":{"slug":"rope-scaling-explained-2026","title":"RoPE Scaling: Extending Context Beyond Training Length","excerpt":"A model trained on short sequences degrades sharply past that length. Rescaling its positional encoding lets it handle longer inputs without retraining from scratch.","category":"ai","date":"January 9, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":218},"older":{"slug":"secure-deletion-explained-2026","title":"Secure Deletion: Why Deleting a File Does Not Delete It","excerpt":"Deleting a file removes the reference, not the data. On modern storage, overwriting does not work reliably either, which is why encryption changed how disposal is done.","category":"guides","date":"January 9, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":162}}