You set temperature to zero specifically so the model would stop being creative. You ran the same prompt twice. You got two different answers — not wildly different, but different enough that your exact-match test failed and you spent an afternoon looking for a bug in your own code.
There is no bug. Temperature zero does considerably less than its reputation suggests, and the remaining variation comes from a place most people never think about: the hardware arithmetic, and who else happened to be making a request at the same moment.
What changed in 2026
- Batching made the effect more visible. Continuous batching means your request shares a batch with whatever else arrived, and batch composition affects the arithmetic — so variability rose as serving efficiency improved.
- Seed parameters spread without solving it. More providers offer a seed. It improves consistency and stops short of a guarantee, which is a distinction the documentation states and users skip.
- Testing practice adapted. Property-based assertions became the norm for LLM systems, largely because teams gave up trying to make exact matching work.
- Reproducibility moved to the trace layer. Rather than reproducing outputs, teams started recording them — logging what was actually returned instead of expecting to regenerate it.
What temperature zero actually does
Generation works by producing a probability distribution over the next token and picking from it. Temperature scales that distribution: high temperature flattens it and makes unlikely tokens more reachable, low temperature sharpens it toward the most probable one.
At temperature zero the sampling step becomes deterministic — always take the highest-probability token. That removes the randomness people associate with LLM variation, and it is genuinely most of it.
What remains is upstream. The probabilities themselves are computed by an enormous chain of floating-point operations on a GPU, and floating-point addition is not associative: (a + b) + c can differ from a + (b + c) in the last bits. Which grouping happens depends on how work is scheduled across parallel units, which depends on batch size and composition.
Usually the difference is far too small to matter. Occasionally two tokens sit close enough in probability that a tiny numerical difference flips which one is highest. That token changes, the next token is conditioned on a different prefix, and the outputs diverge from there. One flipped coin early produces two visibly different paragraphs.
Why batching matters
| Factor |
Affects output |
Under your control |
| Temperature / top-p |
Yes, strongly |
Yes |
| Seed |
Yes, partially |
Where offered |
| Batch composition |
Yes, subtly |
No |
| Hardware generation |
Yes, subtly |
Rarely |
| Model version |
Yes, strongly |
Yes, if pinned |
| Server-side changes |
Yes |
No |
The row that surprises people is batch composition. Your request is processed alongside other requests that happened to arrive at the same time — the continuous batching that makes hosted inference affordable. Different neighbours mean a different batch shape, which means a different reduction order in the arithmetic.
This is why the same prompt can behave differently at 3am and at peak traffic, and why it is not something you can configure away. It is a consequence of the serving architecture, and the alternative — dedicating hardware per request — is not economically realistic.
What to do instead
The productive move is to stop pursuing bit-identical output and get what you actually need another way.
Test properties, not strings. Assert on the things that should hold regardless of wording: valid JSON, required fields present, under the length limit, contains the correct account number, called the right tool with the right arguments. These survive rewording, which is the whole point. LLM regression testing covers building a suite on this basis.
Log outputs rather than reproducing them. For debugging and audit, you need to know what the model returned, not to regenerate it. Store the response. That gives you everything reproducibility would have, without depending on a guarantee that does not exist.
Cache when you need identical answers. If the same question must give the same answer — a compliance requirement, a consistency guarantee to users — cache the response and serve it. This is the only reliable way to get identical output, and it is straightforward.
Pin the model version. This is the largest controllable source of variation by a wide margin. An unpinned model alias can change under you, and that changes behaviour far more than any arithmetic effect.
Common mistakes
- Assuming temperature zero means reproducible. It removes sampling randomness only.
- Exact-match assertions in tests. They flake, get muted, and stop catching real regressions.
- Treating a seed as a guarantee. It improves consistency; the documentation does not promise identity.
- Setting temperature to zero everywhere by reflex. It is not automatically better — for open-ended generation it can produce flatter, more repetitive output.
- Chasing determinism instead of logging. You almost always want the record, not the ability to regenerate.
- Not pinning the model. The dominant source of variation, and entirely controllable.
FAQ
Will a seed make output reproducible?
It improves consistency, particularly within a short window on the same infrastructure. Providers that offer it generally describe it as best-effort, because the batching and arithmetic effects sit below the level a seed controls.
Is a local model deterministic?
Closer to it, since you control batching and hardware. Running single requests on the same hardware with the same settings gets you much nearer to identical output — one reason local models are useful for research where reproducibility matters.
Does this affect structured outputs?
Constrained decoding guarantees the shape — valid JSON matching your schema — not the values. You get a reliably parseable response whose contents may still vary between runs, which is usually the guarantee you actually wanted. See structured outputs.
Should I use temperature zero for classification?
Generally yes. Low temperature suits tasks with a single correct answer. Just do not expect the residual variation to disappear, and test on properties rather than exact strings.
Where to go next
For testing systems that are non-deterministic by nature, read LLM regression testing. For the serving architecture behind the batching effect, continuous batching, and for getting reliably-shaped responses, structured outputs.