Your dashboard says average response latency is 4.2 seconds. That number is nearly useless, because it combines two things with different causes, different fixes, and very different effects on how the product feels.
Split it. Time to first token is how long the user waits before anything appears. Time per output token is how fast text arrives once it starts. A system with a 3-second TTFT and a fast stream feels completely different from one with a 0.3-second TTFT and a slow crawl, even at identical totals.
What changed in 2026
- Streaming became universal for user-facing output. Once responses stream, the two metrics diverge in user experience and measuring them separately became necessary.
- Agent workloads shifted the emphasis. Backend agent loops care about total completion time; interactive chat cares about TTFT. The right target became workload-dependent.
- Prefix caching made TTFT highly variable. A cache hit produces a dramatically lower TTFT than a miss, which turned the metric bimodal.
- Observability tooling caught up. Emitting both metrics per request became standard in LLM tracing rather than something you built yourself.
What drives each
|
TTFT |
TPOT |
| Primary driver |
Prompt length |
Model size |
| Also affected by |
Cache hits, queueing, network |
Batch size, memory bandwidth |
| Phase |
Prefill |
Decode |
| Improved by |
Shorter prompts, prefix caching |
Smaller model, quantisation, speculative decoding |
| User perception |
Highly noticeable |
Only below reading speed |
| Typical variance |
High — cache dependent |
Low |
TTFT is essentially prefill time plus whatever queueing occurred. Long prompts mean long prefill. A prefix cache hit removes most of it, which is why TTFT distributions often have two clear modes rather than a single peak — averaging across them is particularly misleading.
TPOT is dominated by how fast weights can be read from memory for each generated token. It is remarkably stable per model, and it degrades under batch pressure when many sequences compete.
Users notice TTFT
The asymmetry matters for where to spend effort.
Time to first token is dead air. Nothing is on screen, the user does not know whether anything is happening, and it maps directly onto perceived responsiveness. Reducing it from three seconds to one is transformative.
Time per output token only matters relative to reading speed. Once text arrives faster than a person reads, further improvement is invisible — the user is the bottleneck, not the model. Pushing TPOT well below that threshold on a chat interface is optimisation nobody experiences.
The inversion happens on non-interactive work. An agent loop consuming output programmatically does not read; it waits for completion. There total time matters, TPOT multiplied by a long output dominates it, and TTFT is a rounding error. Optimising a backend agent for TTFT is the mirror-image mistake.
So the target follows the consumer: human reading it, optimise TTFT; machine consuming it, optimise total time.
Measuring properly
Emit both per request, alongside prompt length, output length, and whether the cache hit. Without those covariates the numbers are hard to act on.
Report percentiles, not averages. TTFT in particular is bimodal from caching, and a mean falls in a gap where no actual request lives.
Separate queueing from processing. A rising TTFT caused by queue depth is a capacity problem; one caused by longer prompts is a prompt problem. They look identical in a single number — see AI capacity planning.
Track TPOT under load specifically. It is stable at low concurrency and degrades as batches fill, so a figure measured on an idle system will not predict production.
Common mistakes
- Reporting one average latency. Combines two unrelated distributions.
- Optimising TPOT below reading speed for chat. Invisible improvement.
- Optimising TTFT for a backend agent. Wrong metric for the consumer.
- Averaging a bimodal TTFT. The mean describes no real request.
- Not recording cache hits alongside. Removes the main explanatory variable.
- Benchmarking TPOT unloaded. Understates production behaviour.
FAQ
What are good targets?
They depend entirely on the interface and the model, which is why generic figures mislead. The useful discipline is to establish your own baseline, then set targets from what users actually complain about rather than from a published number.
Why is my TTFT so variable?
Almost always prefix cache hits versus misses, with queueing second. Recording the cache flag per request resolves this immediately.
Can I reduce TPOT?
A smaller or quantised model, and speculative decoding, are the main levers. All trade something — quality, complexity, or memory. See quantization explained.
Does streaming reduce total time?
No, it changes when output becomes visible. Total generation time is unchanged; perceived latency improves enormously.
Where to go next
For reducing TTFT directly, read prefix caching. For the architecture that isolates the two phases, disaggregated serving, and for setting targets against capacity, LLM latency budgets.