{"post":{"slug":"ttft-vs-tpot-2026","title":"TTFT vs TPOT: The Two Latency Numbers That Matter","excerpt":"A single average latency figure hides everything useful about an LLM system. Splitting it into time to first token and time per output token tells you what to fix.","category":"ai","date":"January 19, 2026","author":"ByteLedger Team","content":"<span class=\"dropcap\">Y</span>our dashboard says average response latency is 4.2 seconds. That number is nearly useless, because it combines two things with different causes, different fixes, and very different effects on how the product feels.\n\nSplit it. **Time to first token** is how long the user waits before anything appears. **Time per output token** is how fast text arrives once it starts. A system with a 3-second TTFT and a fast stream feels completely different from one with a 0.3-second TTFT and a slow crawl, even at identical totals.\n\n## What changed in 2026\n\n- **Streaming became universal for user-facing output.** Once responses stream, the two metrics diverge in user experience and measuring them separately became necessary.\n- **Agent workloads shifted the emphasis.** Backend agent loops care about total completion time; interactive chat cares about TTFT. The right target became workload-dependent.\n- **Prefix caching made TTFT highly variable.** A cache hit produces a dramatically lower TTFT than a miss, which turned the metric bimodal.\n- **Observability tooling caught up.** Emitting both metrics per request became standard in LLM tracing rather than something you built yourself.\n\n## What drives each\n\n| | TTFT | TPOT |\n| --- | --- | --- |\n| Primary driver | Prompt length | Model size |\n| Also affected by | Cache hits, queueing, network | Batch size, memory bandwidth |\n| Phase | Prefill | Decode |\n| Improved by | Shorter prompts, prefix caching | Smaller model, quantisation, speculative decoding |\n| User perception | Highly noticeable | Only below reading speed |\n| Typical variance | High — cache dependent | Low |\n\n**TTFT** is essentially prefill time plus whatever queueing occurred. Long prompts mean long prefill. A [prefix cache](/blog/prefix-caching-explained-2026) hit removes most of it, which is why TTFT distributions often have two clear modes rather than a single peak — averaging across them is particularly misleading.\n\n**TPOT** is dominated by how fast weights can be read from memory for each generated token. It is remarkably stable per model, and it degrades under batch pressure when many sequences compete.\n\n## Users notice TTFT\n\nThe asymmetry matters for where to spend effort.\n\nTime to first token is dead air. Nothing is on screen, the user does not know whether anything is happening, and it maps directly onto perceived responsiveness. Reducing it from three seconds to one is transformative.\n\nTime per output token only matters relative to reading speed. Once text arrives faster than a person reads, further improvement is invisible — the user is the bottleneck, not the model. Pushing TPOT well below that threshold on a chat interface is optimisation nobody experiences.\n\nThe inversion happens on non-interactive work. An agent loop consuming output programmatically does not read; it waits for completion. There total time matters, TPOT multiplied by a long output dominates it, and TTFT is a rounding error. Optimising a backend agent for TTFT is the mirror-image mistake.\n\nSo the target follows the consumer: human reading it, optimise TTFT; machine consuming it, optimise total time.\n\n## Measuring properly\n\n**Emit both per request**, alongside prompt length, output length, and whether the cache hit. Without those covariates the numbers are hard to act on.\n\n**Report percentiles, not averages.** TTFT in particular is bimodal from caching, and a mean falls in a gap where no actual request lives.\n\n**Separate queueing from processing.** A rising TTFT caused by queue depth is a capacity problem; one caused by longer prompts is a prompt problem. They look identical in a single number — see [AI capacity planning](/blog/ai-capacity-planning-2026).\n\n**Track TPOT under load specifically.** It is stable at low concurrency and degrades as batches fill, so a figure measured on an idle system will not predict production.\n\n## Common mistakes\n\n- **Reporting one average latency.** Combines two unrelated distributions.\n- **Optimising TPOT below reading speed for chat.** Invisible improvement.\n- **Optimising TTFT for a backend agent.** Wrong metric for the consumer.\n- **Averaging a bimodal TTFT.** The mean describes no real request.\n- **Not recording cache hits alongside.** Removes the main explanatory variable.\n- **Benchmarking TPOT unloaded.** Understates production behaviour.\n\n## FAQ\n\n**What are good targets?**\nThey depend entirely on the interface and the model, which is why generic figures mislead. The useful discipline is to establish your own baseline, then set targets from what users actually complain about rather than from a published number.\n\n**Why is my TTFT so variable?**\nAlmost always prefix cache hits versus misses, with queueing second. Recording the cache flag per request resolves this immediately.\n\n**Can I reduce TPOT?**\nA smaller or quantised model, and speculative decoding, are the main levers. All trade something — quality, complexity, or memory. See [quantization explained](/blog/quantization-explained-2026).\n\n**Does streaming reduce total time?**\nNo, it changes when output becomes visible. Total generation time is unchanged; perceived latency improves enormously.\n\n## Where to go next\n\nFor reducing TTFT directly, read [prefix caching](/blog/prefix-caching-explained-2026). For the architecture that isolates the two phases, [disaggregated serving](/blog/disaggregated-serving-2026), and for setting targets against capacity, [LLM latency budgets](/blog/llm-latency-budgets-2026).","hue":219,"metaTitle":"TTFT vs TPOT: The Two LLM Latency Numbers","metaDescription":"Why LLM latency splits into time to first token and time per output token, what drives each, and which one users actually notice.","keywords":["ttft vs tpot","time to first token","time per output token","llm latency metrics","streaming latency","inference performance","user perceived latency","ai observability 2026"],"tldr":["**TTFT is dominated by prompt length; TPOT by model size and batch pressure** — different causes, different fixes.","**Users notice TTFT far more** — it is the wait before anything appears.","**TPOT only needs to beat reading speed** — faster than that is invisible to a human.","**Total latency is a derived number, not a measurement** — TTFT plus TPOT times output length.","**Skip:** optimising average latency. The average of two unrelated distributions tells you nothing actionable."],"readTime":"4 min read","wordCount":830},"related":[{"slug":"streaming-responses-llm-2026","title":"Streaming LLM Responses: Perceived Speed Is the Product","excerpt":"Streaming does not make generation faster, it makes waiting tolerable. Here is how to implement it well, what breaks with structured output, and how to handle failure mid-stream.","category":"ai","date":"July 8, 2026","author":"ByteLedger Team","readTime":"3 min read","hue":209},{"slug":"llm-latency-budgets-2026","title":"LLM Latency Budgets: Allocating the Time You Have","excerpt":"A two-second target is not a model requirement. It is a budget to divide across retrieval, prefill, generation, and everything else — and the model is often not the largest line.","category":"ai","date":"January 3, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":197},{"slug":"ai-inference-providers-compared-2026","title":"AI Inference Providers Compared: Who Serves Open Models","excerpt":"Hosted inference providers serve the same open models at very different speeds and prices. Here is what actually differs between them and how to evaluate one.","category":"ai","date":"August 14, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":200}],"sameCategory":[{"slug":"agent-cost-attribution-2026","title":"Agent Cost Attribution: Finding Where the Tokens Went","excerpt":"A single monthly API bill tells you that agents are expensive. It does not tell you which feature, which customer, or which retry loop is responsible. That gap is fixable.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":220},{"slug":"human-in-the-loop-agents-2026","title":"Human-in-the-Loop Agents: Where to Put the Approval Gate","excerpt":"Approve everything and the agent is a slow autocomplete. Approve nothing and you have handed irreversible actions to a system that can be talked into things.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":197},{"slug":"reasoning-effort-controls-2026","title":"Reasoning Effort Controls: Tuning How Hard a Model Thinks","excerpt":"Fixed thinking budgets gave way to effort levels, and the default is rarely the right setting. Here is how the dial behaves and where each level actually belongs.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":210},{"slug":"retrieval-metrics-explained-2026","title":"Retrieval Metrics: Measuring the Half of RAG That Fails","excerpt":"When a RAG system answers badly, the retrieval usually failed before the model ever spoke. Measuring generation quality alone hides that entirely.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":215}],"newer":{"slug":"treasury-bills-vs-cds-2026","title":"Treasury bills vs CDs in 2026: which one wins for cash savings","excerpt":"T-bills vs CDs in 2026 — which actually pays more after taxes, where to buy each, and when one beats the other.","category":"finance","date":"January 19, 2026","author":"ByteLedger Team","readTime":"3 min read","hue":290},"older":{"slug":"best-crypto-exchanges-2026","title":"Best crypto exchanges in 2026: fees, security, and what survived the shakeout","excerpt":"The crypto exchanges actually worth using in 2026, with honest notes on fees, custody, regulation, and the platforms to stay away from.","category":"finance","date":"January 18, 2026","author":"ByteLedger Team","readTime":"3 min read","hue":295}}