{"post":{"slug":"llm-latency-budgets-2026","title":"LLM Latency Budgets: Allocating the Time You Have","excerpt":"A two-second target is not a model requirement. It is a budget to divide across retrieval, prefill, generation, and everything else — and the model is often not the largest line.","category":"ai","date":"January 3, 2026","author":"ByteLedger Team","content":"<span class=\"dropcap\">S</span>omeone sets a two-second target for an AI feature. The team immediately starts looking at the model — a smaller one, quantisation, a faster provider. Then a trace shows the model call taking 600ms and the retrieval step taking 1.4 seconds.\n\nA latency target is a budget for the whole request path. Deciding how to divide it, before optimising anything, is what makes the effort land where it matters.\n\n## What changed in 2026\n\n- **Streaming made total latency the wrong headline.** For user-facing output, time to first token became the number to budget against — see [TTFT vs TPOT](/blog/ttft-vs-tpot-2026).\n- **Agent loops broke single-request budgeting.** Multi-step runs needed per-step budgets and a total, not one number.\n- **Tracing became standard.** Per-stage spans made the actual distribution visible rather than guessed at.\n- **Prefix caching made the budget bimodal.** Cache hits and misses produce different budgets, and averaging across them misleads.\n\n## Where the time goes\n\nA typical retrieval-augmented request:\n\n| Stage | Typical share | Reducible by |\n| --- | --- | --- |\n| Input processing, auth | Small | Rarely worth it |\n| Embedding the query | Small | Smaller embedding model |\n| Vector search | Moderate | Index tuning, fewer candidates |\n| Reranking | Moderate | Fewer candidates, smaller reranker |\n| Prompt assembly | Small | Rarely worth it |\n| Model prefill | Moderate to large | Shorter prompt, prefix caching |\n| Model generation | Large for long output | Smaller model, shorter output |\n| Post-processing, validation | Small | Rarely worth it |\n\nThe distribution varies enormously by application, which is exactly why measuring precedes optimising. A common finding is that retrieval — particularly reranking a large candidate set — dominates a budget everyone assumed was spent on the model.\n\n## Sequential adds, parallel does not\n\nThe structural lever people miss. Steps that must happen in order sum their latencies. Steps that can run concurrently cost only the slowest.\n\nA pipeline that embeds the query, then searches, then reranks, then generates is fully sequential. Some of it need not be: multiple retrieval sources can run concurrently before fusion, and speculative work — starting a likely retrieval before the classification that confirms it — can overlap stages that look sequential.\n\nRestructuring for concurrency frequently buys more than optimising any single stage, and it costs no quality. It is worth exhausting before reaching for a smaller model.\n\n## Budget at a percentile\n\nAn average within budget is not a system within budget. If the mean is 1.8 seconds against a 2-second target, a substantial share of requests are over it, and those are the ones users complain about.\n\nSet the budget at a percentile — commonly p95 or p99 — and allocate accordingly. That immediately changes which stages matter, because the stages with the widest variance dominate the tail even when their averages are modest.\n\nTwo sources of variance deserve specific attention. **Cache hits versus misses** produce a bimodal distribution where the mean describes no real request. **Queueing under load** is invisible at low traffic and can dominate at peak — and it is a capacity problem rather than a latency problem, which needs a different fix entirely, per [AI capacity planning](/blog/ai-capacity-planning-2026).\n\n## Agent loops need two budgets\n\nA single-request budget does not describe a multi-step run. An agent needs a per-step budget, which bounds any individual operation, and a total budget, which bounds the run.\n\nWithout the per-step limit, one stuck call consumes the whole run. Without the total, a loop making many fast calls runs indefinitely. Both are needed, and they pair with the token-denominated equivalents in [agent token budgets](/blog/agent-token-budgets-2026) and the mechanics in [agent timeouts](/blog/agent-timeouts-explained-2026).\n\nFor agent work, decide early whether the consumer is a human watching progress or a machine waiting for completion. The former wants streaming and per-step feedback; the latter wants total time minimised and does not care about intermediate latency.\n\n## Common mistakes\n\n- **Optimising the model first.** Frequently not the largest line item.\n- **Budgeting on averages.** Hides the tail users experience.\n- **One number for a streaming interface.** TTFT is what is perceived.\n- **Ignoring the sequential structure.** Concurrency is often the cheapest gain.\n- **No per-stage tracing.** Optimising by guesswork.\n- **Same budget for cache hits and misses.** Two different systems.\n- **A single budget for an agent loop.** Needs per-step and total.\n\n## FAQ\n\n**What is a reasonable target?**\nEntirely product-dependent, which is why generic numbers mislead. The useful approach is to find where users notice — usually visible in abandonment or complaint data — and set the budget there rather than at a figure from an article.\n\n**Should I use a smaller model to hit a budget?**\nOnly after exhausting prompt length, caching, concurrency, and retrieval tuning. A smaller model costs quality permanently; the others usually cost nothing.\n\n**How do I budget for variable output length?**\nBudget time to first token, which is independent of output length, and treat total time as derived. That is the honest structure for anything streaming.\n\n**Does a latency budget conflict with a cost budget?**\nFrequently — caching helps both, while a larger model or more retrieval candidates trade one against the other. Making the conflict explicit is better than optimising one and discovering the other moved.\n\n## Where to go next\n\nFor the two metrics to budget against, read [TTFT vs TPOT](/blog/ttft-vs-tpot-2026). For the queueing side, [AI capacity planning](/blog/ai-capacity-planning-2026), and for the agent equivalents, [agent token budgets](/blog/agent-token-budgets-2026).","hue":197,"metaTitle":"LLM Latency Budgets: Allocating the Time You Have","metaDescription":"How to divide a latency target across retrieval, prefill, generation, and post-processing, and why percentiles rather than averages should set the budget.","keywords":["llm latency budget","latency allocation","rag latency","percentile latency","streaming perceived latency","performance budgeting","inference latency","ai performance 2026"],"tldr":["**Budget the whole path, not the model call** — retrieval, network, and post-processing frequently exceed inference.","**Allocate against the user-perceived metric** — for streaming interfaces that is time to first token, not total.","**Sequential steps add; parallel steps take the maximum** — restructuring for concurrency is often the cheapest win.","**Budget at a percentile, not the mean** — an average within budget hides a tail that is not.","**Skip:** optimising the model call before measuring where the time actually goes."],"readTime":"4 min read","wordCount":917},"related":[{"slug":"ttft-vs-tpot-2026","title":"TTFT vs TPOT: The Two Latency Numbers That Matter","excerpt":"A single average latency figure hides everything useful about an LLM system. Splitting it into time to first token and time per output token tells you what to fix.","category":"ai","date":"January 19, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":219},{"slug":"streaming-responses-llm-2026","title":"Streaming LLM Responses: Perceived Speed Is the Product","excerpt":"Streaming does not make generation faster, it makes waiting tolerable. Here is how to implement it well, what breaks with structured output, and how to handle failure mid-stream.","category":"ai","date":"July 8, 2026","author":"ByteLedger Team","readTime":"3 min read","hue":209},{"slug":"streaming-vs-batch-llm-responses-2026","title":"Streaming vs Batch LLM Responses in 2026: Which to Use When","excerpt":"Streaming and batch are two different ways to deliver LLM output, with different latency, cost, and engineering tradeoffs. Here is how to pick between them in 2026.","category":"ai","date":"July 7, 2026","author":"ByteLedger Team","readTime":"3 min read","hue":290}],"sameCategory":[{"slug":"agent-cost-attribution-2026","title":"Agent Cost Attribution: Finding Where the Tokens Went","excerpt":"A single monthly API bill tells you that agents are expensive. It does not tell you which feature, which customer, or which retry loop is responsible. That gap is fixable.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":220},{"slug":"human-in-the-loop-agents-2026","title":"Human-in-the-Loop Agents: Where to Put the Approval Gate","excerpt":"Approve everything and the agent is a slow autocomplete. Approve nothing and you have handed irreversible actions to a system that can be talked into things.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":197},{"slug":"reasoning-effort-controls-2026","title":"Reasoning Effort Controls: Tuning How Hard a Model Thinks","excerpt":"Fixed thinking budgets gave way to effort levels, and the default is rarely the right setting. Here is how the dial behaves and where each level actually belongs.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":210},{"slug":"retrieval-metrics-explained-2026","title":"Retrieval Metrics: Measuring the Half of RAG That Fails","excerpt":"When a RAG system answers badly, the retrieval usually failed before the model ever spoke. Measuring generation quality alone hides that entirely.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":215}],"newer":{"slug":"incremental-builds-explained-2026","title":"Incremental Builds: Not Rebuilding What Did Not Change","excerpt":"A build system that rebuilds everything on every change wastes most of its time. Incremental builds skip unchanged work, and they only help if the change detection is trustworthy.","category":"code","date":"January 3, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":318},"older":{"slug":"sequence-exhaustion-2026","title":"Sequence Exhaustion: The Outage With a Countdown","excerpt":"A 32-bit identifier column runs out at around two billion rows. It is entirely predictable, entirely preventable, and still takes systems down every year.","category":"code","date":"January 3, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":304}}