Someone sets a two-second target for an AI feature. The team immediately starts looking at the model — a smaller one, quantisation, a faster provider. Then a trace shows the model call taking 600ms and the retrieval step taking 1.4 seconds.
A latency target is a budget for the whole request path. Deciding how to divide it, before optimising anything, is what makes the effort land where it matters.
What changed in 2026
- Streaming made total latency the wrong headline. For user-facing output, time to first token became the number to budget against — see TTFT vs TPOT.
- Agent loops broke single-request budgeting. Multi-step runs needed per-step budgets and a total, not one number.
- Tracing became standard. Per-stage spans made the actual distribution visible rather than guessed at.
- Prefix caching made the budget bimodal. Cache hits and misses produce different budgets, and averaging across them misleads.
Where the time goes
A typical retrieval-augmented request:
| Stage |
Typical share |
Reducible by |
| Input processing, auth |
Small |
Rarely worth it |
| Embedding the query |
Small |
Smaller embedding model |
| Vector search |
Moderate |
Index tuning, fewer candidates |
| Reranking |
Moderate |
Fewer candidates, smaller reranker |
| Prompt assembly |
Small |
Rarely worth it |
| Model prefill |
Moderate to large |
Shorter prompt, prefix caching |
| Model generation |
Large for long output |
Smaller model, shorter output |
| Post-processing, validation |
Small |
Rarely worth it |
The distribution varies enormously by application, which is exactly why measuring precedes optimising. A common finding is that retrieval — particularly reranking a large candidate set — dominates a budget everyone assumed was spent on the model.
Sequential adds, parallel does not
The structural lever people miss. Steps that must happen in order sum their latencies. Steps that can run concurrently cost only the slowest.
A pipeline that embeds the query, then searches, then reranks, then generates is fully sequential. Some of it need not be: multiple retrieval sources can run concurrently before fusion, and speculative work — starting a likely retrieval before the classification that confirms it — can overlap stages that look sequential.
Restructuring for concurrency frequently buys more than optimising any single stage, and it costs no quality. It is worth exhausting before reaching for a smaller model.
Budget at a percentile
An average within budget is not a system within budget. If the mean is 1.8 seconds against a 2-second target, a substantial share of requests are over it, and those are the ones users complain about.
Set the budget at a percentile — commonly p95 or p99 — and allocate accordingly. That immediately changes which stages matter, because the stages with the widest variance dominate the tail even when their averages are modest.
Two sources of variance deserve specific attention. Cache hits versus misses produce a bimodal distribution where the mean describes no real request. Queueing under load is invisible at low traffic and can dominate at peak — and it is a capacity problem rather than a latency problem, which needs a different fix entirely, per AI capacity planning.
Agent loops need two budgets
A single-request budget does not describe a multi-step run. An agent needs a per-step budget, which bounds any individual operation, and a total budget, which bounds the run.
Without the per-step limit, one stuck call consumes the whole run. Without the total, a loop making many fast calls runs indefinitely. Both are needed, and they pair with the token-denominated equivalents in agent token budgets and the mechanics in agent timeouts.
For agent work, decide early whether the consumer is a human watching progress or a machine waiting for completion. The former wants streaming and per-step feedback; the latter wants total time minimised and does not care about intermediate latency.
Common mistakes
- Optimising the model first. Frequently not the largest line item.
- Budgeting on averages. Hides the tail users experience.
- One number for a streaming interface. TTFT is what is perceived.
- Ignoring the sequential structure. Concurrency is often the cheapest gain.
- No per-stage tracing. Optimising by guesswork.
- Same budget for cache hits and misses. Two different systems.
- A single budget for an agent loop. Needs per-step and total.
FAQ
What is a reasonable target?
Entirely product-dependent, which is why generic numbers mislead. The useful approach is to find where users notice — usually visible in abandonment or complaint data — and set the budget there rather than at a figure from an article.
Should I use a smaller model to hit a budget?
Only after exhausting prompt length, caching, concurrency, and retrieval tuning. A smaller model costs quality permanently; the others usually cost nothing.
How do I budget for variable output length?
Budget time to first token, which is independent of output length, and treat total time as derived. That is the honest structure for anything streaming.
Does a latency budget conflict with a cost budget?
Frequently — caching helps both, while a larger model or more retrieval candidates trade one against the other. Making the conflict explicit is better than optimising one and discovering the other moved.
Where to go next
For the two metrics to budget against, read TTFT vs TPOT. For the queueing side, AI capacity planning, and for the agent equivalents, agent token budgets.