Two services each handle fifty requests per second. One takes 200-token prompts and produces 50-token replies. The other takes 20,000-token documents and produces 2,000-token summaries. Their capacity requirements differ by roughly two orders of magnitude.
Request rate is not a capacity unit for LLM serving. Tokens are — and even that needs splitting, because input and output tokens consume different resources.
What changed in 2026
- Token-based planning became standard. Sizing on request rate was widely recognised as producing wrong answers.
- Cache memory emerged as the usual bottleneck. Teams found concurrency capped by KV cache space long before compute saturated — see paged attention.
- Prompt lengths grew faster than output lengths. Retrieval, tool definitions, and long system prompts shifted the input-to-output ratio toward prefill.
- Autoscaling on the wrong signal became a documented failure. Scaling on request count or CPU rather than token throughput and queue depth misbehaved predictably.
The units that matter
| Metric |
What it tells you |
| Requests per second |
Almost nothing on its own |
| Input tokens per second |
Prefill compute demand |
| Output tokens per second |
Decode bandwidth demand |
| Concurrent sequences |
KV cache memory demand |
| Average context length |
Cache memory per sequence |
| Queue depth |
Whether you are already short |
Model the workload as a distribution rather than an average. Prompt lengths in particular are usually heavily skewed — most requests modest, a minority enormous — and the tail sizes the system because those long prompts consume cache space and prefill compute disproportionately.
Two resources, two limits
Prefill is compute-bound: processing a long prompt is a large parallel operation that uses the arithmetic units heavily.
Decode is memory-bandwidth-bound: each generated token requires reading the weights, and batching many sequences amortises that read across all of them.
The practical consequence is that a workload heavy on long prompts and short answers is limited by compute, while one with short prompts and long answers is limited by bandwidth. Sizing hardware without knowing which you have produces a system constrained by the wrong thing — which is also the argument for disaggregated serving at scale.
Cache memory caps concurrency
The finding that surprises people: the limit on how many sequences run concurrently is usually GPU memory for the KV cache, not compute.
Each in-flight sequence holds cache proportional to its context length. Long contexts mean fewer concurrent sequences on the same hardware, regardless of how much compute is available. It is entirely normal to see a GPU far from compute saturation while the scheduler refuses new work for lack of cache space.
Which makes context length a capacity parameter, not just a product decision. Doubling typical prompt length roughly halves achievable concurrency. Teams that add retrieval or expand a system prompt frequently see throughput fall for reasons nobody connects to the change.
Levers on this axis: cache quantisation, shorter prompts, and prefix sharing all raise effective concurrency without adding hardware.
Queueing arrives early
Utilisation is a misleading health signal here. Latency degrades non-linearly as a system approaches capacity, so queue depth rises and response times climb while utilisation still looks acceptable.
Plan for headroom rather than saturation. The target utilisation that keeps latency acceptable is usually well below full, and finding it means load testing to the point where the latency percentile you care about breaks — not to the point where throughput stops rising.
Autoscale on queue depth and token throughput rather than request count or CPU. CPU utilisation on a GPU-bound service is close to meaningless, and request count ignores the token shape entirely — see AI inference autoscaling.
Common mistakes
- Planning on requests per second. Wrong unit.
- Averages instead of distributions. The tail sizes the system.
- Ignoring context length as a capacity input. Longer prompts mean lower concurrency.
- Targeting high utilisation. Latency breaks first.
- Autoscaling on CPU. Meaningless for GPU-bound serving.
- Load testing with uniform requests. Real traffic is skewed, and skew is what hurts.
- Not accounting for evaluation and background traffic. Judge models and batch jobs consume the same capacity.
FAQ
How do I forecast growth?
In tokens, and separately for input and output, because their ratio can shift independently of user growth. A product change that adds retrieval increases input tokens without adding a single user.
What headroom should I keep?
Enough that your latency percentile stays within budget at peak, which you find by load testing rather than by rule of thumb. Systems with spiky traffic need more than steady ones.
Should I use reserved or on-demand capacity?
Depends on how predictable your baseline is. A stable baseline with spiky peaks argues for reserved capacity plus on-demand overflow — see reserved GPU capacity.
Does batching change the plan?
Substantially — it is what makes decode efficient. Capacity numbers measured without realistic batching will not predict production.
Where to go next
For the scaling mechanics, read AI inference autoscaling. For the memory limit that caps concurrency, paged attention, and for the latency targets capacity must support, LLM latency budgets.