{"post":{"slug":"ai-capacity-planning-2026","title":"AI Capacity Planning: Sizing for Tokens, Not Requests","excerpt":"Requests per second is the wrong unit. Two workloads at identical request rates can differ tenfold in what they demand, because the token shape is what actually consumes capacity.","category":"ai","date":"January 1, 2026","author":"ByteLedger Team","content":"<span class=\"dropcap\">T</span>wo services each handle fifty requests per second. One takes 200-token prompts and produces 50-token replies. The other takes 20,000-token documents and produces 2,000-token summaries. Their capacity requirements differ by roughly two orders of magnitude.\n\nRequest rate is not a capacity unit for LLM serving. Tokens are — and even that needs splitting, because input and output tokens consume different resources.\n\n## What changed in 2026\n\n- **Token-based planning became standard.** Sizing on request rate was widely recognised as producing wrong answers.\n- **Cache memory emerged as the usual bottleneck.** Teams found concurrency capped by KV cache space long before compute saturated — see [paged attention](/blog/paged-attention-explained-2026).\n- **Prompt lengths grew faster than output lengths.** Retrieval, tool definitions, and long system prompts shifted the input-to-output ratio toward prefill.\n- **Autoscaling on the wrong signal became a documented failure.** Scaling on request count or CPU rather than token throughput and queue depth misbehaved predictably.\n\n## The units that matter\n\n| Metric | What it tells you |\n| --- | --- |\n| Requests per second | Almost nothing on its own |\n| Input tokens per second | Prefill compute demand |\n| Output tokens per second | Decode bandwidth demand |\n| Concurrent sequences | KV cache memory demand |\n| Average context length | Cache memory per sequence |\n| Queue depth | Whether you are already short |\n\nModel the workload as a distribution rather than an average. Prompt lengths in particular are usually heavily skewed — most requests modest, a minority enormous — and the tail sizes the system because those long prompts consume cache space and prefill compute disproportionately.\n\n## Two resources, two limits\n\nPrefill is compute-bound: processing a long prompt is a large parallel operation that uses the arithmetic units heavily.\n\nDecode is memory-bandwidth-bound: each generated token requires reading the weights, and batching many sequences amortises that read across all of them.\n\nThe practical consequence is that a workload heavy on long prompts and short answers is limited by compute, while one with short prompts and long answers is limited by bandwidth. Sizing hardware without knowing which you have produces a system constrained by the wrong thing — which is also the argument for [disaggregated serving](/blog/disaggregated-serving-2026) at scale.\n\n## Cache memory caps concurrency\n\nThe finding that surprises people: the limit on how many sequences run concurrently is usually GPU memory for the KV cache, not compute.\n\nEach in-flight sequence holds cache proportional to its context length. Long contexts mean fewer concurrent sequences on the same hardware, regardless of how much compute is available. It is entirely normal to see a GPU far from compute saturation while the scheduler refuses new work for lack of cache space.\n\nWhich makes context length a capacity parameter, not just a product decision. Doubling typical prompt length roughly halves achievable concurrency. Teams that add retrieval or expand a system prompt frequently see throughput fall for reasons nobody connects to the change.\n\nLevers on this axis: cache quantisation, shorter prompts, and prefix sharing all raise effective concurrency without adding hardware.\n\n## Queueing arrives early\n\nUtilisation is a misleading health signal here. Latency degrades non-linearly as a system approaches capacity, so queue depth rises and response times climb while utilisation still looks acceptable.\n\nPlan for headroom rather than saturation. The target utilisation that keeps latency acceptable is usually well below full, and finding it means load testing to the point where the latency percentile you care about breaks — not to the point where throughput stops rising.\n\nAutoscale on queue depth and token throughput rather than request count or CPU. CPU utilisation on a GPU-bound service is close to meaningless, and request count ignores the token shape entirely — see [AI inference autoscaling](/blog/ai-inference-autoscaling-2026).\n\n## Common mistakes\n\n- **Planning on requests per second.** Wrong unit.\n- **Averages instead of distributions.** The tail sizes the system.\n- **Ignoring context length as a capacity input.** Longer prompts mean lower concurrency.\n- **Targeting high utilisation.** Latency breaks first.\n- **Autoscaling on CPU.** Meaningless for GPU-bound serving.\n- **Load testing with uniform requests.** Real traffic is skewed, and skew is what hurts.\n- **Not accounting for evaluation and background traffic.** Judge models and batch jobs consume the same capacity.\n\n## FAQ\n\n**How do I forecast growth?**\nIn tokens, and separately for input and output, because their ratio can shift independently of user growth. A product change that adds retrieval increases input tokens without adding a single user.\n\n**What headroom should I keep?**\nEnough that your latency percentile stays within budget at peak, which you find by load testing rather than by rule of thumb. Systems with spiky traffic need more than steady ones.\n\n**Should I use reserved or on-demand capacity?**\nDepends on how predictable your baseline is. A stable baseline with spiky peaks argues for reserved capacity plus on-demand overflow — see [reserved GPU capacity](/blog/reserved-gpu-capacity-2026).\n\n**Does batching change the plan?**\nSubstantially — it is what makes decode efficient. Capacity numbers measured without realistic batching will not predict production.\n\n## Where to go next\n\nFor the scaling mechanics, read [AI inference autoscaling](/blog/ai-inference-autoscaling-2026). For the memory limit that caps concurrency, [paged attention](/blog/paged-attention-explained-2026), and for the latency targets capacity must support, [LLM latency budgets](/blog/llm-latency-budgets-2026).","hue":200,"metaTitle":"AI Capacity Planning: Size for Tokens, Not Requests","metaDescription":"Why request rate is the wrong capacity unit for LLM serving, how prefill and decode consume different resources, and what actually caps concurrency.","keywords":["ai capacity planning","llm throughput sizing","tokens per second capacity","gpu capacity planning","queueing latency llm","kv cache limits","inference scaling","ai infrastructure 2026"],"tldr":["**Plan in tokens, not requests** — request rate says nothing about prompt or output length.","**Prefill and decode consume different resources** — one is compute-bound, the other memory-bound.","**Concurrency is usually capped by KV cache memory** — not by compute, which is counterintuitive.","**Queueing appears before saturation** — latency degrades well before utilisation looks high.","**Skip:** capacity planning from average load. The peak and the shape at peak are what size the system."],"readTime":"4 min read","wordCount":865},"related":[{"slug":"gpu-memory-estimate-llm-2026","title":"GPU Memory for LLMs: Estimating What You Actually Need","excerpt":"Model size is only the first term. Here is how to estimate memory for inference and training, including the parts that catch people out at deployment.","category":"ai","date":"July 24, 2026","author":"ByteLedger Team","readTime":"3 min read","hue":240},{"slug":"disaggregated-serving-2026","title":"Disaggregated Serving: Separating Prefill From Decode","excerpt":"Processing a prompt and generating tokens have opposite hardware profiles. Running them on the same machine means one of them is always using the wrong resource.","category":"ai","date":"January 7, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":219},{"slug":"ai-workload-scheduling-2026","title":"AI Workload Scheduling: Keeping Expensive GPUs Busy","excerpt":"Idle accelerators are the largest waste in most AI infrastructure. Here is how scheduling, queueing, and workload mixing lift utilization without hurting latency.","category":"ai","date":"August 17, 2026","author":"ByteLedger Team","readTime":"3 min read","hue":240}],"sameCategory":[{"slug":"agent-cost-attribution-2026","title":"Agent Cost Attribution: Finding Where the Tokens Went","excerpt":"A single monthly API bill tells you that agents are expensive. It does not tell you which feature, which customer, or which retry loop is responsible. That gap is fixable.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":220},{"slug":"human-in-the-loop-agents-2026","title":"Human-in-the-Loop Agents: Where to Put the Approval Gate","excerpt":"Approve everything and the agent is a slow autocomplete. Approve nothing and you have handed irreversible actions to a system that can be talked into things.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":197},{"slug":"reasoning-effort-controls-2026","title":"Reasoning Effort Controls: Tuning How Hard a Model Thinks","excerpt":"Fixed thinking budgets gave way to effort levels, and the default is rarely the right setting. Here is how the dial behaves and where each level actually belongs.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":210},{"slug":"retrieval-metrics-explained-2026","title":"Retrieval Metrics: Measuring the Half of RAG That Fails","excerpt":"When a RAG system answers badly, the retrieval usually failed before the model ever spoke. Measuring generation quality alone hides that entirely.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":215}],"newer":{"slug":"add-stripe-subscriptions-to-ai-app-2026","title":"How to add Stripe subscriptions to your AI-generated app (2026 step-by-step)","excerpt":"Most Stripe tutorials stop at the payment form. This one covers the part that breaks in production: products, checkout sessions, webhooks, and the subscription state machine — for apps shipped from Cursor, v0, Bolt, or Lovable.","category":"guides","date":"January 1, 2026","author":"ByteLedger Team","readTime":"9 min read","hue":145},"older":{"slug":"ai-investing-strategies-2026","title":"AI investing strategies that actually work in 2026","excerpt":"Beyond the hype: how retail investors are quietly using AI to research stocks, manage risk, and outperform their old portfolios.","category":"finance","date":"January 1, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":290}}