Run a language model for one user and the GPU spends most of its time idle. Generation is memory-bandwidth bound and profoundly sequential — one token, then the next, each depending on the last. The hardware capable of enormous parallel arithmetic sits mostly waiting.
Batching fixes that by generating for many sequences at once. The obvious implementation, though, runs into a problem that turns out to dominate real workloads: requests do not finish at the same time, and the naive scheme makes everyone wait for the slowest.
What changed in 2026
- Continuous batching became table stakes. Every serious serving stack ships it. The interesting engineering moved up a layer into scheduling policy and memory management.
- KV cache memory became the headline constraint. As context windows grew, the cache for concurrent sequences — not raw compute — became what caps concurrency on a given card.
- Prefill and decode got scheduled separately. The two phases have different performance characteristics, and treating them as one workload leaves throughput on the table.
- Fairness entered the conversation. Pure throughput maximisation starves long requests. Production schedulers now balance aggregate tokens per second against per-request latency.
Static vs continuous
|
Static batching |
Continuous batching |
| Scheduling unit |
The whole batch |
Each generation step |
| A finished sequence |
Occupies its slot until the batch ends |
Leaves immediately |
| New requests |
Wait for the next batch |
Join at the next step |
| GPU utilisation |
Falls as sequences finish |
Stays high |
| Latency under load |
Sawtooth — depends on batch boundaries |
Smoother |
| Best case |
Uniform output lengths |
Anything ragged |
The mechanism is easier to see with a concrete picture. Batch eight requests statically; one generates 20 tokens and another generates 2,000. From step 21 onward, seven slots are computing padding while one does real work. The GPU is busy and almost entirely unproductive.
Continuous batching — also called in-flight batching or iteration-level scheduling — makes the scheduling decision every step instead. A sequence that emits its stop token releases its slot, the scheduler pulls the next queued request in, and the batch stays full of live work.
Why memory sets the limit
The intuition that more batching is always better runs into the KV cache. Every sequence in flight holds cached attention keys and values for every token it has processed, and that memory grows with context length and with the number of concurrent sequences.
This produces the counter-intuitive behaviour operators notice first: throughput improves as you raise concurrency, then collapses. The collapse is not gradual degradation — it is the scheduler running out of cache space and starting to evict or reject. Sequences get preempted and recomputed, which is strictly wasted work, and effective throughput can end up below where it was at lower concurrency.
Two consequences follow. Long contexts reduce concurrency, because each sequence claims more cache — a workload with 100K-token prompts fits far fewer concurrent users than one with 2K-token prompts on identical hardware. And paged cache allocation matters, because allocating cache in fixed blocks rather than contiguous per-sequence reservations dramatically reduces fragmentation. KV cache explained covers the structure in more detail; the operational summary is that cache memory, not FLOPs, is what you are actually scheduling.
Prefill and decode are different workloads
Processing the prompt (prefill) is compute-bound and highly parallel — every prompt token is available at once. Generating output (decode) is memory-bandwidth-bound and strictly sequential.
Mixing them naively means a long prefill blocks decode steps for everyone else, producing latency spikes that correlate with somebody else's large prompt. Chunked prefill splits long prompts across steps so decode can interleave; separating the two phases onto different resources goes further at the cost of moving cache between them. Either way, the reason your p99 latency moves when a single user pastes a large document is usually this.
Common mistakes
- Measuring throughput without latency. Maximum tokens per second at the cost of thirty-second waits is not a working system. Track both, and track the tail.
- Setting max concurrency from GPU memory alone. Peak cache usage depends on context lengths in flight, which vary with traffic. Size for realistic distributions, not averages.
- Benchmarking with uniform prompts. Identical-length requests are the one case where static batching looks fine. Test with the ragged distribution you actually serve.
- Ignoring preemption metrics. Rising preemption means you are past the useful concurrency point and doing recomputation instead of work.
- Assuming it helps single-user latency. It does not. Continuous batching is a throughput technique; one user alone sees no benefit.
- Rebuilding it yourself. This is solved infrastructure. Use a serving stack that has it.
FAQ
Does this apply if I use a hosted API?
Your provider is running it, and you benefit without configuring anything. Your levers are the ones that affect their scheduler on your behalf: prompt size, output length limits, and how much concurrency you drive. Batch endpoints exist for exactly the throughput-over-latency trade — see batch vs streaming inference.
How much throughput does it actually add?
Entirely dependent on output-length variance. Workloads mixing one-line answers with long code generation see large gains; workloads with uniform short outputs see modest ones. Measure on your own traffic distribution — published multipliers are chosen to look impressive.
Does it hurt time-to-first-token?
It can, because a new request waits for a scheduling step and possibly for cache space. Chunked prefill and admission control are the usual mitigations. The trade is generally favourable, but it is a trade.
How does quantisation interact with this?
Quantisation shrinks weights and, depending on the scheme, the cache. Less cache per sequence means more concurrent sequences, so quantisation often improves throughput more than its raw speed numbers suggest — AI model quantization guide covers the formats.
Where to go next
For the memory structure that governs concurrency, read KV cache explained. For the wider set of serving levers, inference optimization techniques, and for the hardware side of the same question, GPU vs TPU for inference.