{"post":{"slug":"continuous-batching-explained-2026","title":"Continuous Batching Explained: Why LLM Servers Got Fast","excerpt":"Static batching makes every request in a batch wait for the slowest one. Continuous batching lets finished sequences leave and new ones join mid-flight. The throughput difference is large.","category":"code","date":"August 11, 2026","author":"ByteLedger Team","content":"<span class=\"dropcap\">R</span>un a language model for one user and the GPU spends most of its time idle. Generation is memory-bandwidth bound and profoundly sequential — one token, then the next, each depending on the last. The hardware capable of enormous parallel arithmetic sits mostly waiting.\n\nBatching fixes that by generating for many sequences at once. The obvious implementation, though, runs into a problem that turns out to dominate real workloads: requests do not finish at the same time, and the naive scheme makes everyone wait for the slowest.\n\n## What changed in 2026\n\n- **Continuous batching became table stakes.** Every serious serving stack ships it. The interesting engineering moved up a layer into scheduling policy and memory management.\n- **KV cache memory became the headline constraint.** As context windows grew, the cache for concurrent sequences — not raw compute — became what caps concurrency on a given card.\n- **Prefill and decode got scheduled separately.** The two phases have different performance characteristics, and treating them as one workload leaves throughput on the table.\n- **Fairness entered the conversation.** Pure throughput maximisation starves long requests. Production schedulers now balance aggregate tokens per second against per-request latency.\n\n## Static vs continuous\n\n| | Static batching | Continuous batching |\n| --- | --- | --- |\n| Scheduling unit | The whole batch | Each generation step |\n| A finished sequence | Occupies its slot until the batch ends | Leaves immediately |\n| New requests | Wait for the next batch | Join at the next step |\n| GPU utilisation | Falls as sequences finish | Stays high |\n| Latency under load | Sawtooth — depends on batch boundaries | Smoother |\n| Best case | Uniform output lengths | Anything ragged |\n\nThe mechanism is easier to see with a concrete picture. Batch eight requests statically; one generates 20 tokens and another generates 2,000. From step 21 onward, seven slots are computing padding while one does real work. The GPU is busy and almost entirely unproductive.\n\nContinuous batching — also called in-flight batching or iteration-level scheduling — makes the scheduling decision every step instead. A sequence that emits its stop token releases its slot, the scheduler pulls the next queued request in, and the batch stays full of live work.\n\n## Why memory sets the limit\n\nThe intuition that more batching is always better runs into the KV cache. Every sequence in flight holds cached attention keys and values for every token it has processed, and that memory grows with context length and with the number of concurrent sequences.\n\nThis produces the counter-intuitive behaviour operators notice first: throughput improves as you raise concurrency, then collapses. The collapse is not gradual degradation — it is the scheduler running out of cache space and starting to evict or reject. Sequences get preempted and recomputed, which is strictly wasted work, and effective throughput can end up below where it was at lower concurrency.\n\nTwo consequences follow. Long contexts reduce concurrency, because each sequence claims more cache — a workload with 100K-token prompts fits far fewer concurrent users than one with 2K-token prompts on identical hardware. And paged cache allocation matters, because allocating cache in fixed blocks rather than contiguous per-sequence reservations dramatically reduces fragmentation. [KV cache explained](/blog/kv-cache-explained-2026) covers the structure in more detail; the operational summary is that cache memory, not FLOPs, is what you are actually scheduling.\n\n## Prefill and decode are different workloads\n\nProcessing the prompt (prefill) is compute-bound and highly parallel — every prompt token is available at once. Generating output (decode) is memory-bandwidth-bound and strictly sequential.\n\nMixing them naively means a long prefill blocks decode steps for everyone else, producing latency spikes that correlate with somebody else's large prompt. Chunked prefill splits long prompts across steps so decode can interleave; separating the two phases onto different resources goes further at the cost of moving cache between them. Either way, the reason your p99 latency moves when a single user pastes a large document is usually this.\n\n## Common mistakes\n\n- **Measuring throughput without latency.** Maximum tokens per second at the cost of thirty-second waits is not a working system. Track both, and track the tail.\n- **Setting max concurrency from GPU memory alone.** Peak cache usage depends on context lengths in flight, which vary with traffic. Size for realistic distributions, not averages.\n- **Benchmarking with uniform prompts.** Identical-length requests are the one case where static batching looks fine. Test with the ragged distribution you actually serve.\n- **Ignoring preemption metrics.** Rising preemption means you are past the useful concurrency point and doing recomputation instead of work.\n- **Assuming it helps single-user latency.** It does not. Continuous batching is a throughput technique; one user alone sees no benefit.\n- **Rebuilding it yourself.** This is solved infrastructure. Use a serving stack that has it.\n\n## FAQ\n\n**Does this apply if I use a hosted API?**\nYour provider is running it, and you benefit without configuring anything. Your levers are the ones that affect their scheduler on your behalf: prompt size, output length limits, and how much concurrency you drive. Batch endpoints exist for exactly the throughput-over-latency trade — see [batch vs streaming inference](/blog/batch-vs-streaming-inference-2026).\n\n**How much throughput does it actually add?**\nEntirely dependent on output-length variance. Workloads mixing one-line answers with long code generation see large gains; workloads with uniform short outputs see modest ones. Measure on your own traffic distribution — published multipliers are chosen to look impressive.\n\n**Does it hurt time-to-first-token?**\nIt can, because a new request waits for a scheduling step and possibly for cache space. Chunked prefill and admission control are the usual mitigations. The trade is generally favourable, but it is a trade.\n\n**How does quantisation interact with this?**\nQuantisation shrinks weights and, depending on the scheme, the cache. Less cache per sequence means more concurrent sequences, so quantisation often improves throughput more than its raw speed numbers suggest — [AI model quantization guide](/blog/ai-model-quantization-guide-2026) covers the formats.\n\n## Where to go next\n\nFor the memory structure that governs concurrency, read [KV cache explained](/blog/kv-cache-explained-2026). For the wider set of serving levers, [inference optimization techniques](/blog/inference-optimization-techniques-2026), and for the hardware side of the same question, [GPU vs TPU for inference](/blog/gpu-vs-tpu-for-inference-2026).","hue":305,"metaTitle":"Continuous Batching Explained: Faster LLM Serving","metaDescription":"How continuous batching improves LLM inference throughput over static batching, why KV cache memory sets the real limit, and when the gain is worth chasing.","keywords":["continuous batching","llm inference throughput","static vs continuous batching","iteration level scheduling","kv cache memory","llm serving optimization","inflight batching","gpu utilization llm 2026"],"tldr":["**Static batching wastes GPU on ragged output lengths** — the whole batch runs until the longest sequence finishes, and short ones idle in their slots.","**Continuous batching schedules per token, not per batch** — a finished sequence frees its slot immediately and a queued request takes it.","**The gain scales with output-length variance** — uniform outputs see little benefit; mixed chat and code workloads see a lot.","**Memory, not compute, is the usual ceiling** — the KV cache for concurrent sequences is what limits how many fit.","**Skip:** tuning this yourself if you use a hosted inference API. It is already running; your levers are prompt size and concurrency."],"readTime":"5 min read","wordCount":1033},"related":[{"slug":"inference-optimization-techniques-2026","title":"Inference Optimization Techniques for 2026: A Field Guide","excerpt":"Serving cost and latency are usually solved by stacking a handful of proven techniques, not one big trick. Here is what actually moves the needle in 2026 and the order to apply it in.","category":"code","date":"August 6, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":322},{"slug":"ai-workload-scheduling-2026","title":"AI Workload Scheduling: Keeping Expensive GPUs Busy","excerpt":"Idle accelerators are the largest waste in most AI infrastructure. Here is how scheduling, queueing, and workload mixing lift utilization without hurting latency.","category":"ai","date":"August 17, 2026","author":"ByteLedger Team","readTime":"3 min read","hue":240},{"slug":"kv-cache-quantization-2026","title":"KV Cache Quantisation: More Sequences in the Same Memory","excerpt":"At long context lengths the attention cache outgrows the model weights. Storing it at lower precision buys back concurrency, at a quality cost that is usually smaller than expected.","category":"ai","date":"January 12, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":205}],"sameCategory":[{"slug":"bazel-build-explained-2026","title":"Bazel Explained: What Hermetic Builds Actually Cost","excerpt":"Bazel delivers reproducible, precisely cached builds and demands that you describe everything. Here is what you get, what it costs, and who should say no.","category":"code","date":"August 19, 2026","author":"ByteLedger Team","readTime":"3 min read","hue":145},{"slug":"graphql-federation-2026","title":"GraphQL Federation: One Graph, Many Teams","excerpt":"Federation composes independently-owned subgraphs into a single API. Here is what it solves, the operational cost it adds, and when a simpler gateway is enough.","category":"code","date":"August 19, 2026","author":"ByteLedger Team","readTime":"3 min read","hue":300},{"slug":"hermetic-builds-explained-2026","title":"Hermetic Builds Explained: Same Inputs, Same Output","excerpt":"A hermetic build depends only on declared inputs. Here is what that property gives you, the common leaks that break it, and how to move toward it incrementally.","category":"code","date":"August 19, 2026","author":"ByteLedger Team","readTime":"3 min read","hue":130},{"slug":"monorepo-tooling-compared-2026","title":"Monorepo Tooling Compared: What You Actually Need","excerpt":"Monorepo tools solve task orchestration and caching, not repository layout. Here is what each tier provides and how to tell when you have outgrown the simple option.","category":"code","date":"August 19, 2026","author":"ByteLedger Team","readTime":"3 min read","hue":150}],"newer":{"slug":"contextual-retrieval-explained-2026","title":"Contextual Retrieval: Giving Chunks Their Context Back","excerpt":"Splitting a document destroys the context each piece depended on. Prepending a short description of where a chunk came from fixes more retrieval failures than tuning chunk size.","category":"ai","date":"August 11, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":198},"older":{"slug":"cross-encoder-reranking-2026","title":"Cross-Encoder Reranking: The Cheapest Big Quality Win","excerpt":"Reranking a shortlist with a model that reads query and document together is usually the highest-return change in a retrieval pipeline. Here is how to do it.","category":"ai","date":"August 11, 2026","author":"ByteLedger Team","readTime":"3 min read","hue":205}}