Most teams trying to make model serving faster reach for one big idea first, and the honest answer is that no single technique gets you there. Cheap, fast inference in 2026 comes from stacking a short list of well-understood techniques in a sensible order: batching that keeps hardware busy, cache management that raises concurrency, quantization that shrinks memory, and speculative decoding that cuts latency on top of all of it. Each one is well proven; the skill is applying them in the right order and measuring each in isolation. None of these techniques are exotic research ideas anymore — they ship as configuration flags in the major open-source serving frameworks — which makes the real failure mode less about implementation difficulty and more about applying too many changes at once and losing the ability to tell which one actually helped.
What changed in 2026
- Continuous batching became the default serving pattern. vLLM, TensorRT-LLM, and SGLang all pack new requests into in-flight batches instead of waiting for a batch to finish, replacing static batching almost everywhere.
- Disaggregated prefill and decode spread into mainstream open-source stacks, no longer limited to a handful of frontier labs running custom infrastructure.
- FP8 became a safe default on current hardware, and FP4 moved from a research curiosity to a production option for some model families.
- Speculative decoding matured into a standard serving flag rather than a research trick teams had to implement from scratch.
- Compiler-level optimization became routine.
torch.compile, XLA, and similar just-in-time compilation paths now ship enabled by default in more serving stacks, folding operator fusion into the standard deployment path instead of a manual tuning step.
The technique landscape
| Technique |
What it does |
Typical win |
| Continuous batching |
Packs new requests into in-flight batches instead of waiting for one to finish |
Large throughput gain, minimal latency cost |
| Paged attention / KV-cache paging |
Allocates KV-cache in fixed-size blocks instead of one contiguous region per request |
More concurrent requests per GPU |
| Quantization (FP8 / INT8 / FP4) |
Shrinks weight and activation precision |
Lower memory footprint, often higher throughput |
| Speculative decoding |
A small draft model proposes tokens the large model verifies in one pass |
Lower latency per request |
| Operator fusion / compilation |
Fuses kernels via TensorRT or torch.compile to cut overhead |
Lower latency, better hardware utilization |
| Tensor / pipeline parallelism |
Splits a model across multiple chips |
Enables serving models too large for one chip |
These techniques are not independent of each other. Continuous batching raises throughput by keeping the accelerator busy, but it also means a single slow request can sit in the same batch as many fast ones, so watch tail latency, not just average throughput, once it is enabled. Quantization and speculative decoding compound well together, since a smaller model footprint leaves more memory free for the KV cache that speculative decoding also depends on.
How to apply them in order
- Establish a correctness and quality baseline first. You cannot judge an optimization against a moving target.
- Turn on continuous batching and paged attention. These are close to free wins at the serving-framework level and should come before anything else.
- Quantize, then re-validate quality on your own evaluation set, not a general benchmark.
- Add speculative decoding if latency still matters after the above, since it costs extra compute on the draft model.
- Only then consider parallelism or sharding, and only for models that genuinely do not fit on one chip.
Common mistakes
- Turning on every optimization simultaneously. You lose the ability to attribute a regression, or a win, to any specific change.
- Quantizing without an evaluation set. A demo that "looks fine" misses accuracy regressions on your specific task distribution, especially for reasoning-heavy work.
- Ignoring KV-cache memory when sizing concurrency. Throughput collapses when cache eviction thrashes under real concurrent load, which single-request tests never reveal.
- Treating speculative decoding as free. It adds compute for the draft model and helps most when generation is memory-bandwidth-bound, not always.
FAQ
Does quantization always hurt accuracy?
Not meaningfully for most tasks at FP8. INT4 and FP4 need more careful validation, particularly on reasoning-heavy tasks.
What is continuous batching?
A serving technique that adds new requests into an already-running batch instead of waiting for the current batch to finish, keeping the accelerator busy.
Is speculative decoding worth the added complexity?
Yes when latency matters and a well-matched draft model is available. It is extra engineering for workloads that are already cheap and fast enough.
Do I need all of these techniques at once?
No. Start with batching and KV-cache paging, then add the rest only as your specific bottleneck demands it.
Does the order these are applied in actually matter?
Yes. Batching and cache management change the shape of the workload itself, so measuring quantization or speculative decoding before those are in place gives you numbers that will not hold once the rest of the stack changes.
Where to go next
See GPU vs TPU for inference for the hardware side of this same problem, streaming vs batch processing for a related throughput-versus-latency tradeoff, and what a race condition is for why shared state like a KV cache needs careful handling under concurrent requests.