The most expensive line in an AI infrastructure budget is usually hardware that is sitting idle. A cluster provisioned for peak interactive traffic runs at a fraction of capacity most of the day, and every idle accelerator-hour is billed exactly the same as a busy one. Before buying more capacity, most organizations can extract considerably more from what they already have.
Scheduling is how.
What changed in 2026
- Continuous batching became standard in serving stacks. Rather than assembling fixed batches, modern servers add and remove requests from an in-flight batch continuously, which raised throughput substantially with no latency penalty.
- Mixed-workload clusters spread. Running latency-sensitive inference and preemptible batch work on the same hardware, with priority-based preemption, moved from advanced practice to common configuration.
- Utilization became a reported metric. Platform teams started tracking accelerator utilization the way they track server CPU, which made the waste visible.
- Disaggregated serving gained traction. Separating the prefill and generation phases onto different hardware pools improved efficiency for workloads with very uneven prompt and output lengths.
Where the capacity goes
| Cause of idle time |
Typical fix |
| Provisioned for peak, running at average |
Mix in preemptible batch work |
| Fixed batch sizes waiting to fill |
Continuous batching |
| Memory-bound generation with idle compute |
Larger effective batch; better memory bandwidth |
| Long prompts blocking short requests |
Separate queues by request shape |
| Development and experimentation on production hardware |
Separate lower-priority pool |
| Failed jobs restarting from zero |
Checkpointing |
Continuous batching deserves the top spot in most environments. Under fixed batching, a request arriving just after a batch starts waits for the whole batch to finish. Under continuous batching it joins immediately and departs when it completes. The throughput gain is large and the latency effect is positive rather than negative, which is an unusually good trade.
Mixing workloads safely
The scheduling insight is that interactive traffic is bursty and batch work is infinitely patient. Running both on the same pool, with interactive at high priority and batch preemptible, fills the troughs without touching the peaks.
This only works with checkpointing. A preempted batch job that restarts from the beginning wastes everything it had done, and if preemption is frequent the job may never finish. Checkpoint at intervals short enough that a preemption costs minutes rather than hours.
Separate queues by request shape. A very long prompt occupies memory and compute differently from a short one, and mixing them in one queue lets a few large requests degrade latency for many small ones. Routing by estimated cost — the same classification used in AI model routers explained — keeps the fast path fast.
If you are buying capacity rather than running it, most of this is the provider's problem, and it shows up in your pricing rather than your scheduling. The equivalent lever is choosing the right endpoint: batch inference cost savings is scheduling optimization you get by asking for it.
Common mistakes
- Adding capacity before measuring utilization. The most expensive way to solve a scheduling problem.
- Fixed batch sizes. Leaves substantial throughput unclaimed on modern serving stacks.
- Preemption without checkpointing. Turns idle-time recovery into wasted work.
- One queue for all request shapes. Long requests starve short ones and tail latency suffers.
- Development on the production pool. Experiments consume capacity unpredictably and at the worst moments.
FAQ
What utilization should I target?
Higher than you probably have, and short of saturation — running at the limit removes the headroom that absorbs bursts. The right number depends on how spiky your traffic is.
Does continuous batching hurt latency?
Generally it improves it, because requests stop waiting for a batch to fill. Very large in-flight batches can increase per-token latency, so the batch size cap still matters.
Is disaggregated prefill worth the complexity?
For workloads with very long prompts and short outputs, or the reverse, it can be. For balanced workloads the added operational complexity usually is not repaid.
How do I measure utilization properly?
Accelerator utilization percentage alone is misleading — a memory-bound workload shows high utilization while doing little useful work. Track tokens served per accelerator-hour alongside it.
Where to go next
For the capacity purchasing side, read AI compute leasing explained. For the hosted equivalent, batch inference cost savings and AI inference providers compared.