The monitoring shows GPU utilisation pinned near 100%. The obvious conclusion is that the hardware is maxed out and you need more of it. Then someone measures tokens per second and finds it well below what the same hardware achieves elsewhere.
The metric is not lying exactly. It is answering a narrower question than people assume: was at least one kernel executing during this sampling interval? A single small operation using a fraction of the device counts identically to a fully saturated one.
What changed in 2026
- Token throughput displaced utilisation as the headline metric. Teams stopped reporting GPU utilisation as a capacity indicator once its limitations became widely understood.
- Batch occupancy monitoring spread. Tracking how full batches actually are became standard in serving dashboards.
- Memory bandwidth entered the conversation. The bandwidth-bound nature of decode became common knowledge rather than specialist detail.
- Profiling tooling improved. Distinguishing compute-bound from bandwidth-bound phases got easier without deep expertise.
Why decode leaves the arithmetic units idle
Generating a token requires reading the model weights from memory and applying them to a small amount of data. For a single sequence, that is an enormous read and very little arithmetic.
The device spends most of its time waiting on memory. The arithmetic units — the thing utilisation loosely tracks — are mostly idle between operations, yet something is always executing, so utilisation reads high.
Batching is what fixes the ratio. Read the weights once, apply them to sixty-four sequences, and you have done sixty-four times the arithmetic for the same memory traffic. This is why throughput scales so well with batch size during decode, and why a serving system with small batches wastes hardware while reporting healthy utilisation.
| Situation |
Utilisation |
Throughput |
Actual state |
| Large batches, decode |
High |
High |
Healthy |
| Small batches, decode |
High |
Low |
Wasting the device |
| Long prefill |
High |
High |
Compute-bound, fine |
| Waiting on cache space |
Moderate |
Low |
Memory-limited |
| Idle between requests |
Low |
Low |
Under-loaded |
Rows two and four are the diagnostic cases, and utilisation cannot distinguish them from row one.
Metrics that actually tell you something
Tokens per second, split into input and output. This is the thing you are buying, and it is directly comparable across configurations.
Batch occupancy — the average number of sequences in flight relative to what the configuration allows. Persistently low occupancy with a full queue means something is preventing batches from filling, usually cache memory.
Preemption rate. Sequences being evicted and recomputed is strictly wasted work and a clear signal you are past the memory limit — see paged attention.
Queue depth alongside occupancy. A deep queue with low occupancy is the clearest indication of a memory constraint rather than a compute one.
Memory usage split between weights and cache. If cache dominates and occupancy is low, cache quantisation or shorter contexts will buy more throughput than more compute.
Diagnosing the common case
The pattern that appears repeatedly: high utilisation, low throughput, deep queue.
That is almost always small batches caused by cache pressure. Sequences cannot be admitted because there is no cache space, so batches stay small, so each expensive weight read serves few sequences, so throughput is poor while the device stays busy.
The fixes are all memory-side rather than compute-side: quantise the cache, reduce context length, enable prefix sharing, or add memory. Adding compute makes no difference, which is why the utilisation-driven instinct to buy more GPUs frequently disappoints — see KV cache quantisation.
Common mistakes
- Treating utilisation as saturation. It measures occupancy.
- Buying compute for a memory problem. No improvement.
- Not measuring batch occupancy. The number that explains throughput.
- Ignoring preemption. Wasted work hiding as activity.
- Comparing utilisation across workloads. Prefill-heavy and decode-heavy behave differently.
- Setting a utilisation target. Target throughput; utilisation is a side effect.
FAQ
Is high utilisation ever a good sign?
Alongside high throughput and full batches, yes — that is a healthy saturated system. Alone, it means very little.
What throughput should I expect?
Depends on the model, hardware, batch size, and prompt shape, which is why published figures rarely transfer. Establish your own baseline and compare configurations against it.
Does this apply during training?
Training is more consistently compute-bound, so utilisation is a somewhat better proxy there. The gradient-accumulation and precision choices still affect efficiency in ways utilisation does not show.
How do I know if I am bandwidth-bound?
Increase batch size. If throughput rises roughly proportionally, you were bandwidth-bound and under-batched. If it does not, you are compute-bound or constrained elsewhere.
Where to go next
For the memory limit that usually causes this, read paged attention and KV cache quantisation. For sizing against real throughput, AI capacity planning.