{"post":{"slug":"gpu-utilization-explained-2026","title":"GPU Utilisation: The Metric That Lies to You","excerpt":"Your dashboard shows 95% GPU utilisation and throughput is poor. The metric counts whether the device is busy, not whether it is doing anything worthwhile.","category":"ai","date":"January 26, 2026","author":"ByteLedger Team","content":"<span class=\"dropcap\">T</span>he monitoring shows GPU utilisation pinned near 100%. The obvious conclusion is that the hardware is maxed out and you need more of it. Then someone measures tokens per second and finds it well below what the same hardware achieves elsewhere.\n\nThe metric is not lying exactly. It is answering a narrower question than people assume: was at least one kernel executing during this sampling interval? A single small operation using a fraction of the device counts identically to a fully saturated one.\n\n## What changed in 2026\n\n- **Token throughput displaced utilisation as the headline metric.** Teams stopped reporting GPU utilisation as a capacity indicator once its limitations became widely understood.\n- **Batch occupancy monitoring spread.** Tracking how full batches actually are became standard in serving dashboards.\n- **Memory bandwidth entered the conversation.** The bandwidth-bound nature of decode became common knowledge rather than specialist detail.\n- **Profiling tooling improved.** Distinguishing compute-bound from bandwidth-bound phases got easier without deep expertise.\n\n## Why decode leaves the arithmetic units idle\n\nGenerating a token requires reading the model weights from memory and applying them to a small amount of data. For a single sequence, that is an enormous read and very little arithmetic.\n\nThe device spends most of its time waiting on memory. The arithmetic units — the thing utilisation loosely tracks — are mostly idle between operations, yet something is always executing, so utilisation reads high.\n\nBatching is what fixes the ratio. Read the weights once, apply them to sixty-four sequences, and you have done sixty-four times the arithmetic for the same memory traffic. This is why throughput scales so well with batch size during decode, and why a serving system with small batches wastes hardware while reporting healthy utilisation.\n\n| Situation | Utilisation | Throughput | Actual state |\n| --- | --- | --- | --- |\n| Large batches, decode | High | High | Healthy |\n| Small batches, decode | High | Low | Wasting the device |\n| Long prefill | High | High | Compute-bound, fine |\n| Waiting on cache space | Moderate | Low | Memory-limited |\n| Idle between requests | Low | Low | Under-loaded |\n\nRows two and four are the diagnostic cases, and utilisation cannot distinguish them from row one.\n\n## Metrics that actually tell you something\n\n**Tokens per second**, split into input and output. This is the thing you are buying, and it is directly comparable across configurations.\n\n**Batch occupancy** — the average number of sequences in flight relative to what the configuration allows. Persistently low occupancy with a full queue means something is preventing batches from filling, usually cache memory.\n\n**Preemption rate.** Sequences being evicted and recomputed is strictly wasted work and a clear signal you are past the memory limit — see [paged attention](/blog/paged-attention-explained-2026).\n\n**Queue depth alongside occupancy.** A deep queue with low occupancy is the clearest indication of a memory constraint rather than a compute one.\n\n**Memory usage split** between weights and cache. If cache dominates and occupancy is low, cache quantisation or shorter contexts will buy more throughput than more compute.\n\n## Diagnosing the common case\n\nThe pattern that appears repeatedly: high utilisation, low throughput, deep queue.\n\nThat is almost always small batches caused by cache pressure. Sequences cannot be admitted because there is no cache space, so batches stay small, so each expensive weight read serves few sequences, so throughput is poor while the device stays busy.\n\nThe fixes are all memory-side rather than compute-side: quantise the cache, reduce context length, enable prefix sharing, or add memory. Adding compute makes no difference, which is why the utilisation-driven instinct to buy more GPUs frequently disappoints — see [KV cache quantisation](/blog/kv-cache-quantization-2026).\n\n## Common mistakes\n\n- **Treating utilisation as saturation.** It measures occupancy.\n- **Buying compute for a memory problem.** No improvement.\n- **Not measuring batch occupancy.** The number that explains throughput.\n- **Ignoring preemption.** Wasted work hiding as activity.\n- **Comparing utilisation across workloads.** Prefill-heavy and decode-heavy behave differently.\n- **Setting a utilisation target.** Target throughput; utilisation is a side effect.\n\n## FAQ\n\n**Is high utilisation ever a good sign?**\nAlongside high throughput and full batches, yes — that is a healthy saturated system. Alone, it means very little.\n\n**What throughput should I expect?**\nDepends on the model, hardware, batch size, and prompt shape, which is why published figures rarely transfer. Establish your own baseline and compare configurations against it.\n\n**Does this apply during training?**\nTraining is more consistently compute-bound, so utilisation is a somewhat better proxy there. The gradient-accumulation and precision choices still affect efficiency in ways utilisation does not show.\n\n**How do I know if I am bandwidth-bound?**\nIncrease batch size. If throughput rises roughly proportionally, you were bandwidth-bound and under-batched. If it does not, you are compute-bound or constrained elsewhere.\n\n## Where to go next\n\nFor the memory limit that usually causes this, read [paged attention](/blog/paged-attention-explained-2026) and [KV cache quantisation](/blog/kv-cache-quantization-2026). For sizing against real throughput, [AI capacity planning](/blog/ai-capacity-planning-2026).","hue":218,"metaTitle":"GPU Utilisation: The Metric That Lies to You","metaDescription":"Why GPU utilisation can read high while throughput is poor, what memory bandwidth limits mean for inference, and which metrics to watch instead.","keywords":["gpu utilization","memory bandwidth bound","inference throughput metrics","batch occupancy","gpu monitoring llm","arithmetic intensity","performance diagnosis","ai infrastructure 2026"],"tldr":["**Utilisation measures occupancy, not efficiency** — one tiny kernel running counts the same as a saturated device.","**Memory bandwidth is the real limit during generation** — arithmetic units idle while weights are read.","**Batch occupancy tells you more** — whether batches are actually full is the throughput question.","**High utilisation with low throughput means small batches** — the device is busy doing very little per pass.","**Skip:** targeting a utilisation number. Target tokens per second, which is the thing you actually want."],"readTime":"4 min read","wordCount":827},"related":[{"slug":"ai-workload-scheduling-2026","title":"AI Workload Scheduling: Keeping Expensive GPUs Busy","excerpt":"Idle accelerators are the largest waste in most AI infrastructure. Here is how scheduling, queueing, and workload mixing lift utilization without hurting latency.","category":"ai","date":"August 17, 2026","author":"ByteLedger Team","readTime":"3 min read","hue":240},{"slug":"kv-cache-quantization-2026","title":"KV Cache Quantisation: More Sequences in the Same Memory","excerpt":"At long context lengths the attention cache outgrows the model weights. Storing it at lower precision buys back concurrency, at a quality cost that is usually smaller than expected.","category":"ai","date":"January 12, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":205},{"slug":"lpddr6-explained-2026","title":"LPDDR6 Explained: Why Phone Memory Is the Bottleneck","excerpt":"On-device AI made memory bandwidth the limiting spec in phones and laptops. Here is what LPDDR6 changes architecturally and what it means for local models.","category":"tech","date":"August 18, 2026","author":"ByteLedger Team","readTime":"3 min read","hue":245}],"sameCategory":[{"slug":"agent-cost-attribution-2026","title":"Agent Cost Attribution: Finding Where the Tokens Went","excerpt":"A single monthly API bill tells you that agents are expensive. It does not tell you which feature, which customer, or which retry loop is responsible. That gap is fixable.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":220},{"slug":"human-in-the-loop-agents-2026","title":"Human-in-the-Loop Agents: Where to Put the Approval Gate","excerpt":"Approve everything and the agent is a slow autocomplete. Approve nothing and you have handed irreversible actions to a system that can be talked into things.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":197},{"slug":"reasoning-effort-controls-2026","title":"Reasoning Effort Controls: Tuning How Hard a Model Thinks","excerpt":"Fixed thinking budgets gave way to effort levels, and the default is rarely the right setting. Here is how the dial behaves and where each level actually belongs.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":210},{"slug":"retrieval-metrics-explained-2026","title":"Retrieval Metrics: Measuring the Half of RAG That Fails","excerpt":"When a RAG system answers badly, the retrieval usually failed before the model ever spoke. Measuring generation quality alone hides that entirely.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":215}],"newer":{"slug":"defi-lending-platforms-2026","title":"DeFi lending platforms in 2026: Aave, Compound, Morpho explained","excerpt":"DeFi lending grew up. Here is how Aave, Compound, and Morpho actually work in 2026, with honest risk and yield comparisons.","category":"finance","date":"January 26, 2026","author":"ByteLedger Team","readTime":"3 min read","hue":285},"older":{"slug":"how-to-cut-grocery-bill-2026","title":"How to cut your grocery bill 30% in 2026 (real tactics, not coupons)","excerpt":"A grounded 2026 guide to actually cutting your grocery bill 30%: store choice, meal planning, freezer math, and the apps that pay you back.","category":"finance","date":"January 26, 2026","author":"ByteLedger Team","readTime":"3 min read","hue":290}}