{"post":{"slug":"ai-inference-autoscaling-2026","title":"AI Inference Autoscaling: Why the Usual Signals Fail","excerpt":"Scaling on CPU utilisation works for web services and is meaningless for GPU inference. The signals that work are different, and the cold start is long enough to change the strategy.","category":"ai","date":"February 13, 2026","author":"ByteLedger Team","content":"<span class=\"dropcap\">T</span>he autoscaling configuration that has worked for every web service you run — target 70% CPU, scale up when exceeded — does nothing useful in front of an LLM. CPU sits at 15% while the GPU is fully saturated and requests are queueing.\n\nInference scaling needs different signals, and the long time to bring a replica into service changes the strategy as much as the signals do.\n\n## What changed in 2026\n\n- **Queue-based scaling became the recommended default.** Provider and framework guidance converged on queue depth as the primary signal.\n- **Model loading times stayed long.** Despite faster storage and better caching, bringing a large model into GPU memory remained slow enough to dominate scaling behaviour.\n- **Scale-to-zero found its niche.** Viable for genuinely intermittent workloads that tolerate a long first request, unusable for anything interactive.\n- **Warm pools became common.** Keeping loaded replicas idle emerged as the pragmatic answer to cold starts.\n\n## Signals that work\n\n| Signal | Useful? | Notes |\n| --- | --- | --- |\n| CPU utilisation | No | Stays low on GPU-bound work |\n| GPU utilisation | Partly | Can read high while capacity remains |\n| Requests per second | Weak | Ignores token shape entirely |\n| **Queue depth** | Yes | Direct measure of unmet demand |\n| **Tokens per second** | Yes | Tracks real throughput demand |\n| Time to first token | Yes | Degrades when queueing starts |\n| Batch occupancy | Yes | Shows whether batches are filling |\n\nQueue depth is the strongest single signal because it measures exactly the thing you care about: work arriving faster than it is being served. It rises before latency does, which gives the scaling action time to take effect.\n\nGPU utilisation is deceptive on its own. It can read high while a system still has capacity — a small batch keeps the device busy without saturating throughput — and it does not distinguish useful work from inefficiency. Pair it with batch occupancy to know whether the GPU is busy or merely occupied.\n\nTime to first token degrading is a good confirmatory signal, since it is what users experience, and it lags queue depth slightly — see [TTFT vs TPOT](/blog/ttft-vs-tpot-2026).\n\n## Cold starts change everything\n\nA web service replica starts in seconds. An inference replica must acquire a GPU, pull a large model artefact, and load it into memory — a process measured in minutes for large models.\n\nThat gap makes reactive scaling structurally inadequate. By the time a new replica is serving, the burst that triggered it may be over, and users experienced the full degradation while waiting.\n\nThree responses, usually combined:\n\n**Predictive scaling.** Scale on known patterns — time of day, business hours, scheduled batch jobs — rather than waiting for demand. Most workloads are more predictable than they feel.\n\n**Warm pools.** Keep replicas loaded and idle, ready to accept traffic immediately. Costly, and the cost is the price of elasticity for this workload class.\n\n**Faster loading.** Cache model artefacts on local storage rather than pulling from object storage each time, and use faster load paths where available. This reduces the problem without removing it — see [model warmup strategies](/blog/model-warmup-strategies-2026).\n\n## Asymmetric thresholds\n\nScale up aggressively, scale down conservatively.\n\nAdding a replica early costs some money. Removing one you need again in ninety seconds costs a full cold start during which users are queueing — a far worse outcome.\n\nSo: a low threshold and short window for scaling up, a high threshold and long cooldown for scaling down. For spiky traffic the asymmetry should be pronounced, with scale-down measured in tens of minutes rather than minutes.\n\nScale-to-zero is the extreme case of aggressive scale-down. It works for genuinely intermittent internal tools where a multi-minute first request is acceptable. It does not work for anything user-facing, and teams adopting it for cost reasons frequently discover this after shipping.\n\n## Common mistakes\n\n- **CPU-based scaling.** Meaningless for GPU work.\n- **Reactive-only scaling.** Cold starts make it too slow.\n- **Symmetric up and down thresholds.** Causes thrashing.\n- **GPU utilisation alone.** Does not distinguish busy from saturated.\n- **Scale-to-zero on interactive services.** The first request is unacceptable.\n- **Not accounting for parallel-group granularity.** Multi-GPU deployments scale in groups, not single devices — see [tensor parallelism](/blog/tensor-parallelism-explained-2026).\n- **Ignoring provider capacity limits.** Autoscaling into a quota you do not have fails silently.\n\n## FAQ\n\n**What threshold should I use for queue depth?**\nLow enough that scaling completes before latency breaches your budget, which depends on your cold start time. A long cold start requires an early trigger.\n\n**Is scale-to-zero ever right?**\nFor intermittent internal or batch workloads where a slow first request is fine. Not for anything a person is waiting on.\n\n**How do I handle daily patterns?**\nSchedule capacity ahead of known peaks rather than reacting to them. Predictable load should be provisioned, not autoscaled.\n\n**Does this apply to hosted APIs?**\nYour provider handles the scaling. Your equivalent concerns are rate limits and quota, which behave like capacity limits and need the same planning — see [AI capacity planning](/blog/ai-capacity-planning-2026).\n\n## Where to go next\n\nFor sizing the baseline, read [AI capacity planning](/blog/ai-capacity-planning-2026). For reducing cold start cost, [model warmup strategies](/blog/model-warmup-strategies-2026), and for committed capacity as an alternative to elasticity, [reserved GPU capacity](/blog/reserved-gpu-capacity-2026).","hue":207,"metaTitle":"AI Inference Autoscaling: Why Usual Signals Fail","metaDescription":"Why CPU-based autoscaling fails for GPU inference, which signals actually track demand, and how long model load times change scaling strategy.","keywords":["ai inference autoscaling","gpu autoscaling","queue depth scaling","model cold start","scale to zero llm","inference elasticity","kubernetes gpu scaling","ai infrastructure 2026"],"tldr":["**CPU utilisation is not a signal for GPU-bound serving** — it stays low while the GPU saturates.","**Scale on queue depth and token throughput** — those track actual demand.","**Model loading makes cold starts long** — scaling up after demand arrives is already too late.","**Scale up fast, scale down slowly** — the asymmetry avoids thrashing on spiky traffic.","**Skip:** aggressive scale-down on bursty workloads. Removing a replica you need again in ninety seconds costs more than keeping it."],"readTime":"4 min read","wordCount":887},"related":[{"slug":"model-warmup-strategies-2026","title":"Model Warmup: Why the First Request Is Always Slow","excerpt":"A freshly started inference replica serves its first requests far slower than its steady state. Warming it before traffic arrives is the difference between elastic and merely scalable.","category":"ai","date":"January 13, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":222},{"slug":"ai-workload-scheduling-2026","title":"AI Workload Scheduling: Keeping Expensive GPUs Busy","excerpt":"Idle accelerators are the largest waste in most AI infrastructure. Here is how scheduling, queueing, and workload mixing lift utilization without hurting latency.","category":"ai","date":"August 17, 2026","author":"ByteLedger Team","readTime":"3 min read","hue":240},{"slug":"gpu-memory-estimate-llm-2026","title":"GPU Memory for LLMs: Estimating What You Actually Need","excerpt":"Model size is only the first term. Here is how to estimate memory for inference and training, including the parts that catch people out at deployment.","category":"ai","date":"July 24, 2026","author":"ByteLedger Team","readTime":"3 min read","hue":240}],"sameCategory":[{"slug":"agent-cost-attribution-2026","title":"Agent Cost Attribution: Finding Where the Tokens Went","excerpt":"A single monthly API bill tells you that agents are expensive. It does not tell you which feature, which customer, or which retry loop is responsible. That gap is fixable.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":220},{"slug":"human-in-the-loop-agents-2026","title":"Human-in-the-Loop Agents: Where to Put the Approval Gate","excerpt":"Approve everything and the agent is a slow autocomplete. Approve nothing and you have handed irreversible actions to a system that can be talked into things.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":197},{"slug":"reasoning-effort-controls-2026","title":"Reasoning Effort Controls: Tuning How Hard a Model Thinks","excerpt":"Fixed thinking budgets gave way to effort levels, and the default is rarely the right setting. Here is how the dial behaves and where each level actually belongs.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":210},{"slug":"retrieval-metrics-explained-2026","title":"Retrieval Metrics: Measuring the Half of RAG That Fails","excerpt":"When a RAG system answers badly, the retrieval usually failed before the model ever spoke. Measuring generation quality alone hides that entirely.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":215}],"newer":{"slug":"password-rotation-myths-2026","title":"Password Rotation: Why the 90-Day Rule Made Things Worse","excerpt":"Forced periodic password changes were standard policy for decades. The research showed they produced weaker passwords, and the guidance reversed — though many organisations have not noticed.","category":"guides","date":"February 14, 2026","author":"ByteLedger Team","readTime":"3 min read","hue":158},"older":{"slug":"ai-search-perplexity-tips-2026","title":"AI search Perplexity tips in 2026: the power-user techniques worth knowing","excerpt":"Perplexity is the AI search tool most people use as a fancy chatbot. The power-user features turn it into a research tool that beats Google for specific kinds of work.","category":"ai","date":"February 13, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":208}}