The autoscaling configuration that has worked for every web service you run — target 70% CPU, scale up when exceeded — does nothing useful in front of an LLM. CPU sits at 15% while the GPU is fully saturated and requests are queueing.
Inference scaling needs different signals, and the long time to bring a replica into service changes the strategy as much as the signals do.
What changed in 2026
- Queue-based scaling became the recommended default. Provider and framework guidance converged on queue depth as the primary signal.
- Model loading times stayed long. Despite faster storage and better caching, bringing a large model into GPU memory remained slow enough to dominate scaling behaviour.
- Scale-to-zero found its niche. Viable for genuinely intermittent workloads that tolerate a long first request, unusable for anything interactive.
- Warm pools became common. Keeping loaded replicas idle emerged as the pragmatic answer to cold starts.
Signals that work
| Signal |
Useful? |
Notes |
| CPU utilisation |
No |
Stays low on GPU-bound work |
| GPU utilisation |
Partly |
Can read high while capacity remains |
| Requests per second |
Weak |
Ignores token shape entirely |
| Queue depth |
Yes |
Direct measure of unmet demand |
| Tokens per second |
Yes |
Tracks real throughput demand |
| Time to first token |
Yes |
Degrades when queueing starts |
| Batch occupancy |
Yes |
Shows whether batches are filling |
Queue depth is the strongest single signal because it measures exactly the thing you care about: work arriving faster than it is being served. It rises before latency does, which gives the scaling action time to take effect.
GPU utilisation is deceptive on its own. It can read high while a system still has capacity — a small batch keeps the device busy without saturating throughput — and it does not distinguish useful work from inefficiency. Pair it with batch occupancy to know whether the GPU is busy or merely occupied.
Time to first token degrading is a good confirmatory signal, since it is what users experience, and it lags queue depth slightly — see TTFT vs TPOT.
Cold starts change everything
A web service replica starts in seconds. An inference replica must acquire a GPU, pull a large model artefact, and load it into memory — a process measured in minutes for large models.
That gap makes reactive scaling structurally inadequate. By the time a new replica is serving, the burst that triggered it may be over, and users experienced the full degradation while waiting.
Three responses, usually combined:
Predictive scaling. Scale on known patterns — time of day, business hours, scheduled batch jobs — rather than waiting for demand. Most workloads are more predictable than they feel.
Warm pools. Keep replicas loaded and idle, ready to accept traffic immediately. Costly, and the cost is the price of elasticity for this workload class.
Faster loading. Cache model artefacts on local storage rather than pulling from object storage each time, and use faster load paths where available. This reduces the problem without removing it — see model warmup strategies.
Asymmetric thresholds
Scale up aggressively, scale down conservatively.
Adding a replica early costs some money. Removing one you need again in ninety seconds costs a full cold start during which users are queueing — a far worse outcome.
So: a low threshold and short window for scaling up, a high threshold and long cooldown for scaling down. For spiky traffic the asymmetry should be pronounced, with scale-down measured in tens of minutes rather than minutes.
Scale-to-zero is the extreme case of aggressive scale-down. It works for genuinely intermittent internal tools where a multi-minute first request is acceptable. It does not work for anything user-facing, and teams adopting it for cost reasons frequently discover this after shipping.
Common mistakes
- CPU-based scaling. Meaningless for GPU work.
- Reactive-only scaling. Cold starts make it too slow.
- Symmetric up and down thresholds. Causes thrashing.
- GPU utilisation alone. Does not distinguish busy from saturated.
- Scale-to-zero on interactive services. The first request is unacceptable.
- Not accounting for parallel-group granularity. Multi-GPU deployments scale in groups, not single devices — see tensor parallelism.
- Ignoring provider capacity limits. Autoscaling into a quota you do not have fails silently.
FAQ
What threshold should I use for queue depth?
Low enough that scaling completes before latency breaches your budget, which depends on your cold start time. A long cold start requires an early trigger.
Is scale-to-zero ever right?
For intermittent internal or batch workloads where a slow first request is fine. Not for anything a person is waiting on.
How do I handle daily patterns?
Schedule capacity ahead of known peaks rather than reacting to them. Predictable load should be provisioned, not autoscaled.
Does this apply to hosted APIs?
Your provider handles the scaling. Your equivalent concerns are rate limits and quota, which behave like capacity limits and need the same planning — see AI capacity planning.
Where to go next
For sizing the baseline, read AI capacity planning. For reducing cold start cost, model warmup strategies, and for committed capacity as an alternative to elasticity, reserved GPU capacity.