A new inference replica reports healthy and starts receiving traffic. The first few requests take several times longer than normal, then latency settles. Users who arrived during that window experienced a service that looked broken.
Loading model weights is the obvious part of a cold start and not the whole of it. Several things happen lazily on first use, and they happen while a real user waits unless you trigger them deliberately.
What changed in 2026
- Autoscaling made cold starts routine. Elastic inference meant replicas started and stopped constantly rather than once per deploy.
- Compilation caching improved. Persisting compiled kernels across restarts removed a large share of first-request cost where supported.
- Readiness semantics got attention. The distinction between loaded and warm became a recognised source of latency spikes.
- Warm pools became a standard cost line. Keeping idle loaded replicas was accepted as the price of elasticity.
What actually happens on a cold start
| Phase |
Cost |
Reducible by |
| Acquire hardware |
Varies by provider |
Reserved capacity |
| Pull model artefact |
Large for big models |
Local caching |
| Load weights into GPU memory |
Substantial |
Faster storage, smaller model |
| Allocate KV cache pools |
Moderate |
Pre-allocation at startup |
| Compile kernels for shapes |
Moderate, per shape |
Compilation cache, warmup |
| First real inference |
Slow |
Warmup requests |
The two people forget are kernel compilation and allocation. Many inference stacks compile optimised kernels for the specific tensor shapes they encounter, and that compilation happens the first time each shape appears. A replica serving its first long prompt compiles for that shape while the user waits.
Memory allocation behaves similarly — cache pools grow on demand unless pre-allocated, and growing them mid-request is slow.
Warm with realistic requests
The mistake is warming with a trivial request. Sending a two-token prompt exercises almost none of the paths real traffic uses, so the replica reports warm and then compiles for a realistic shape on the first genuine request.
Effective warmup sends requests resembling production: a prompt of typical length, generating a reasonable number of tokens, ideally across a few representative shapes if your traffic varies widely — a short one, a long one, and one with tool definitions if you use them.
That takes longer than a trivial warmup and it is the point. You are paying the compilation and allocation cost deliberately, on a request nobody is waiting for.
If your workload has a stable system prompt, warming with it also populates the prefix cache, so the first real request may hit rather than miss — see prefix caching.
Do not route traffic until warm
A readiness check that returns healthy as soon as the process starts will route users into the cold path. The check must not pass until warmup has completed.
That means separating liveness from readiness properly: liveness says the process is alive, readiness says it can serve at expected performance. Conflating them is the single most common cause of latency spikes during scale-up events.
The corresponding load balancer behaviour matters too. Some balancers send a share of traffic to a new backend immediately to test it, which defeats a correct readiness check. Confirm the routing layer respects readiness rather than probing independently.
Local artefact caching is the other high-value change. Pulling a large model from object storage on every replica start dominates the total, and caching it on instance-local storage — or using a shared fast filesystem — removes most of that repeatedly.
Common mistakes
- Trivial warmup requests. Do not exercise real shapes.
- Readiness passing before warm. Routes users into the slow path.
- Pulling the artefact on every start. Dominates cold start time.
- No compilation cache. Recompiles identical kernels every restart.
- Warming one shape when traffic varies. Compilation happens per shape.
- Scale-down that ignores warmup cost. Removing a warm replica discards real investment.
- Not measuring cold start. It is invisible in steady-state metrics.
FAQ
How long should warmup take?
As long as it takes to exercise your real request shapes — typically a handful of requests. Measure first-request latency after warmup; if it is still elevated, the warmup was not representative.
Does this apply to hosted APIs?
Not to you directly; the provider manages it. It is a self-hosted concern, though provider-side cold starts can explain occasional unexplained slow requests.
Is a warm pool worth the cost?
If your traffic is spiky and user-facing, usually yes — the alternative is users experiencing cold starts. If traffic is steady, replicas rarely start and the pool is waste.
How does this interact with scale-to-zero?
Directly and badly. Scale-to-zero guarantees a cold start on the next request, which is why it suits only latency-tolerant workloads — see AI inference autoscaling.
Where to go next
For the scaling behaviour warmup supports, read AI inference autoscaling. For sizing steady-state capacity, AI capacity planning, and for the cache warmup populates, prefix caching.