Spot capacity is spare hardware sold cheaply on the condition that it can be taken back. For a workload that can be interrupted and resumed, that is a substantial saving on the single largest line in most AI budgets.
For a workload that cannot, it is a way to lose days of training to an event you were warned about and did not handle.
What changed in 2026
- Spot became standard for training. Enough teams built reliable checkpointing that interruptible capacity moved from adventurous to normal for non-urgent training.
- Frameworks improved recovery. Built-in support for resuming distributed jobs after node loss reduced the engineering burden.
- Reclamation notices got more consistent. Advance warning, though short, became something you can reliably build on.
- Availability stayed the real variable. Spot pricing mattered less than whether the capacity existed at all in a given pool.
Checkpoint economics
The checkpoint interval is an optimisation with a clean form: you lose, on average, half an interval of work per interruption, and you pay the checkpoint cost every interval.
Too infrequent and each interruption discards hours. Too frequent and you spend a meaningful fraction of the run writing state instead of training.
| Interval |
Expected loss per interruption |
Overhead |
| Very long |
Hours of work |
Negligible |
| Moderate |
Minutes |
Small |
| Very short |
Seconds |
Significant |
The sensible target is an interval where the expected loss is comfortably smaller than the cumulative checkpoint overhead you would pay by going shorter. In practice most teams land on something in the tens of minutes and then find that writing checkpoints is slower than expected, which pushes toward asynchronous or sharded checkpoint writing.
Checkpoint verification deserves a mention: an unverified checkpoint is a hope. Restore from one deliberately, once, before relying on the mechanism — the same argument as restore drills for databases.
Handle the notice
Spot reclamation typically comes with a short warning — enough to save state if your process is listening for it.
Implementing that handler is the highest-value piece of work in this area. Catch the signal, write a checkpoint immediately, and exit cleanly. That converts a lost interval into a lost few seconds.
The handler needs to be fast, which usually means keeping checkpoint state ready to flush rather than assembling it on demand. It also needs to be tested, because a handler that has never fired is an assumption.
Multi-node is harder
Single-node training loses one machine and restarts. Distributed training loses one node and the entire job stalls, because the remaining nodes are waiting on a peer that no longer exists.
Three things make this manageable. Elastic training, where the job continues with fewer workers and rebalances, avoids a full restart. Fast coordinated recovery, where all workers restart from a shared checkpoint, is simpler and costs the time to restart. And diversification across instance types and availability zones reduces the chance of losing several nodes at once, since reclamation correlates strongly within a pool.
For large distributed runs, a mixed strategy is common: on-demand or reserved capacity for a stable core, spot for additional workers that can come and go.
Not for serving
Spot suits training, batch inference, evaluation runs, and data preparation — anything that can resume.
It does not suit interactive serving. Losing a replica mid-request drops user traffic, and the cold start to replace it is long enough that capacity gaps are visible. Teams attracted by the pricing sometimes try this and discover the failure mode during a reclamation event.
The exception is a serving pool where spot replicas are strictly overflow above a reserved core, and losing them degrades capacity rather than availability.
Common mistakes
- No reclamation handler. Losing the full interval every time for no reason.
- Checkpoint interval set by intuition. Do the arithmetic; it is simple.
- Never testing a restore. An unverified checkpoint is not a checkpoint.
- Single instance type and zone. Correlated reclamation takes everything at once.
- Spot for interactive serving. Drops user traffic.
- Ignoring checkpoint write time. It can be slower than the training step it protects.
- Storing checkpoints only locally. Reclaimed instances take local storage with them.
FAQ
How much cheaper is it?
Substantially, and it varies by region, instance type, and demand. The variability matters as much as the level — budget on the assumption that pricing and availability both move.
What happens if capacity is unavailable?
Your job does not start, or does not scale to the size you requested. Building for graceful degradation to fewer workers is better than assuming the request succeeds.
Can I mix spot and on-demand in one job?
Yes, and it is a good pattern — a stable core plus interruptible additional capacity gives most of the saving with much less risk.
Does this work for fine-tuning?
Well, since fine-tuning runs are typically shorter and checkpoint cheaply. Adapter training in particular has small checkpoints — see QLoRA.
Where to go next
For the committed tier this complements, read reserved GPU capacity. For hardware selection, GPU procurement, and for the memory techniques that reduce how much capacity you need, gradient accumulation.