{"post":{"slug":"spot-instances-for-training-2026","title":"Spot Instances for Training: Cheap Compute You Can Lose","excerpt":"Interruptible capacity costs a fraction of on-demand and can be reclaimed with little warning. Whether that is a bargain or a disaster depends entirely on your checkpointing.","category":"ai","date":"January 2, 2026","author":"ByteLedger Team","content":"<span class=\"dropcap\">S</span>pot capacity is spare hardware sold cheaply on the condition that it can be taken back. For a workload that can be interrupted and resumed, that is a substantial saving on the single largest line in most AI budgets.\n\nFor a workload that cannot, it is a way to lose days of training to an event you were warned about and did not handle.\n\n## What changed in 2026\n\n- **Spot became standard for training.** Enough teams built reliable checkpointing that interruptible capacity moved from adventurous to normal for non-urgent training.\n- **Frameworks improved recovery.** Built-in support for resuming distributed jobs after node loss reduced the engineering burden.\n- **Reclamation notices got more consistent.** Advance warning, though short, became something you can reliably build on.\n- **Availability stayed the real variable.** Spot pricing mattered less than whether the capacity existed at all in a given pool.\n\n## Checkpoint economics\n\nThe checkpoint interval is an optimisation with a clean form: you lose, on average, half an interval of work per interruption, and you pay the checkpoint cost every interval.\n\nToo infrequent and each interruption discards hours. Too frequent and you spend a meaningful fraction of the run writing state instead of training.\n\n| Interval | Expected loss per interruption | Overhead |\n| --- | --- | --- |\n| Very long | Hours of work | Negligible |\n| Moderate | Minutes | Small |\n| Very short | Seconds | Significant |\n\nThe sensible target is an interval where the expected loss is comfortably smaller than the cumulative checkpoint overhead you would pay by going shorter. In practice most teams land on something in the tens of minutes and then find that writing checkpoints is slower than expected, which pushes toward asynchronous or sharded checkpoint writing.\n\nCheckpoint *verification* deserves a mention: an unverified checkpoint is a hope. Restore from one deliberately, once, before relying on the mechanism — the same argument as [restore drills](/blog/restore-drills-2026) for databases.\n\n## Handle the notice\n\nSpot reclamation typically comes with a short warning — enough to save state if your process is listening for it.\n\nImplementing that handler is the highest-value piece of work in this area. Catch the signal, write a checkpoint immediately, and exit cleanly. That converts a lost interval into a lost few seconds.\n\nThe handler needs to be fast, which usually means keeping checkpoint state ready to flush rather than assembling it on demand. It also needs to be tested, because a handler that has never fired is an assumption.\n\n## Multi-node is harder\n\nSingle-node training loses one machine and restarts. Distributed training loses one node and the entire job stalls, because the remaining nodes are waiting on a peer that no longer exists.\n\nThree things make this manageable. **Elastic training**, where the job continues with fewer workers and rebalances, avoids a full restart. **Fast coordinated recovery**, where all workers restart from a shared checkpoint, is simpler and costs the time to restart. And **diversification** across instance types and availability zones reduces the chance of losing several nodes at once, since reclamation correlates strongly within a pool.\n\nFor large distributed runs, a mixed strategy is common: on-demand or reserved capacity for a stable core, spot for additional workers that can come and go.\n\n## Not for serving\n\nSpot suits training, batch inference, evaluation runs, and data preparation — anything that can resume.\n\nIt does not suit interactive serving. Losing a replica mid-request drops user traffic, and the [cold start](/blog/model-warmup-strategies-2026) to replace it is long enough that capacity gaps are visible. Teams attracted by the pricing sometimes try this and discover the failure mode during a reclamation event.\n\nThe exception is a serving pool where spot replicas are strictly overflow above a reserved core, and losing them degrades capacity rather than availability.\n\n## Common mistakes\n\n- **No reclamation handler.** Losing the full interval every time for no reason.\n- **Checkpoint interval set by intuition.** Do the arithmetic; it is simple.\n- **Never testing a restore.** An unverified checkpoint is not a checkpoint.\n- **Single instance type and zone.** Correlated reclamation takes everything at once.\n- **Spot for interactive serving.** Drops user traffic.\n- **Ignoring checkpoint write time.** It can be slower than the training step it protects.\n- **Storing checkpoints only locally.** Reclaimed instances take local storage with them.\n\n## FAQ\n\n**How much cheaper is it?**\nSubstantially, and it varies by region, instance type, and demand. The variability matters as much as the level — budget on the assumption that pricing and availability both move.\n\n**What happens if capacity is unavailable?**\nYour job does not start, or does not scale to the size you requested. Building for graceful degradation to fewer workers is better than assuming the request succeeds.\n\n**Can I mix spot and on-demand in one job?**\nYes, and it is a good pattern — a stable core plus interruptible additional capacity gives most of the saving with much less risk.\n\n**Does this work for fine-tuning?**\nWell, since fine-tuning runs are typically shorter and checkpoint cheaply. Adapter training in particular has small checkpoints — see [QLoRA](/blog/qlora-explained-2026).\n\n## Where to go next\n\nFor the committed tier this complements, read [reserved GPU capacity](/blog/reserved-gpu-capacity-2026). For hardware selection, [GPU procurement](/blog/gpu-procurement-guide-2026), and for the memory techniques that reduce how much capacity you need, [gradient accumulation](/blog/gradient-accumulation-explained-2026).","hue":204,"metaTitle":"Spot Instances for Training: Cheap, Interruptible GPUs","metaDescription":"How to run training on interruptible spot capacity — checkpoint frequency, handling reclamation notices, and why multi-node jobs need extra care.","keywords":["spot instances training","preemptible gpu","interruptible compute","checkpointing strategy","fault tolerant training","cloud cost reduction ai","spot reclamation","ml infrastructure 2026"],"tldr":["**Checkpoint frequently enough that losing progress is cheap** — the interval should cost less than it saves.","**Handle the reclamation warning** — a short notice period is enough to save state if you are listening for it.","**Multi-node training amplifies the risk** — losing one node stalls the whole job unless it can recover.","**Diversify instance types and zones** — reclamation correlates within a pool.","**Skip:** spot for latency-sensitive serving. Losing a serving replica mid-request is a different problem from losing a training step."],"readTime":"4 min read","wordCount":881},"related":[{"slug":"ai-compute-leasing-explained-2026","title":"AI Compute Leasing Explained: Renting GPUs at Scale","excerpt":"Neoclouds, reserved capacity, and spot GPUs all price the same silicon very differently. Here is how compute leasing works and which contract shape fits which workload.","category":"ai","date":"August 17, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":240},{"slug":"reserved-gpu-capacity-2026","title":"Reserved GPU Capacity: Committing to Get Availability","excerpt":"The discount is the advertised benefit. The real reason to commit is that on-demand GPUs are frequently unavailable when you need them, and a reservation is a queue position.","category":"ai","date":"January 16, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":217},{"slug":"distillation-pipeline-guide-2026","title":"Distillation Pipelines: Training a Small Model From a Big One","excerpt":"Distillation transfers capability from a large model to a small one for a specific task. Here is how to build the pipeline and where the quality actually lands.","category":"ai","date":"August 18, 2026","author":"ByteLedger Team","readTime":"3 min read","hue":221}],"sameCategory":[{"slug":"agent-cost-attribution-2026","title":"Agent Cost Attribution: Finding Where the Tokens Went","excerpt":"A single monthly API bill tells you that agents are expensive. It does not tell you which feature, which customer, or which retry loop is responsible. That gap is fixable.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":220},{"slug":"human-in-the-loop-agents-2026","title":"Human-in-the-Loop Agents: Where to Put the Approval Gate","excerpt":"Approve everything and the agent is a slow autocomplete. Approve nothing and you have handed irreversible actions to a system that can be talked into things.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":197},{"slug":"reasoning-effort-controls-2026","title":"Reasoning Effort Controls: Tuning How Hard a Model Thinks","excerpt":"Fixed thinking budgets gave way to effort levels, and the default is rarely the right setting. Here is how the dial behaves and where each level actually belongs.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":210},{"slug":"retrieval-metrics-explained-2026","title":"Retrieval Metrics: Measuring the Half of RAG That Fails","excerpt":"When a RAG system answers badly, the retrieval usually failed before the model ever spoke. Measuring generation quality alone hides that entirely.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":215}],"newer":{"slug":"spin-off-tax-treatment-2026","title":"Spin-Offs: What Happens to Your Basis When a Company Splits","excerpt":"A company you own spins off a division and shares in the new entity appear in your account. Usually no tax is due, and your original basis must be allocated between two holdings.","category":"finance","date":"January 2, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":287},"older":{"slug":"tensor-parallelism-explained-2026","title":"Tensor Parallelism: Splitting One Layer Across GPUs","excerpt":"When a model is too large for one GPU, you split it. Tensor parallelism divides individual layers across devices, which is fast and demands an interconnect most clusters do not have.","category":"ai","date":"January 2, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":203}}