{"post":{"slug":"activation-checkpointing-2026","title":"Activation Checkpointing: Recomputing to Save Memory","excerpt":"Training holds every intermediate activation for the backward pass. Discarding most of them and recomputing on demand trades compute for a large memory saving.","category":"ai","date":"January 9, 2026","author":"ByteLedger Team","content":"<span class=\"dropcap\">T</span>raining requires the backward pass to see the intermediate values the forward pass produced. The straightforward approach stores all of them, which means memory proportional to model depth times sequence length times batch size.\n\nAt long sequence lengths that term dominates everything else — frequently exceeding the weights and optimiser state combined. Activation checkpointing attacks it directly: keep only a few values, discard the rest, and recompute what is needed during the backward pass.\n\n## What changed in 2026\n\n- **Long-context training made it mandatory.** As target sequence lengths grew, activation memory became the binding constraint for most training runs.\n- **Selective checkpointing displaced full.** Choosing which activations to keep, based on recomputation cost, replaced the all-or-nothing approach.\n- **Framework support matured.** Automatic policies that decide what to checkpoint reduced the manual tuning burden.\n- **Attention implementations reduced the pressure.** Memory-efficient attention lowered activation memory independently, which changed where checkpointing pays.\n\n## The trade\n\n| | Store everything | Full checkpointing |\n| --- | --- | --- |\n| Activation memory | Proportional to depth | Roughly proportional to its square root |\n| Forward compute | 1x | 1x |\n| Backward compute | 1x | ~2x for recomputed segments |\n| Total compute | 1x | ~1.3x typically |\n| Enables | Shorter sequences, smaller batches | Longer sequences, larger batches |\n\nRoughly a third more compute for a memory reduction large enough to change what you can train. When memory is the constraint, that is an obviously good trade — you were not using the compute anyway, because you could not fit the workload.\n\nWhen compute is the constraint, it is a bad trade. Adding 30% to training time to free memory you have spare achieves nothing. Knowing which regime you are in is the whole decision, and it is answered by checking whether you are hitting out-of-memory errors or simply want the run to finish sooner.\n\n## Selective beats full\n\nFull checkpointing discards activations uniformly and recomputes them uniformly. That is wasteful, because recomputation cost varies enormously across operations.\n\nSome activations are cheap to recompute — elementwise operations, normalisations. Some are expensive — attention, large matrix multiplications.\n\nSelective checkpointing keeps the expensive ones and discards the cheap ones. You retain most of the memory saving while paying much less recomputation, because the operations you now recompute are the ones that were cheap to begin with.\n\nModern frameworks offer automatic policies that make this choice based on measured cost. Where they do not, checkpointing at transformer block boundaries is a reasonable manual default — it is coarse, simple, and captures much of the benefit.\n\n## Composing with other techniques\n\nEach memory-reduction technique addresses different memory, which is why they stack:\n\n**Activation checkpointing** reduces activation memory during a pass.\n**[Gradient accumulation](/blog/gradient-accumulation-explained-2026)** reduces the per-pass batch, which reduces activations proportionally.\n**[Mixed precision](/blog/mixed-precision-training-2026)** reduces the size of everything stored.\n**Parameter-efficient methods** like [QLoRA](/blog/qlora-explained-2026) remove optimiser state for frozen parameters.\n**[Tensor parallelism](/blog/tensor-parallelism-explained-2026)** splits everything across devices.\n\nAll of them cost something — time, precision, or communication. The sensible order is cheapest-first: mixed precision, then adapter methods if applicable, then checkpointing, then accumulation, then parallelism. Reach for each only when the previous ones have not made the workload fit.\n\n## Common mistakes\n\n- **Checkpointing when compute-bound.** Trading the resource you lack for the one you have.\n- **Full rather than selective.** Recomputing cheap operations for no reason.\n- **Not measuring the actual slowdown.** It varies by model and hardware; assumptions are frequently wrong.\n- **Applying it before cheaper techniques.** Precision and adapters cost less.\n- **Checkpointing at bad boundaries.** Segments that are too fine add overhead; too coarse saves little.\n- **Forgetting it affects throughput comparisons.** Benchmarks with and without are not comparable.\n\n## FAQ\n\n**How much memory does it save?**\nEnough to change what fits, typically reducing activation memory substantially rather than marginally. The exact figure depends on model depth and how selective your policy is.\n\n**Does it affect training results?**\nNo — recomputed activations are identical to stored ones. Small numerical differences can appear with non-deterministic kernels, and the optimisation trajectory should match.\n\n**Should I use it for fine-tuning?**\nOnly if you are memory-constrained. Adapter fine-tuning uses far less memory to begin with, so checkpointing is frequently unnecessary there.\n\n**Does it help at inference?**\nNo. Inference does not need activations for a backward pass, so there is nothing to checkpoint. Inference memory is dominated by weights and KV cache — see [paged attention](/blog/paged-attention-explained-2026).\n\n## Where to go next\n\nFor the complementary batch technique, read [gradient accumulation](/blog/gradient-accumulation-explained-2026). For the cheapest memory saving to try first, [mixed precision training](/blog/mixed-precision-training-2026), and for training fewer parameters entirely, [QLoRA](/blog/qlora-explained-2026).","hue":213,"metaTitle":"Activation Checkpointing: Recompute to Save Memory","metaDescription":"How activation checkpointing trades recomputation for training memory, why selective checkpointing beats full, and when the trade is wrong.","keywords":["activation checkpointing","gradient checkpointing","training memory optimization","recomputation trade off","long sequence training","selective checkpointing","memory bound training","model training 2026"],"tldr":["**Activations dominate training memory at long sequences** — often more than weights and optimiser state combined.","**Discard and recompute rather than store** — roughly a third more compute for a large memory reduction.","**Selective checkpointing beats full** — keep the expensive-to-recompute activations, discard the cheap ones.","**It composes with accumulation and parallelism** — different memory, all reducible together.","**Skip:** checkpointing if you are compute-bound rather than memory-bound. You are trading the wrong direction."],"readTime":"4 min read","wordCount":776},"related":[{"slug":"flash-attention-explained-2026","title":"Flash Attention Explained: Speed From Memory Movement","excerpt":"Flash attention makes transformers faster without changing the math. Here is what it actually optimizes, why memory bandwidth was the bottleneck, and when it matters.","category":"ai","date":"June 26, 2026","author":"ByteLedger Team","readTime":"3 min read","hue":208},{"slug":"kv-cache-quantization-2026","title":"KV Cache Quantisation: More Sequences in the Same Memory","excerpt":"At long context lengths the attention cache outgrows the model weights. Storing it at lower precision buys back concurrency, at a quality cost that is usually smaller than expected.","category":"ai","date":"January 12, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":205},{"slug":"gradient-accumulation-explained-2026","title":"Gradient Accumulation: A Large Batch on Small Hardware","excerpt":"You need an effective batch size your GPU cannot hold. Accumulating gradients over several small passes before updating gets you the same result, more slowly.","category":"ai","date":"January 5, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":198}],"sameCategory":[{"slug":"agent-cost-attribution-2026","title":"Agent Cost Attribution: Finding Where the Tokens Went","excerpt":"A single monthly API bill tells you that agents are expensive. It does not tell you which feature, which customer, or which retry loop is responsible. That gap is fixable.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":220},{"slug":"human-in-the-loop-agents-2026","title":"Human-in-the-Loop Agents: Where to Put the Approval Gate","excerpt":"Approve everything and the agent is a slow autocomplete. Approve nothing and you have handed irreversible actions to a system that can be talked into things.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":197},{"slug":"reasoning-effort-controls-2026","title":"Reasoning Effort Controls: Tuning How Hard a Model Thinks","excerpt":"Fixed thinking budgets gave way to effort levels, and the default is rarely the right setting. Here is how the dial behaves and where each level actually belongs.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":210},{"slug":"retrieval-metrics-explained-2026","title":"Retrieval Metrics: Measuring the Half of RAG That Fails","excerpt":"When a RAG system answers badly, the retrieval usually failed before the model ever spoke. Measuring generation quality alone hides that entirely.","category":"ai","date":"August 19, 2026","author":"ByteLedger Team","readTime":"5 min read","hue":215}],"newer":{"slug":"qbi-deduction-explained-2026","title":"The QBI Deduction: A Break for Business Owners With Cliffs","excerpt":"A deduction of up to a fifth of qualified business income, with income thresholds, wage tests, and a category of professions treated less favourably than others.","category":"finance","date":"January 10, 2026","author":"ByteLedger Team","readTime":"4 min read","hue":295},"older":{"slug":"best-high-yield-savings-accounts-2026","title":"Best high-yield savings accounts in 2026: where to actually park your cash","excerpt":"We compared the top HYSAs in 2026 on APY, fees, FDIC limits, and the small-print details that decide which one is worth the 10 minutes to open.","category":"finance","date":"January 9, 2026","author":"ByteLedger Team","readTime":"9 min read","hue":38}}