Training requires the backward pass to see the intermediate values the forward pass produced. The straightforward approach stores all of them, which means memory proportional to model depth times sequence length times batch size.
At long sequence lengths that term dominates everything else — frequently exceeding the weights and optimiser state combined. Activation checkpointing attacks it directly: keep only a few values, discard the rest, and recompute what is needed during the backward pass.
What changed in 2026
- Long-context training made it mandatory. As target sequence lengths grew, activation memory became the binding constraint for most training runs.
- Selective checkpointing displaced full. Choosing which activations to keep, based on recomputation cost, replaced the all-or-nothing approach.
- Framework support matured. Automatic policies that decide what to checkpoint reduced the manual tuning burden.
- Attention implementations reduced the pressure. Memory-efficient attention lowered activation memory independently, which changed where checkpointing pays.
The trade
|
Store everything |
Full checkpointing |
| Activation memory |
Proportional to depth |
Roughly proportional to its square root |
| Forward compute |
1x |
1x |
| Backward compute |
1x |
~2x for recomputed segments |
| Total compute |
1x |
~1.3x typically |
| Enables |
Shorter sequences, smaller batches |
Longer sequences, larger batches |
Roughly a third more compute for a memory reduction large enough to change what you can train. When memory is the constraint, that is an obviously good trade — you were not using the compute anyway, because you could not fit the workload.
When compute is the constraint, it is a bad trade. Adding 30% to training time to free memory you have spare achieves nothing. Knowing which regime you are in is the whole decision, and it is answered by checking whether you are hitting out-of-memory errors or simply want the run to finish sooner.
Selective beats full
Full checkpointing discards activations uniformly and recomputes them uniformly. That is wasteful, because recomputation cost varies enormously across operations.
Some activations are cheap to recompute — elementwise operations, normalisations. Some are expensive — attention, large matrix multiplications.
Selective checkpointing keeps the expensive ones and discards the cheap ones. You retain most of the memory saving while paying much less recomputation, because the operations you now recompute are the ones that were cheap to begin with.
Modern frameworks offer automatic policies that make this choice based on measured cost. Where they do not, checkpointing at transformer block boundaries is a reasonable manual default — it is coarse, simple, and captures much of the benefit.
Composing with other techniques
Each memory-reduction technique addresses different memory, which is why they stack:
Activation checkpointing reduces activation memory during a pass.
Gradient accumulation reduces the per-pass batch, which reduces activations proportionally.
Mixed precision reduces the size of everything stored.
Parameter-efficient methods like QLoRA remove optimiser state for frozen parameters.
Tensor parallelism splits everything across devices.
All of them cost something — time, precision, or communication. The sensible order is cheapest-first: mixed precision, then adapter methods if applicable, then checkpointing, then accumulation, then parallelism. Reach for each only when the previous ones have not made the workload fit.
Common mistakes
- Checkpointing when compute-bound. Trading the resource you lack for the one you have.
- Full rather than selective. Recomputing cheap operations for no reason.
- Not measuring the actual slowdown. It varies by model and hardware; assumptions are frequently wrong.
- Applying it before cheaper techniques. Precision and adapters cost less.
- Checkpointing at bad boundaries. Segments that are too fine add overhead; too coarse saves little.
- Forgetting it affects throughput comparisons. Benchmarks with and without are not comparable.
FAQ
How much memory does it save?
Enough to change what fits, typically reducing activation memory substantially rather than marginally. The exact figure depends on model depth and how selective your policy is.
Does it affect training results?
No — recomputed activations are identical to stored ones. Small numerical differences can appear with non-deterministic kernels, and the optimisation trajectory should match.
Should I use it for fine-tuning?
Only if you are memory-constrained. Adapter fine-tuning uses far less memory to begin with, so checkpointing is frequently unnecessary there.
Does it help at inference?
No. Inference does not need activations for a backward pass, so there is nothing to checkpoint. Inference memory is dominated by weights and KV cache — see paged attention.
Where to go next
For the complementary batch technique, read gradient accumulation. For the cheapest memory saving to try first, mixed precision training, and for training fewer parameters entirely, QLoRA.