An agent has been working for eighteen minutes. It has read forty files, made a dozen decisions, and written half the changes it planned. Then a tool call times out and the run dies.
Restarting from scratch pays for all eighteen minutes again, takes another eighteen, and — because the model is not deterministic — may not make the same decisions the second time. Worse, if it already applied half its changes, a fresh run starts from a state it does not know about.
Checkpointing is the alternative: persist enough after each step that the run can continue from where it stopped.
What changed in 2026
- Runs got long enough for this to matter. An agent working for many minutes across many steps makes restart genuinely expensive in a way a three-step run never did.
- Caching made resume cheap. Re-sending an unchanged prefix hits cache rather than re-billing, so continuing costs far less than starting over.
- Replay became a debugging standard. Re-running a failed trace with cached results to reproduce a bug turned into normal practice rather than an unusual technique.
- Side-effect tracking got attention. The realisation that resuming a run which already acted on the world is a distinct and harder problem than resuming a read-only one.
What a checkpoint contains
At minimum, the state needed to reconstruct the next request:
The message history. Every turn so far, including tool calls and results — and including any compaction blocks, or you lose the compacted state entirely.
The step index and loop state. Where in the plan the agent was, and whatever counters or budget tracking you maintain.
Applied side effects. Which actions have actually been taken against the world. This is the part most implementations omit and the part that causes real damage.
Configuration. Model, effort, tools available. Resuming with a different model against a history produced by another is a source of confusing behaviour.
Persist after each completed step rather than mid-step. A checkpoint taken between a tool call and its result leaves you resuming into an inconsistent state where the model expects a result you never captured.
Replay and resume are different
They get conflated and they serve different purposes.
|
Replay |
Resume |
| Purpose |
Reproduce a run for debugging |
Continue interrupted work |
| Model calls |
Served from a cache of prior results |
Live from the point of failure |
| Side effects |
Must be suppressed |
Must not be repeated |
| Determinism |
Exact, by construction |
Not required |
| Typical use |
Diagnosing a bad output |
Recovering from a crash |
Replay re-runs a recorded trace, serving each model call and tool result from what was captured the first time. Because nothing is live, it is exactly reproducible — which is what makes it useful for debugging a non-deterministic system where you otherwise cannot reproduce anything. Change one prompt, replay, and see whether the outcome differs.
Resume picks up live from the last checkpoint. The earlier steps are history in the message array; the next step is a real call. Since the prefix is unchanged, prompt caching means you largely pay for the new work only.
The side effect problem
This is the hard part, and it is what separates a toy checkpoint system from a usable one.
A read-only agent resumes trivially — worst case it re-reads a file. An agent that sends messages, writes to a database, calls a payment API, or merges a branch cannot simply replay its history, because the world already changed.
Three approaches, in rough order of preference.
Make actions idempotent. Give each side-effecting call a stable key derived from the run and step, so repeating it is a no-op. This is the cleanest solution and requires the downstream systems to support it — see idempotency explained.
Record applied effects in the checkpoint. Before resuming, skip any step whose effect is already marked applied. Simple and depends on the record being written atomically with the effect, which is the usual distributed-systems difficulty.
Separate planning from execution. Have the agent produce a plan, checkpoint the plan, then execute steps against it with tracking. Resume re-executes only unapplied steps. More structure, considerably more predictable — and it pairs naturally with a human approval gate over the plan, per human-in-the-loop agents.
Common mistakes
- Checkpointing mid-step. Resumes into an inconsistent history.
- Not persisting compaction blocks. The compacted state is lost and context grows unexpectedly.
- Ignoring side effects. The most common cause of duplicate emails and double writes.
- Resuming with a different model or configuration. Behaviour diverges from the history.
- Checkpointing short runs. The machinery costs more than restarting.
- Storing checkpoints without expiry. Message histories are large and accumulate quickly.
- No replay capability. Debugging a non-deterministic failure without it means guessing.
FAQ
How often should I checkpoint?
After each completed step is the usual answer. If steps are very fast and cheap, every few steps is fine. The right frequency is whatever makes the cost of losing work smaller than the cost of persisting it.
Where should checkpoints live?
Anywhere durable and fast enough not to slow the loop — object storage or a database both work. They are write-heavy and read-rarely, and message histories are large, so plan for volume and set a retention policy.
Does resume produce the same result as an uninterrupted run?
Not necessarily, and that is fine. The model is not deterministic, so the continuation may differ from what the original run would have done — see LLM determinism. What matters is that it continues sensibly from real prior state.
Can I checkpoint subagent runs?
Yes, and you need to decide whether a parent resume re-runs completed subagents or reuses their results. Reusing is cheaper and requires their outputs to be part of the parent's checkpoint.
Where to go next
For the timeouts that trigger most recovery, read agent timeouts. For making repeated actions safe, idempotency explained, and for the compaction state a checkpoint must preserve, context compaction.