Context caching is the practice of having an LLM provider store the internal computation for a chunk of your prompt so it does not have to redo that work on the next call. If your application sends the same long system prompt, document, or set of examples on every request, caching turns that repeated cost into a one-time cost plus a much cheaper "read" on every call after. It is one of the few optimizations that lowers both spend and latency at the same time, which is why it moved from a niche API flag to a default expectation in production LLM apps.
What changed in 2026
- Caching became automatic on more providers, requiring no explicit flag for prefixes over a minimum token length, though manual control still gets better hit rates.
- Cache retention windows lengthened, with extended tiers now available for a higher write cost, useful for background agents that call the same context sporadically.
- Agent frameworks started managing caching explicitly, structuring growing transcripts so they stay cacheable instead of re-sending an unstructured blob each turn.
- Cache hit rate became a first-class dashboard metric, alongside token count and latency, as a meaningful lever for cost control.
How context caching actually works
Every LLM call runs the model attention mechanism over the full input, building an internal representation (the KV cache) of everything the model has "read" so far. Normally that state is thrown away the moment the response finishes. Context caching keeps it around, keyed to the exact token sequence, so a later call with the same prefix skips straight to the new tokens instead of recomputing the whole prefix.
The catch is that it only works on an exact-match prefix. Change a single token near the start of the cached section — a timestamp, a reordered instruction, an inserted user name — and the cache misses entirely, at full price. This is why prompt structure matters more with caching than without it: stable content (system instructions, reference documents, tool schemas) belongs at the front; volatile content (the latest message, current date, session values) belongs at the end.
When caching actually pays off
Caching is a bet: you pay more up front to write the cache, hoping to earn it back on reads. It pays off when the same prefix is reused enough times before it expires.
| Scenario |
Cacheable prefix |
Worth it? |
| Chatbot with a long system prompt, many users |
System prompt + tool definitions |
Yes — reused on every message from every user |
| Document Q&A over one long PDF, many questions |
The PDF text |
Yes — reused across every question in the session |
| Agent loop reasoning over a growing transcript |
Everything before the newest turn |
Yes, if the framework structures it correctly |
| One-off single-turn requests with unique prompts |
None |
No — nothing repeats, cache write cost is wasted |
| Prompts under the minimum cacheable length |
N/A |
No — most providers set a token floor (often 1k–2k tokens) |
Caching versus long-context and RAG
Context caching is not a substitute for choosing between a long context window and retrieval — it is a cost layer that sits underneath either choice. If you feed a model a large retrieved corpus, cache the parts that do not change between queries. If you rely on a long context window instead of retrieval, caching is often what makes that approach economically viable at all, since reprocessing tens of thousands of tokens on every turn would otherwise be prohibitively expensive. Deciding what belongs in the cacheable prefix versus the volatile tail is really a context engineering decision, not just a pricing one.
Common mistakes
Putting the user question before the reference material. This guarantees a cache miss on every call, because the model reads the variable part first. Reorder so static content comes first.
Ignoring cache expiration. Most caches expire after a short idle window. Bursty traffic with long gaps between calls will not benefit even with a correctly structured prompt.
Assuming caching reduces output cost. It only discounts the input side. A workflow generating long outputs from a short, cached prompt will not see dramatic savings.
FAQ
Does context caching change the model output?
No. Caching is a computation shortcut, not a change to what the model sees or produces. A cached and uncached call with identical input should return equivalent output (subject to normal sampling variance).
How long does a cache last?
It varies by provider and tier. Check current documentation for exact numbers before relying on a specific duration.
Is caching worth it for a low-traffic app?
Usually not. A prefix cached once and read once is a net loss, since writes cost more than a normal call.
Does caching work with tool calling?
Yes — tool definitions are often the most cache-friendly part of a prompt, since they rarely change within a session.
Where to go next