Your assistant sends a two-thousand-token system prompt with every request — instructions, examples, tool definitions, style guidance. The user asks a short question and gets a short answer.
Without caching, the model processes those two thousand tokens from scratch every time. On a short exchange that is the overwhelming majority of the compute spent, repeated identically, thousands of times a day.
What changed in 2026
- It became a default rather than an optimisation. Serving frameworks and hosted APIs enabled prefix reuse broadly, often without requiring configuration.
- Longer system prompts made it more valuable. As tool definitions and instructions grew, the fraction of each request that is repeated grew with them.
- Automatic detection spread. Rather than marking cache points manually, more systems detect shared prefixes across requests.
- Cache-aware prompt design became a practice. Ordering prompts for cacheability turned into a normal part of building an LLM application.
Why prefill is the expensive part
Generation has two phases with different characteristics.
Prefill processes the entire prompt at once. It is compute-bound and highly parallel — every prompt token can be handled simultaneously.
Decode produces output one token at a time, each depending on the last. It is memory-bandwidth-bound and sequential.
For a request with a long prompt and a short response, prefill is most of the work. Prefix caching removes the repeated portion of it: if the first two thousand tokens are identical to a previous request, their computed keys and values are already in memory and can be reused directly.
The saving shows up twice — in cost, because you are not recomputing, and in time to first token, because the model can begin generating almost immediately rather than after processing the whole prompt. On interactive interfaces that latency improvement is more noticeable than the cost one.
Exact prefix matching
The rule that governs everything practical about this: matching is on an exact prefix. Any difference invalidates from that point onward.
Change one character at position 50 of a 2000-token prompt and nothing after position 50 can be reused. The cache does not resynchronise.
Which produces one clear design principle: stable content first, volatile content last.
| Position |
Put here |
Why |
| First |
System prompt, tool definitions |
Never changes |
| Middle |
Few-shot examples, retrieved context |
Changes rarely or per-session |
| Last |
Conversation history, user message |
Changes every request |
| Never early |
Timestamps, request IDs, random values |
Invalidates everything |
The common self-inflicted failure is a timestamp or session identifier near the top of a system prompt. It looks harmless and it means the cache never hits, on any request, ever. If your cache hit rate is zero, this is the first thing to check.
Tool definitions deserve attention too. If your code builds the tool list in a non-deterministic order — iterating a set, say — the serialised prompt differs between requests and the cache misses despite the content being identical. Sort them.
Where it does and does not help
Helps a lot: long stable system prompts, few-shot examples reused across requests, multi-turn conversations where the history accumulates as a growing stable prefix, agent loops where each turn re-sends everything before it.
The agent case is the strongest. An agent making twenty tool calls re-sends the entire conversation each turn, and each turn's prefix is the previous turn plus a bit. Almost all of it is cacheable, and without caching an agent loop is quadratically expensive in prompt tokens.
Helps little: short prompts, requests where nearly everything varies, and one-off requests that share no prefix with anything.
Cache entries also expire. A prefix used once an hour may not still be resident; one used constantly will be. This is why traffic volume affects hit rate independently of prompt design.
Common mistakes
- Volatile content early in the prompt. Guarantees a zero hit rate.
- Non-deterministic tool or field ordering. Identical content, different bytes.
- Not measuring hit rate. The one number that tells you whether any of this is working.
- Rebuilding the system prompt per request. String assembly that varies subtly defeats matching.
- Assuming it applies to output. Only the prompt prefix is cacheable; generation is not.
- Frequent prompt edits on a busy system. Each change invalidates the cache until it rebuilds.
FAQ
How do I know if it is working?
Usage reporting typically distinguishes cached from uncached input tokens. If the cached figure is zero across repeated similar requests, something is invalidating the prefix — see prompt caching for the audit.
Does it change the output?
No. The cached values are exactly what recomputation would produce. This is a pure efficiency mechanism.
How does it relate to paged attention?
Paging is what makes prefix sharing practical — identical prefixes point at the same physical blocks rather than duplicating them. See paged attention.
Should I restructure prompts around it?
Ordering stable content first costs nothing and is worth doing regardless. Beyond that, restructure only if you measure a poor hit rate on a workload where it should be high.
Where to go next
For the cost mechanics and cache audit, read prompt caching. For the memory system underneath, paged attention, and for the latency metric it improves, TTFT vs TPOT.