Prompt caching is the most underused cost lever in AI engineering. Most teams know it exists, few have actually measured their cache hit rate, and almost none have restructured their prompts to maximize it. In 2026, with frontier model pricing still steep at scale, ignoring prompt caching is leaving real money on the table — often 60–90% of your input token spend.
What changed in 2026
- All major providers support it. Anthropic, OpenAI, and Google (Gemini context caching) all offer prompt caching with broadly similar mechanics. The competition pushed prices down.
- Cache write pricing normalized. Most providers charge ~125% of normal input cost for a cache write, then ~10% for cache reads. The payback period is 1–2 re-uses.
- Longer TTLs arrived. Some tiers now offer 1-hour or ephemeral-session caching, making agentic multi-turn loops dramatically cheaper.
- Tool definitions count. Large tool schemas (OpenAPI specs, function lists) are now cacheable — critical for agents with many tools.
How prompt caching works
When you send a prompt, the model runs a forward pass to compute key-value (KV) activations for every token. Caching stores those KV activations. On the next call with the same prefix, the model skips the forward pass for the cached tokens and only processes the new (non-cached) portion.
The fundamental rule: the cached prefix must be byte-for-byte identical. Even a single character change invalidates the cache for everything after that point.
Prompt structure for maximum cache hits
The golden rule is static content first, dynamic content last.
[SYSTEM PROMPT — static] ← cacheable
[RETRIEVED DOCUMENTS — static] ← cacheable
[FEW-SHOT EXAMPLES — static] ← cacheable
[TOOL DEFINITIONS — static] ← cacheable
[CONVERSATION HISTORY] ← partially cacheable
[CURRENT USER QUERY — dynamic] ← always recomputed
Most teams have this backwards — they put user context at the top and instructions at the bottom. Flip it.
Provider comparison (2026)
| Provider |
Cache cost (read) |
Cache cost (write) |
Default TTL |
Notes |
| Anthropic Claude |
~10% of input |
~125% of input |
5 min (extendable) |
Explicit cache_control markers |
| OpenAI GPT-4o |
~50% of input |
Normal input cost |
5–60 min |
Automatic, prefix-based |
| Google Gemini |
~25% of input |
~100% of input |
1 hour default |
Explicit context caching API |
OpenAI's is automatic (no code changes) but less controllable; Anthropic's is explicit and gives you fine-grained cache breakpoints.
How to measure cache effectiveness
Track these metrics in your logging layer:
- Cache hit rate = cached_input_tokens / total_input_tokens
- Effective cost per call = (cached_tokens × cache_read_price) + (new_tokens × full_price)
- Cache write amortization = how many reads needed to break even on a write (~2 for most providers)
A healthy cache hit rate for a RAG application with a stable system prompt is 70–85%. Below 50% means your prompt structure needs work.
How to pick the right caching strategy
- Identify your largest stable prefix. System prompt + document context is usually 60–80% of your tokens.
- Pin that prefix with explicit cache markers (Anthropic) or rely on automatic detection (OpenAI).
- Batch calls that share the same stable prefix within the cache TTL window.
- Use session-level caching for multi-turn agents — keep the same conversation object alive.
- Pre-warm by sending a cache-write call before a burst of requests hits.
Common mistakes
Putting dynamic data in the cached prefix. A timestamp, user ID, or request-specific UUID in your system prompt means zero cache hits. Move all dynamic content to the end.
Not accounting for the write cost. Cache writes cost more than regular calls. Don't cache prompts you only send once — it costs more, not less.
Ignoring tool definitions. If you're using an agent with 10 tools, each with a verbose OpenAPI schema, those can be 2,000+ tokens per call. Cache them.
Misreading usage dashboards. Some providers show token counts pre-cache-expansion. Make sure you're reading the actual billed tokens, not the raw counts.
Assuming same cache across models. Cache is per-model-version. Upgrading from claude-sonnet-4-6 to a newer snapshot invalidates all cached prefixes.
What to skip
- Caching conversation history beyond a few turns — old turns drift out of TTL; only cache the static system prefix reliably.
- Manual prefix hashing — let the provider handle cache key logic; trying to implement it yourself adds bugs.
- Over-engineering for marginal tokens — focus on the big stable blocks (system prompt, docs) first; micro-optimizing 50-token snippets isn't worth it.
FAQ
Does caching affect output quality?
No. Caching only affects the input processing step. The model generates the same way regardless of whether tokens were cached.
Can I cache across different users?
Yes, if they share the same prompt prefix — which is exactly the case for a shared system prompt. User-specific content should be at the end, outside the cache block.
What breaks the cache?
Any byte-level change before the cache breakpoint: whitespace, punctuation, encoding differences, version numbers in prompts.
Is caching available on all tiers?
Most providers enable it on paid tiers. Free/tier-1 plans may have limited or no caching. Check your plan's usage documentation.
Where to go next