LLM bills sneak up on every team — a prototype that costs $20/month becomes a $4,000/month production system overnight once real users arrive. The good news is that 2026's tooling makes serious cost reduction achievable without sacrificing output quality, as long as you measure the right things and pull the right levers. Here's the full playbook.
What changed in 2026
- Prompt caching is now universal. Anthropic, OpenAI, and Google all offer automatic or explicit caching on prefix tokens. A repeated system prompt costs ~10% of the original on cache hit. Teams that ignore this are leaving the biggest single saving on the table.
- Model tiers widened dramatically. The gap between flagship and mini/flash/haiku models is narrower on quality, and wider on cost — making intelligent routing a genuine 10× lever.
- Speculative decoding and distillation made self-hosted inference 40–60% cheaper than 2025 for teams at enough volume to justify the infrastructure.
- Token-level billing transparency improved: most providers now surface exact cache hit rates and prompt/completion breakdowns per request.
The cost driver map
Before optimizing, know where your tokens go:
| Driver |
Typical share |
Easiest fix |
| System prompt repeated per call |
30–60% of input |
Prompt caching |
| Retrieved context (RAG chunks) |
20–40% of input |
Smaller chunks, reranking |
| Completion length |
100% of output |
Constrain format, structured output |
| Model tier overkill |
n/a (unit cost) |
Routing to cheaper model |
| Synchronous calls with batch workloads |
n/a (rate premium) |
Async batch API |
Prompt caching
Cache-friendly prompts keep the static prefix (system prompt + examples + retrieved docs that don't change call-to-call) at the front, and put variable content at the end. On Anthropic Claude, use cache_control: {type: "ephemeral"} on the prefix blocks. On OpenAI, automatic prompt caching fires when the prefix matches a prior call. Savings: 50–80% of input token costs on high-reuse workloads. See Prompt caching explained in 2026 for implementation details.
Model routing
Not every task deserves a frontier model. A well-implemented router classifies the incoming request and dispatches accordingly:
| Task type |
Right model tier |
Cost relative to flagship |
| Simple classification / extraction |
Flash / Haiku / Mini |
~5% |
| Short-form summarization |
Flash / Haiku / Mini |
~5–10% |
| Code generation (under 200 lines) |
Mid-tier (Sonnet, GPT-4o) |
~20–30% |
| Complex reasoning, long context |
Frontier (Opus, GPT-4o full) |
100% |
| Bulk data transformation |
Batch + mini |
~2–5% |
Build the router as a tiny classifier call or a keyword heuristic — the router itself should cost under 0.1% of total spend.
Batching and async workloads
If a call doesn't need a real-time response (nightly enrichment, report generation, bulk embeddings), use the async batch API. OpenAI Batch API and Anthropic Message Batches both price at 50% of the sync rate with up to 24-hour turnaround. For embeddings specifically, always batch — single-document embedding calls waste 80% of the API round-trip time.
Context window discipline
Every token in the context window costs money. Strategies:
- Reranker before context stuffing — send 20 retrieved chunks through a cross-encoder reranker, keep the top 4. Cuts RAG context by 80% with minimal quality loss.
- Summarize conversation history rather than appending it indefinitely. After 5 turns, summarize the prior turns into 200 words.
- Structured output constraints reduce completion length by 30–50% — if you need JSON, say so.
How to pick the right lever
- Profile first. Log prompt tokens, completion tokens, model, and latency per call for 1 week. You'll see the 20% of call types that drive 80% of cost.
- Apply caching to any call where the first 1,000+ tokens repeat across requests.
- Route the high-volume, low-complexity calls to a cheaper tier. Run evals on a sample to confirm quality holds.
- Batch everything that isn't user-facing synchronous.
- Then tune context size, completion constraints, and consider self-hosting if volume justifies it (typically >$20k/month API spend).
Common mistakes
Optimizing tokens before understanding task distribution. If 60% of your cost is one nightly report, no amount of per-call micro-optimization on the chat endpoint matters.
Switching models without evals. A 10× cheaper model that degrades task success by 15% is a false economy. Run evals on your real tasks before routing.
Ignoring completion costs. Input tokens get all the attention, but on verbose tasks, completion tokens dominate. Structured output and explicit length limits pay back here.
Manual cost tracking. Costs change monthly. Use a provider dashboard or an observability tool (Langfuse, LangSmith, Helicone) to alert on spend anomalies.
What to skip
- Token compression libraries that mangle your prompt to save 5% — the quality risk outweighs it for most cases.
- Self-hosting before you hit $20k+/month in API spend — the infra overhead usually costs more than it saves below that threshold.
- Prompt trimming by hand instead of building a proper reranker — unscalable and fragile.
FAQ
How much can I realistically save?
Teams that implement caching + routing typically cut costs 60–80% without measurable quality regression. The floor is usually 40% with minimal effort.
Does prompt caching work across different users?
Only for the shared prefix — the system prompt and fixed examples. Per-user context is always charged fresh.
When does self-hosted inference make sense?
At ~$20k+/month in API bills, open-weight models (Llama 3.3, Mistral Large) on dedicated GPUs start penciling out. Below that, managed APIs win on total cost of ownership.
How do I measure quality while cutting costs?
Build a regression eval suite on your real tasks before changing anything. Run it after every routing change. See AI evals guide in 2026 for setup.
Where to go next