Token prices are published and easy to compare. They are also nearly useless on their own, because the question that determines whether your product works is not what a token costs but what a user costs.
The gap between those is filled with fan-out, retries, evaluation traffic, and a usage distribution that is far more skewed than most people assume.
What changed in 2026
- Agentic features broke naive cost models. One user action triggering dozens of model calls made per-token reasoning obviously inadequate.
- Usage-based pricing spread. More AI products moved away from flat unlimited plans after encountering the tail.
- Caching became a first-order economic factor. The difference between cached and fresh input tokens changed feature viability.
- Cost attribution matured. Tagging spend by feature and customer became standard rather than aspirational — see agent cost attribution.
Build the unit from the action
Start with a user-visible action — sending a message, generating a report, running an analysis — and count everything it triggers.
| Component |
Frequently counted |
Often missed |
| Primary model call |
Yes |
— |
| Retrieval and embedding |
Sometimes |
— |
| Reranking |
Rarely |
Yes |
| Subagent or tool-loop calls |
Rarely |
Yes |
| Guard model screening |
Rarely |
Yes |
| Retries on failure |
Rarely |
Yes |
| Evaluation sampling |
Almost never |
Yes |
| Abandoned or cancelled work |
Almost never |
Yes |
The right column is where cost models go wrong. An agentic action with a handful of subagent calls, a reranker per retrieval, a guard model on input and output, and occasional retries can be an order of magnitude above the primary call everyone budgeted for.
Evaluation traffic in particular scales with the traffic it measures, so it grows exactly when you least want a surprise — see online evals.
The distribution is skewed
Average cost per user is a comforting and misleading number, because usage is not normally distributed.
A small fraction of users generate a large share of consumption. In agentic products the skew is more pronounced, because a heavy user does not just make more requests — their requests tend to be larger, longer-running, and more likely to fan out.
Two implications. Flat unlimited pricing is a bet on the tail, and on an agentic product that bet is frequently lost. And provisioning on average usage under-provisions for peak, since the heavy users are also concentrated in time.
Model the distribution rather than the mean. The numbers that matter are median cost per user, the cost at the high percentiles, and what fraction of total spend the top decile represents.
Caching changes the answer
A cached prompt prefix costs a small fraction of fresh input. On a workload with a long stable system prompt and short user messages, that is the difference between viable and not.
Agent loops benefit most, because each turn re-sends everything before it. Without caching, an agent loop's prompt cost grows roughly quadratically with turns; with it, most of each turn's prefix is cached — see prefix caching.
Which means a cost model built without caching can be wrong by a large multiple, in the favourable direction. It also means anything that breaks caching — a timestamp in the system prompt, non-deterministic tool ordering — has a direct and substantial cost consequence.
Batch endpoints are the other large lever: roughly half price for anything with no user waiting, which covers evaluation, indexing, and enrichment — see batch APIs.
Common mistakes
- Reasoning in tokens rather than actions. Wrong unit.
- Counting only the primary call. Misses the majority in agentic products.
- Averaging over a skewed distribution. Understates exposure.
- Ignoring evaluation traffic. Scales with what it measures.
- Modelling without caching. Wrong by a large factor.
- Flat pricing on agentic features. Unbounded exposure to the tail.
- No per-feature attribution. Cannot tell which feature is unprofitable.
FAQ
How do I find my real cost per action?
Tag every model call with the user action that triggered it, then aggregate by trace. Without a trace ID linking fan-out back to one action, this is not reconstructable — see agent cost attribution.
What margin should I target?
That is a business question rather than a technical one. The technical requirement is knowing the number, which most teams do not, and knowing how it moves with usage.
Should I cap usage?
Some limit is prudent on anything agentic, and it need not be visible — a per-user budget that almost nobody reaches protects against abuse and runaway loops without affecting normal users.
How do costs change as I scale?
Per-unit costs typically fall through caching hit rates improving with volume and through committed pricing. Total cost still grows with usage, so unit economics matter more at scale rather than less.
Where to go next
For measuring where spend goes, read agent cost attribution and cost per conversation. For the largest cost lever available, prefix caching.