A chat interface looks like a sequence of independent exchanges. It is not. The model has no memory between calls, so every turn re-sends the entire conversation so far.
Turn one sends the system prompt and one message. Turn ten sends the system prompt plus nine previous exchanges plus the new message. The input grows every turn, and the cost of a conversation is the sum of all those growing inputs.
What changed in 2026
- Caching made long conversations affordable. Without prefix caching, chat economics at depth are poor; with it, they are manageable — see prefix caching.
- Compaction moved into providers. Server-side history summarisation gave a second lever beyond caching — see context compaction.
- Conversation-level measurement spread. Teams stopped measuring per message and started measuring per conversation.
- Agent loops inherited the same shape. An agent turn is structurally a conversation turn, so the same growth applies.
The quadratic shape
For a conversation of N turns with roughly equal-sized messages, total input tokens scale with the square of N. Ten turns costs far more than twice five turns.
| Turn |
Input sent |
Cumulative input |
| 1 |
System + 1 message |
1 unit |
| 5 |
System + 9 messages |
~15 units |
| 10 |
System + 19 messages |
~55 units |
| 20 |
System + 39 messages |
~210 units |
The exact numbers depend on message length; the shape does not. Output tokens grow linearly, so at depth the input term dominates completely — a long conversation's cost is overwhelmingly what you sent, not what you received.
That is why a chat product's cost is not proportional to messages sent. It is closer to proportional to the square of typical conversation length, which is a different and much less comfortable relationship.
What bounds it
Caching is the primary defence and the reason chat products work economically. The prefix at turn ten is the conversation at turn nine, which was just sent — so almost all of it is cacheable. Cached input costs a fraction of fresh input, converting the quadratic term into something far smaller.
This makes cache hit rate a direct economic metric for a chat product rather than a technical detail. Anything that breaks caching — volatile content early in the prompt, non-deterministic ordering — has a compounding cost effect at depth.
Compaction bounds the growth rather than discounting it. Summarising older turns caps how large the prefix can become, at the cost of a summarisation call and some detail loss. For very long conversations it is the only thing that keeps the prefix bounded at all.
Trimming is the crude version: drop the oldest turns beyond a window. Cheap, and it loses information abruptly rather than gracefully, and it invalidates the cache prefix every time the window slides — which can cost more than it saves.
The distribution matters more than the mean
Conversation lengths are heavily skewed. Most conversations are short; a small fraction run very long.
Because cost grows quadratically, that small fraction contributes disproportionately — frequently a majority of total spend from a minority of conversations. An average conversation cost computed across all conversations therefore describes almost none of your spend.
The practical measurement is cost by conversation-length bucket, and the share of total spend each bucket represents. That immediately shows whether your problem is many short conversations or a few very long ones, and those need different responses.
It also informs product decisions. If a small number of extremely long conversations dominate, a soft limit, a summarisation prompt, or an encouragement to start a new conversation may be worth more than any technical optimisation.
Common mistakes
- Measuring cost per message. Hides the quadratic shape entirely.
- Averaging across all conversations. The tail dominates and disappears in the mean.
- Breaking the cache prefix. Compounds at depth.
- Trimming with a sliding window. Invalidates the cache on every slide.
- Unbounded conversation length. No ceiling on per-conversation cost.
- Ignoring the system prompt size. It is re-sent every turn; length there is multiplied.
FAQ
How long is too long?
Depends on your economics, and the shape means the answer arrives faster than intuition suggests. Plot cost against turn count for your own product; the point where it becomes uncomfortable is usually visible.
Should I limit conversation length?
A soft limit — suggesting a fresh conversation, or compacting automatically — is usually better than a hard cut-off. Users rarely need the full history and often benefit from a summary.
Does the system prompt matter much?
More than anywhere else, because it is re-sent every turn. A thousand tokens of system prompt in a twenty-turn conversation is twenty thousand tokens of input, and it is the most cacheable part of the request.
Does this apply to agents?
Directly. An agent loop is a conversation where the other party is a set of tools, and tool results are frequently larger than chat messages — so the growth is steeper — see ReAct agents.
Where to go next
For the mechanism that makes this affordable, read prefix caching. For bounding growth at depth, context compaction, and for the broader picture, LLM unit economics.