A production system prompt accumulates. Someone adds an instruction to fix a bug, someone else adds an example, a third person adds a clarification to the clarification. Nobody removes anything, because removal risks breaking a behaviour whose origin nobody remembers. Two years later the prompt is long, partly contradictory, and billed on every single request.
Compression is the deliberate reversal of that accumulation, and it pays twice — in tokens and in accuracy.
What changed in 2026
- Prompt caching reduced the urgency for static prefixes. With cached prefixes discounted heavily, the cost argument weakened for the fixed part of a prompt while the accuracy argument remained.
- Automated compression tools matured. Methods that drop low-information tokens while preserving meaning became usable, with the caveat that they suit data better than instructions.
- Example pruning got recognized as the main lever. Teams found that few-shot examples were where most removable tokens lived.
- Context rot strengthened the accuracy case. Wider understanding that irrelevant context actively hurts accuracy, per context rot explained, made compression a quality practice rather than only a cost one.
What compresses and what does not
| Content |
Compressible |
Notes |
| Verbose instructions |
Yes, substantially |
Rewrite tightly; same meaning, fewer words |
| Redundant instructions |
Yes; delete |
Often several say the same thing |
| Few-shot examples |
Yes; prune |
Fewer, better-chosen examples usually win |
| Formatting boilerplate |
Yes |
Use a schema instead of describing the format |
| Retrieved documents |
Carefully |
Summarizing risks dropping the needed detail |
| User input |
No |
Never paraphrase what the user actually said |
| Business rules and constraints |
No |
Precision matters more than length |
| Legal or compliance text |
No |
Verbatim for a reason |
Examples are where the easy savings are. Teams add examples to fix specific failures and never revisit whether they still help. Removing them one at a time while running an evaluation set frequently reveals that half contribute nothing measurable — they were fixing a problem a later model handles natively.
Doing it safely
Measure, do not read. A prompt that looks equivalent to a human may not behave equivalently, and the only way to know is running your evaluation set before and after. This is the same discipline as prompt A/B testing applied to a change intended to be neutral.
Compress one thing at a time. Removing three examples and rewriting the instructions simultaneously means an observed regression cannot be attributed. Sequential changes with a measurement between each is slower and actually tells you something.
Replace format descriptions with structure. A paragraph explaining the required output shape is tokens spent describing something a schema enforces directly and more reliably — the mechanism in constrained decoding explained.
Restructure for caching rather than only shortening. Moving all static content to the front and variable content to the end lets prompt caching cover the fixed portion, which addresses the cost side without touching the text at all.
Be careful with automated compression on instructions. Tools that drop low-information tokens work reasonably on documents and can quietly delete a negation or a condition in a rule. Use them on data, review them on instructions.
Common mistakes
- Compressing by eye. Looks equivalent, behaves differently.
- Changing several things at once. No attribution when something breaks.
- Paraphrasing user input. Never do this; the user's exact words matter.
- Automated compression on business rules. A dropped negation inverts a rule.
- Keeping every example forever. The largest easy saving, routinely untouched.
- Optimizing tokens while ignoring cache structure. Reordering may save more than rewriting.
FAQ
How much can I usually cut?
On a prompt that has accumulated for a while, often a substantial fraction with no behavioural change. The savings come mostly from examples and redundant instructions.
Does prompt caching make this unnecessary?
It addresses cost for the static portion and not the accuracy cost of irrelevant context, nor the variable portion.
Should I compress retrieved documents?
Cautiously. Better retrieval that fetches less is preferable to summarizing what you fetched, since summarization can drop the specific detail the question needed.
What about compressing conversation history?
That is a different technique with its own design — see context compression.
Where to go next
For conversation-level compaction, read context compression. For measuring changes, prompt A/B testing, and for the cost model, token counting explained.