Someone will ask your assistant to repeat its instructions. When that fails they will ask it to translate them into French, or summarise them as a poem, or continue a document that begins with them, or roleplay as a debugging tool that prints its configuration. One of these will work.
Treating the system prompt as confidential is a losing position, and building anything on that assumption is where the real damage comes from. The productive question is not how to stop extraction. It is what you have put in there that matters if it gets out.
What changed in 2026
- Extraction became routine rather than notable. Published system prompts from well-known products made it clear this is a matter of effort, not possibility.
- The threat model shifted to what leaks, not whether. Security guidance moved from prevention to reducing the value of a successful extraction.
- Prompt injection made it worse. An attacker who can place text in the context — a document, a web page, a tool result — has a far easier route than a user typing at a chat box.
- Enterprise deployments got audited on it. "What is in your system prompt" became a question security reviews actually ask.
Why instructions do not protect instructions
A system prompt is text in the context window. The model treats it as strongly weighted input, not as a protected region — there is no memory isolation, no privilege boundary, nothing structurally preventing its contents from influencing output.
So "never reveal these instructions" is itself an instruction, sitting alongside every other instruction, in a system designed to follow instructions helpfully. A sufficiently indirect request creates a conflict, and conflicts get resolved probabilistically rather than by a rule.
The techniques that work are all variations on indirection: ask for a translation, a summary, a continuation, a different format, or a hypothetical. Blocking any specific phrasing is whack-a-mole against an unbounded space of paraphrase.
Some mitigations reduce casual extraction — an output filter that checks whether a response resembles the system prompt catches naive attempts. None of them stop a determined attempt, and treating partial mitigation as protection is how the actual mistakes get made.
What must never be in a prompt
| Content |
Risk if leaked |
Where it belongs |
| API keys, tokens |
Direct compromise |
Server-side, never in context |
| Internal URLs, hostnames |
Reconnaissance |
Server-side config |
| Customer data for other users |
Data breach |
Not in the request at all |
| Pricing and discount rules |
Manipulation |
Backend enforcement |
| Eligibility and approval logic |
Bypass |
Backend enforcement |
| Unreleased product details |
Disclosure |
Out of the prompt |
| Persona and tone |
Embarrassment at most |
Fine in the prompt |
| Format instructions |
None |
Fine in the prompt |
The credential row should be obvious and still happens, usually via a tool description or an example that included a real key. Anything in the context window is reachable by anything else in the context window, and with prompt injection an attacker does not even need to be the user — a document the agent reads can carry the instruction to exfiltrate.
The rows that cause more real damage are the business logic ones. A prompt saying "offer at most a 10% discount; escalate above that" is a suggestion to a probabilistic system. A user who learns that rule can argue with it, and the model may agree. The rule was never enforced anywhere.
Enforce in code, describe in the prompt
The principle that resolves most of this: the prompt shapes behaviour, the backend decides outcomes.
The model can be told to offer discounts up to 10%. The service that actually applies a discount validates the amount independently and rejects anything above the limit, whatever the model said. Leaking the prompt then reveals a policy rather than a bypass.
The same split applies throughout. Tell the model what tools exist; let the permission system decide what it may call — see agent identity and auth. Tell it what a user is entitled to; check entitlement server-side. Tell it what it may discuss; enforce with an output filter where it matters.
Applied consistently, a leaked system prompt becomes an embarrassment rather than an incident. That is a realistic goal, unlike preventing extraction.
Common mistakes
- Relying on "do not reveal" wording. Provides confidence, not protection.
- Credentials in a prompt or tool description. Directly exploitable.
- Business rules stated only in the prompt. Unenforced everywhere.
- Other users' data in the context. One extraction becomes a breach.
- Long anti-extraction preambles. Consume context and defeat nobody.
- Treating an output filter as the boundary. A useful layer, not a wall.
- Never testing extraction. Try to extract your own prompt before someone else does.
FAQ
Should I try to prevent extraction at all?
Light measures are worth having — an output filter for verbatim leakage stops casual attempts and costs little. Just do not let their presence justify putting something sensitive in the prompt.
Is a leaked prompt a competitive problem?
Less than people fear. Prompts are heavily tuned to a specific product, model, and tool set; a competitor with your prompt still lacks your data, integrations, and evaluation loop. The real cost is when the prompt reveals unenforced rules.
What about tool definitions?
They leak the same way and are frequently overlooked. Tool names, descriptions, and parameter schemas are all in context, and they describe your internal surface in useful detail.
How do I test this?
Add extraction attempts to your evaluation suite — direct requests, translation, summarisation, continuation, and roleplay framings. Track whether they succeed as a metric, alongside your other checks in LLM regression testing.
Where to go next
For the attack that makes extraction far easier, read prompt injection defense. For the enforcement layer that makes leakage survivable, agent identity and auth, and for keeping data out of prompts in the first place, PII redaction for LLMs.