The model you are serving has safety behaviour trained into it, and that behaviour is general rather than specific to your product. It will decline obviously harmful requests and has no idea that in your context discussing a competitor's pricing is prohibited, or that outputs must never mention a particular internal system.
A guard model is a separate, usually much smaller classifier that screens what goes in and what comes out against rules you define.
What changed in 2026
- Purpose-built small guard models became widely available. Dedicated classifiers displaced repurposing a general model for screening.
- Bidirectional screening became standard. Checking outputs as well as inputs stopped being optional in regulated deployments.
- Latency structure got understood. The asymmetry between input screening, which parallelises, and output screening, which does not, shaped implementations.
- False positive rates became the headline metric. Teams recognised that over-blocking damages products more visibly than occasional misses.
Screen both directions
Input and output screening catch different things.
Input screening catches prompts that should not be processed — policy violations, injection attempts, requests outside your product's scope. Blocking early saves the cost of generation entirely.
Output screening catches responses that should not be delivered. This is the direction people skip, and it is where several important failures live: a benign question producing a response containing personal data pulled from retrieval, a model repeating something from a poisoned context, or output that is fine generally and prohibited in your specific setting.
An input filter alone assumes that safe inputs produce safe outputs, which is not true in a retrieval-augmented system where the context comes from a corpus you do not fully control — see context poisoning.
The latency asymmetry
|
Input screening |
Output screening |
| Can run in parallel |
Yes, alongside generation start |
No |
| Adds to time to first token |
Only if blocking before generation |
No |
| Adds to total latency |
Little |
Full classifier latency |
| Interacts with streaming |
Cleanly |
Badly |
Input screening can overlap with generation: start both, and if the guard rejects, discard the partial generation. You pay some wasted tokens on blocked requests, which is usually a small fraction, and add nothing to latency on legitimate ones.
Output screening cannot overlap, because it needs the output. That is a straightforward addition to total latency, and it conflicts badly with streaming — you cannot stream tokens to a user and then decide the response was unacceptable.
The usual compromises: screen in chunks as they generate, accepting some risk on already-streamed text; or buffer entirely and lose the streaming experience. Which is acceptable depends on how severe an unfiltered response would be.
False positives cost more
A guard model has an error rate in both directions, and the two errors are not symmetric in product impact.
A miss lets through something that should have been blocked. Depending on your domain, this ranges from embarrassing to serious.
A false positive blocks a legitimate request. The user sees a refusal for something entirely reasonable, concludes the product is broken or paternalistic, and frequently stops using it.
Teams tune guards conservatively to minimise misses and produce systems that refuse constantly. That is a worse product than one with a slightly higher miss rate, in most non-regulated contexts.
Measure both rates on real traffic. A guard blocking a noticeable fraction of legitimate requests is doing more harm than the risk it mitigates, and that is only visible if you look at what it blocked rather than at aggregate counts.
Use rules where rules work
A guard model is a probabilistic classifier with an error rate. A regular expression matching card number formats is exact, instant, and free.
Where the property you need to check can be expressed as a rule — a format, a forbidden exact string, a length limit, a required disclaimer — use the rule. Reserve the model for judgements that genuinely require understanding, which is a smaller set than it first appears. This is the same layering as LLM output validation.
Common mistakes
- Input screening only. Misses output-side failures entirely.
- Tuning for zero misses. Produces a product that refuses constantly.
- Not measuring false positives. The dominant cost, invisible in aggregate metrics.
- A model where a rule would do. Slower, more expensive, less exact.
- Buffering all output to screen it. Loses streaming for every request.
- Using the main model as its own guard. Shares the blind spots.
- No logging of blocks. No way to tune what you cannot inspect.
FAQ
How big should the guard model be?
Small. Classification is far easier than generation, and a compact model runs fast enough to sit in the request path. A large guard defeats the economics.
Can one model screen both directions?
Usually yes, with different prompts or configurations for the two jobs. The properties you check differ, so treat them as two tasks even on one model.
Does this replace the model's own safety training?
No, it layers on top. The base model's behaviour handles general cases; the guard handles your specific policy — see refusal training.
How do I handle a block gracefully?
Explain what happened without being accusatory, and give a route forward where one exists. A bare refusal with no explanation is the worst version of a false positive.
Where to go next
For the deterministic layer that should come first, read LLM output validation. For the model's built-in behaviour, refusal training, and for the broader policy layer, AI guardrails.