The old way to get JSON from a model was to ask nicely, parse the result, and retry when it failed. It failed often enough — a trailing comma, a stray explanatory sentence, a truncated object — that most production pipelines grew a retry loop and a repair function.
Constrained decoding eliminates the problem at the source. At each generation step, the sampler is shown only the tokens that could continue a valid document under your schema. Everything else is masked out. Invalid output is not caught and corrected; it is never generated.
What changed in 2026
- Schema-guaranteed modes became standard. Structured output support moved from an experimental feature to a baseline capability across major providers and open-source serving stacks.
- Grammar support broadened past JSON. Constraining generation to arbitrary formal grammars — not just object schemas — became available in more runtimes.
- The quality tradeoff got documented. Practitioners established that heavy constraint can reduce answer quality, which shifted advice toward simpler schemas rather than more elaborate ones.
- Retry logic started disappearing. Pipelines built around parse-and-retry began removing that machinery, which cut both latency and token spend.
What it does and does not guarantee
| Property |
Guaranteed |
| Output parses as valid JSON |
Yes |
| All required fields present |
Yes |
| Field types match the schema |
Yes |
| Enum values are from the allowed set |
Yes |
| Values are factually correct |
No |
| Values are relevant to the input |
No |
| The model refused appropriately |
No |
| Reasoning quality is preserved |
No; can degrade |
That table is the whole picture. Constrained decoding is a syntax guarantee, and it is a complete one. It says nothing about whether the number in the field is the right number. Teams that adopt it sometimes relax their validation because parsing never fails any more, which is exactly backwards — the parse errors were catching some semantic problems as a side effect, and now they do not.
Where it hurts
Forcing structure too early constrains the model's ability to work through a problem. If your schema requires the answer field first and the reasoning field second, the model must commit to an answer before it has articulated why — and quality drops measurably on anything requiring thought.
The fix is ordering: put reasoning fields before conclusion fields in the schema, so the model generates its working first. This costs nothing and often recovers most of the gap.
Deeply nested schemas are the other problem. Each level of nesting adds constraint that the model must satisfy while also producing sensible content, and quality degrades as the structure gets more demanding. A flat object with ten fields generally produces better values than a three-level tree with the same information.
For very hard reasoning, consider generating freely and extracting structure in a second cheap call. That costs an extra call and frequently produces better answers — the tradeoff is worth measuring on your evaluation set rather than assuming. See eval-driven development for AI for how to run that comparison properly.
Common mistakes
- Relaxing validation because parsing cannot fail. Semantic validation still matters and now has to be explicit.
- Putting the conclusion field first. Reasoning fields belong before the answer they support.
- Very deep nesting. Flatten where you can; quality tracks schema simplicity.
- Requiring every field. Optional fields let the model signal uncertainty instead of fabricating a value.
- Using enums with too many options. A long enum is a constraint the model satisfies by picking something, not necessarily the right thing.
FAQ
Does constrained decoding slow generation down?
Slightly, from computing the valid token mask at each step. In most implementations the overhead is small relative to the model forward pass, and it is more than offset by eliminating retries.
Does it work with any schema?
Providers support a subset of JSON Schema, and the supported subset varies. Check the specific documentation; unsupported constructs are usually ignored rather than enforced.
Can I constrain to formats other than JSON?
Grammar-based constraint supports arbitrary formal grammars in several runtimes, which covers structured text, code, and domain-specific formats.
Does it prevent hallucination?
No. It prevents malformed output. A hallucinated value in a correctly typed field passes every check.
Where to go next
For the surrounding practice, read structured outputs guide. For measuring whether the constraint is helping or hurting, eval-driven development for AI, and for testing the boundaries, LLM guardrail testing.