Streaming a text response is easy — append each chunk and display it. Streaming a structured response is not, because the intermediate states are not valid documents. {"title": "Quarterly rep is what you have after a few chunks, and no parser accepts it.
The naive response is to buffer everything and parse at the end, which discards the entire benefit of streaming and leaves the user watching a spinner while data is already arriving.
What changed in 2026
- Structured outputs made this common. Once schema-constrained generation became reliable, streaming structured responses became a normal product requirement.
- Repair-based parsing became the standard approach. Completing partial documents and parsing the result displaced hand-written incremental parsers.
- Field-level streaming emerged in tooling. Libraries that emit completed fields as they finalise reduced the amount of custom code needed.
- Agent UIs raised the stakes. Showing tool arguments as they stream became a common interface pattern.
Repair, then parse
The technique that works and is much simpler than writing an incremental parser: take the partial text, close whatever is open, and parse the result.
Track the structure as you go — which brackets and braces are unclosed, whether you are inside a string, whether an escape is pending — then append the closing characters needed to make it syntactically complete. Parse that with an ordinary parser.
You get a valid object representing everything received so far, with the incomplete tail either truncated or present as a partial value. Recompute it on each chunk; the cost is negligible relative to the model call producing the tokens.
Edge cases that need care: a partial escape sequence inside a string, a partial Unicode escape, a trailing comma where the next element has not arrived, and a number that may still gain digits. Each is handled by dropping the incomplete trailing element rather than trying to interpret it.
Which fields are safe to show
The critical distinction: a field whose value is fully received will not change; a field still arriving will.
| State |
Safe to render? |
| Earlier fields, complete |
Yes |
| Current field, string mid-arrival |
Only if you accept it growing |
| Current field, number mid-arrival |
No — digits may follow |
| Array elements before the last |
Yes |
| The last array element |
No, until the next one starts |
| Any field not yet started |
No |
The rule that follows: emit the stable prefix, buffer the volatile tail. Everything before the currently-arriving element is final. The current element is not.
For a string being appended to, growing text in place is usually acceptable and reads naturally — that is what text streaming already looks like. For a number, it is not: rendering 4 and then 42 and then 420 shows the user three different wrong values.
So the pragmatic policy is to stream string values character by character where that suits the interface, and to hold everything else until complete.
When not to stream structured output
If the whole response is small, streaming buys nothing. The complexity of partial parsing is only worth it when the response is large enough that waiting is noticeable.
If the consumer is code rather than a person, streaming is usually pointless — the code cannot act on a partial object anyway, so waiting for completion is simpler and safer.
And if any field requires validation before use, streaming it to a user before validation means potentially showing something you will then reject. Validate first for anything where a wrong value has consequences — see LLM output validation.
The strongest case for streaming structured output is a long response where early fields are independently useful: a report with sections, a list that grows, or an agent's tool arguments shown as they form.
Common mistakes
- Parsing each chunk with a standard parser. Fails until the final chunk.
- Rendering incomplete numbers. Shows a sequence of wrong values.
- Rendering the last array element. It may still be growing.
- Buffering everything. Discards the benefit of streaming.
- Not handling partial escapes. A common crash in repair logic.
- Streaming to code. No benefit; adds complexity.
- Showing unvalidated values. May be retracted.
FAQ
Does schema-constrained generation guarantee valid partial output?
It guarantees the final output is valid. Intermediate states are still incomplete documents, so partial parsing is still required.
Should I write my own repair logic?
Libraries exist and handle the edge cases. Writing it yourself is instructive and prone to the same partial-escape bugs everyone hits.
How often should I re-parse?
Every chunk is fine — repair and parse on a small document is trivially cheap relative to generation. Batching a few chunks reduces UI churn if rendering is expensive.
What about streaming tool call arguments?
The same technique applies, and it is a good use case: showing the tool and its arguments as they form makes an agent's behaviour visible — see tool schema design.
Where to go next
For the generation side that produces these documents, read structured outputs. For validating what arrives, LLM output validation, and for the latency metrics streaming improves, TTFT vs TPOT.