A model generating a four-hundred-word answer takes the same total time whether or not you stream it. Streaming changes only when the user starts seeing output — and that difference is the gap between an interface that feels responsive and one that feels stuck. It is a perception change, and perception is what users actually experience.
Implementing it is straightforward. Implementing it well means handling the cases that only appear in production.
What changed in 2026
- Streaming became the baseline expectation. Users conditioned by chat interfaces began reading a non-streaming response as a hung request.
- Structured output streaming matured. Incremental parsers that surface partial objects as they arrive made streaming compatible with typed responses.
- Reasoning models complicated the picture. Models that think before answering produce a long silent period even when streaming, which needed its own interface treatment.
- Resumable streams appeared. Support for reconnecting to an interrupted generation, rather than restarting it, reduced waste on flaky connections.
What to get right
| Concern |
Approach |
| Time to first token |
Optimize prompt length and caching; this is the felt number |
| Transport |
Server-sent events for one-way; websockets when you need duplex |
| Buffering by proxies |
Disable it explicitly; a buffering proxy silently defeats streaming |
| Mid-stream errors |
Decide whether to keep, mark, or remove the partial output |
| Cancellation |
Propagate it to the provider so you stop being billed |
| Structured output |
Incremental parser, or stream a text field within the object |
| Reasoning delay |
Show thinking state; a blank pause reads as failure |
| Very short responses |
Consider not streaming; the flicker is worse |
Proxy buffering is the classic invisible failure. Everything works locally, then you deploy behind a reverse proxy or a CDN that buffers the response, and streaming silently becomes non-streaming with no error anywhere. Disable buffering explicitly for streaming endpoints and verify in the deployed environment rather than in development.
The mid-stream failure problem
When a generation fails halfway, the user is already reading a partial answer. There is no clean recovery, only choices with different costs.
Removing the partial output and showing an error is honest and jarring — text disappears from under someone mid-read. Keeping it with a visible error marker preserves what may be useful and risks someone acting on a truncated answer. Retrying silently and replacing the content produces a confusing rewrite.
The workable default for most applications is to keep the partial output, mark it clearly as incomplete, and offer a retry. That respects what the user has already read while being unambiguous that it did not finish.
Cancellation deserves equal attention. When a user navigates away or clicks stop, propagate that to the provider. Otherwise generation continues to completion, you are billed for tokens nobody will read, and the capacity is wasted — the kind of quiet cost that shows up in AI agent cost per task.
For reasoning models, the streaming experience needs adaptation. There is a genuine pause before any output appears, and an interface that shows nothing during it reads as broken. Surface a thinking indicator, and where the provider exposes reasoning progress, show it.
Common mistakes
- Proxy buffering left enabled. Streaming works in development and not in production.
- No cancellation propagation. Billed for output nobody sees.
- Streaming very short responses. Flicker without benefit.
- Assuming partial JSON parses. It does not; use an incremental parser.
- Silent pause on reasoning models. Users read it as a failure.
- No mid-stream error strategy. Whatever you do by default will be the wrong thing.
FAQ
Does streaming cost more?
No. Same tokens, same price. It changes delivery, not consumption.
Server-sent events or websockets?
Server-sent events for one-way streaming — simpler, reconnects automatically, works over ordinary HTTP. Websockets when the client also needs to send mid-stream.
How do I stream structured output?
Either an incremental JSON parser surfacing partial objects, or design the schema so prose lives in one field you stream as text while the rest arrives at the end.
Should I stream in a background job?
No. Streaming exists for a waiting human. For batch work, use the batch inference path instead.
Where to go next
For voice equivalents, read realtime voice API. For structured output constraints, constrained decoding explained, and for cost, token counting explained.