A surprising share of what teams do with language models has nobody waiting for it. Classifying a backlog of support tickets. Generating embeddings for a corpus. Summarising yesterday's transcripts. Enriching a database overnight. Scoring evaluation samples.
All of it typically goes through the same synchronous endpoint as the chat feature where a user is staring at a spinner, at the same price. Batch endpoints exist precisely for the difference, and they charge roughly half.
What changed in 2026
- Batch became a default consideration for offline work. Enough teams found the savings material that "can this be batch?" entered the design conversation.
- Evaluation traffic moved to batch. Scoring production samples with a judge model is embarrassingly parallel and latency-insensitive, which makes it an obvious fit.
- Indexing pipelines adopted it. Contextualising and embedding a corpus at ingestion is offline work with a real per-item cost.
- Turnaround times stayed variable. Completion windows are upper bounds, not promises, and batches frequently finish sooner — which is not something to depend on.
What you trade
|
Synchronous |
Batch |
| Price |
Standard |
Roughly half |
| Latency |
Seconds |
Up to a stated window, often hours |
| Result order |
Immediate, one response |
Any order, keyed by your ID |
| Failure granularity |
The request fails |
Individual items can fail |
| Rate limits |
Your standard quota |
Usually a separate, larger allowance |
| Streaming |
Available |
Not applicable |
| Model quality |
Same |
Same |
That last row deserves emphasis because people assume otherwise: batch is not a cheaper model or a degraded service. It is the same model, and you are paid to be flexible about when.
The rate limit row is an underrated secondary benefit. Batch quotas are typically separate from and larger than synchronous ones, so moving bulk work to batch also relieves pressure on the quota your interactive traffic depends on.
The shape of a batch request
You assemble a list of requests, each with a custom ID you choose and the full parameters for that call. Submit, poll for completion, then stream the results.
Two structural differences catch people.
Each request is fully independent. There is no shared system prompt or shared configuration across the batch — every item carries its own model, parameters, and messages. That means a mistake in how you generate the requests replicates across every item, and you find out when the whole batch comes back wrong. Validate a handful synchronously before submitting ten thousand.
Results come back in any order. This is the bug that reliably appears in first implementations: code that zips the results array against the input array by position. Key everything by your custom ID and build a lookup. Position is meaningless.
Make the custom ID something you can trace back — a record ID rather than a sequence number — so a failed item tells you which row it belongs to without a second lookup.
Handling partial failure
A batch completing is not the same as every item succeeding. Each result carries its own status, and individual items can fail while the batch as a whole is reported as ended.
Check every result. Expect at least four outcomes: succeeded, errored, cancelled, and expired. Errored items need the same classification as any failure — a content-length error is not retryable in the same way a transient one is, per LLM fallback strategies.
Design the retry path before you need it. The natural pattern is to collect failed custom IDs, decide which are worth retrying, and submit a smaller follow-up batch. Retrying the entire batch because some items failed doubles your bill for work already done.
And handle expiry. If a batch does not complete within its window, unfinished items expire. That is not a failure of your code and it does mean you need somewhere for them to go.
What fits
Good candidates share three traits: nobody is waiting, the work is naturally parallel, and per-item cost matters at volume.
Concretely — bulk classification and tagging, embedding generation for indexing, contextual retrieval enrichment at ingestion, dataset generation, offline evaluation scoring, translation of a document set, and periodic report generation.
Poor candidates are anything interactive, anything where one result feeds the next request, and anything that must complete by a hard deadline you cannot pad. Sequential dependency is the disqualifier people miss: a chain where each step depends on the last cannot be batched, only pipelined.
Common mistakes
- Matching results by position. They come back in any order.
- Not validating before bulk submission. A prompt error replicated across ten thousand items.
- Treating batch completion as total success. Check each item's status.
- Retrying the whole batch on partial failure. Pay twice for the successful items.
- Using it for interactive features. The window is hours.
- Opaque custom IDs. A sequence number tells you nothing about which record failed.
- Assuming it finishes early. It often does; building a schedule on that is a mistake.
FAQ
Is the model actually the same?
Yes. The discount is for scheduling flexibility, not a lesser model. Output quality should match the synchronous endpoint for identical parameters.
How long does it really take?
Variable, and frequently well inside the stated window. Treat the window as the guarantee and anything faster as luck, because building a dependency on early completion means an occasional missed deadline.
Can I cancel one?
Generally yes, and already-processed items are typically still billed. Cancel early if you spot a mistake rather than waiting to discard the results.
Does prompt caching apply?
Interaction between caching and batch varies by provider and is worth checking. Where both apply to a workload with a large shared prefix, the combined saving is substantial — see prompt caching.
Where to go next
For the wider set of cost levers, read AI inference cost optimization. For the latency tradeoff in general terms, batch vs streaming inference, and for the indexing workload batch suits best, contextual retrieval.