Providers have spare capacity at some hours and not others. Batch inference is how they sell it: submit a job with no latency requirement, accept a completion window measured in hours rather than seconds, and pay roughly half. The model is the same, the output is the same, and the only thing you give up is knowing when it will finish.
For a surprising share of production AI work, that is a trade with no downside at all.
What changed in 2026
- Batch became a standard offering. Most major providers shipped batch endpoints with comparable discount structures, which made it a portable pattern rather than a vendor-specific optimization.
- Completion times improved in practice. Jobs frequently finished well inside the stated window, though the guarantee stayed at the window rather than the observed median.
- Discount stacking clarified. Combining batch pricing with cached prompt prefixes produced compounding savings, and providers documented how the two interact.
- Tooling caught up. Batch job submission, polling, and result reconciliation moved into SDKs and orchestration frameworks instead of being hand-rolled.
What qualifies
| Workload |
Batch suitable |
Why |
| Backfilling summaries over an archive |
Yes |
Nobody is waiting |
| Classifying or tagging a content library |
Yes |
Bulk, offline, tolerant of delay |
| Running an evaluation suite |
Yes |
Results needed by morning, not now |
| Generating embeddings for a new corpus |
Yes |
One-time bulk job |
| Nightly report generation |
Yes |
Scheduled, with slack in the window |
| Enriching records in a data warehouse |
Yes |
Batch-shaped by nature |
| Interactive chat |
No |
A person is waiting |
| Live content moderation |
No |
Latency is the requirement |
| Agent tool calls mid-run |
No |
Blocks the run |
The pattern is simply whether a human is blocked. If the answer is no, batch is almost always the right endpoint, and running that work on the synchronous API is paying double for a speed nobody benefits from.
Restructuring for batch
The engineering change is moving from request-response to job-submit-and-collect, which is a familiar shape but a real refactor if your pipeline assumes synchronous calls.
Submit work in reasonably large units — the overhead per job favours fewer, bigger submissions — but not so large that a failure forces you to reprocess everything. Include a stable identifier on every request so results can be matched back reliably, because batch results do not necessarily return in submission order.
Handle partial failure explicitly. Individual requests within a batch can fail while the job as a whole succeeds, and a pipeline that assumes all-or-nothing will silently drop records. Reconcile by identifier and re-submit the failures rather than the whole batch.
Stack the discounts where you can. If every request in a batch shares a long system prompt or a common document prefix, prompt caching applies on top of the batch discount, and the combined saving is larger than either alone. Both stack with the tier selection described in AI model routers explained — routing decides which model, batch decides which endpoint.
Common mistakes
- Treating the window as an estimate. Design for the stated upper bound, then be pleasantly surprised.
- No stable request identifiers. Matching results back becomes guesswork.
- Assuming all-or-nothing completion. Partial failures are normal and must be reconciled.
- Batching interactive work behind a spinner. Users will not wait hours, and the discount does not justify it.
- Leaving evaluation runs on the synchronous API. Evaluation is the single most obviously batchable workload and the one teams most often forget.
FAQ
How much cheaper is batch really?
Roughly half at most providers, though the exact discount and which models are eligible vary. Check current pricing for your specific model.
What if a job does not complete in the window?
Behavior varies — some providers cancel and return partial results, others extend. Read the specific terms and design your retry logic around them.
Can I cancel a running batch job?
Generally yes, with results for already-completed requests typically returned. Confirm with your provider.
Does batch use a different model?
No. Same model, same weights, same quality. The difference is scheduling, not capability.
Where to go next
For the wider cost picture, read AI cost optimization and AI API cost comparison. For choosing where to run the work, AI inference providers compared.