The OpenAI Batch API, and the same JSONL on open models
OpenAI's Batch API set the standard shape for batch LLM jobs: JSONL files, a 24-hour window, half-price tokens. This page covers how it works, what its limits are, and how to point the exact same pipeline at open models when GPT pricing or model choice becomes the constraint.
OpenAI's batch surface in six rows.
The load-bearing facts: discount, turnaround, formats, limits, and what it takes to get access.
- Discount
- 50% off standard per-token pricing
- Completion window
- 24h; jobs can finish much earlier, no timing SLA
- Format
- JSONL via the Files API; output + error files per job
- Job limits
- Up to 50,000 requests and 200MB per input file
- Rate limits
- Separate per-model batch queue caps on enqueued tokens
- Models
- OpenAI models only
As publicly documented by OpenAI, August 2026. Verify current values in OpenAI's docs.
The parts that shape real usage.
Rate cards agree everywhere (50% off is the industry number); these are the differences that decide the bill and the build.
Pricing: the 50% is real, the base rate is the catch
Batch halves OpenAI's standard per-token rates, and for GPT-class quality on GPT-only workloads that is a fair deal. The arithmetic to check is the base: half of a frontier-model rate is often still a multiple of an open model's full realtime price. If your workload is classification, extraction, summarization, or OCR, benchmark a 31B-or-smaller open model before assuming you need half-price GPT.
Limits and rate limits in practice
The per-job caps (50,000 requests, 200MB) match what you get here and are rarely the binding constraint. What teams actually hit is the enqueued-token limit: each model has a cap on how many tokens you can have queued at once, so very large backfills serialize into sequential jobs. Plan big runs as a chain of jobs per model rather than one giant submission.
Response time: what 24h means
The window is a deadline, not an estimate. Small jobs often complete in minutes; large ones can use most of the window, and anything unfinished at 24h is expired with the completed portion returned and billed. Build pipelines around the deadline, not around observed latency: yesterday's 20-minute job proves nothing about tonight's.
The same request, both shapes.
The surfaces are deliberately compatible. Your JSONL records keep their shape; the model field and the base URL change. files.create, batches.create, status polling, and output download are the same SDK calls.
{"custom_id":"row-1","method":"POST","url":"/v1/chat/completions","body":{"model":"gpt-5-mini","messages":[{"role":"user","content":"Classify this ticket: ..."}]}}{"custom_id":"row-1","method":"POST","url":"/v1/chat/completions","body":{"model":"openrelay/gpt-oss-120b","messages":[{"role":"user","content":"Classify this ticket: ..."}]}}Stay on OpenAI when
- The workload needs a frontier OpenAI model specifically
- You depend on OpenAI-only request features end to end
- Batch spend is small enough that pricing is not a lever
Run it on OpenRelay when
- Open models handle the task and the per-token base rate matters
- You want one batch surface for OCR, classification, and generation models
- You are already OpenAI-SDK-shaped and want a base-URL migration, not a rewrite
OpenAI batch, answered.
How does the OpenAI Batch API work?
You upload a JSONL file where each line is a complete request with a custom_id, create a batch pointing at that file with a 24h completion window, poll its status, and download an output file of responses plus an error file for failed rows. Tokens bill at 50% of standard rates. OpenRelay implements the same flow with the same SDK calls.
What are the OpenAI Batch API limits?
Per job: 50,000 requests and a 200MB input file. Separately, each model has a batch queue cap on total enqueued tokens, which is the limit large backfills actually hit. OpenRelay's per-job limits are the same 50,000 records and 200MB.
How long does the OpenAI Batch API take to respond?
Anywhere from minutes to the full 24 hours. The window is a guarantee boundary, not a latency estimate: unfinished requests at 24h are marked expired, with completed work returned and billed. Treat any batch pipeline, on any provider, as a deadline system.
What is the best OpenAI Batch API alternative?
For staying on GPT models, Azure OpenAI offers the same batch mechanism inside Azure. For moving off proprietary models, OpenRelay runs the identical JSONL format and SDK surface against open models (GPT-OSS, Gemma, GLM, DeepSeek-OCR) with the same 50% batch discount, so the switch is the base URL and the model field.
Is the OpenAI Batch API cheaper than realtime open models?
Often not. Half-price GPT can still cost more per token than full-price open models, and open models also take the 50% batch cut here. Run a 500-row quality benchmark on your actual task; if a small open model passes, the batch-vs-batch price difference is usually 5-20x.
Run the benchmark batch.
500 rows of your real workload on open models settles the pricing question in an afternoon. Deposit $5 to get $10.