Batch Inference API
Batch inference for DeepSeek, Qwen, Llama & open models
Submit millions of prompts as one job. An OpenAI-compatible Batch API. JSONL in, JSONL out, 24-hour window, billed at 50% below realtime and run on distributed GPUs. A drop-in OpenAI Batch API alternative for open models.
- OpenAI-compatible
- JSONL in / JSONL out
- 24h window
- 50% off realtime
How it works
Four calls, start to finish.
The same workflow as the OpenAI Batch API (upload, create, track, download), so existing batch pipelines port over by swapping the base URL.
Upload a JSONL file
Each line is a full OpenAI-compatible request with a custom_id. Up to 50,000 records per job.
Create the batch
POST /v1/batches with your file id, endpoint, and a 24h completion window. Get back a job id and live request_counts.
Track status or get a webhook
Poll the job through validating → in_progress → completed, or let a webhook ping you when it finishes.
Download results
Pull a JSONL of {custom_id, response} plus a separate error file. Failed rows are isolated, so you re-submit just those.
OpenAI Batch API alternative
Keep the SDK. Change one line.
The endpoint shape matches OpenAI's: files.create, batches.create, output and error files, so you point your existing client at OpenRelay and run the same job against open models. No proprietary lock-in, no rewrite.
from openai import OpenAI
client = OpenAI(
base_url="https://inference.openrelay.inc/v1",
api_key="vl_••••••••", # same SDK, new base URL
)
# 1. upload the JSONL
f = client.files.create(file=open("batch.jsonl", "rb"), purpose="batch")
# 2. create the batch (24h window, 50% off)
batch = client.batches.create(
input_file_id=f.id,
endpoint="/v1/chat/completions",
completion_window="24h",
)
# 3. poll: validating → in_progress → completed
batch = client.batches.retrieve(batch.id)
print(batch.status, batch.request_counts)
# 4. download {custom_id, response} JSONL
results = client.files.content(batch.output_file_id)Open models
Run batch jobs on the models people search for.
The DeepSeek API, Qwen API, and Llama API, plus every other model in the catalog, through one OpenAI-compatible batch endpoint. A single job can mix models line by line.
DeepSeek R1
DeepSeek
Frontier open reasoning for bulk evals, synthetic data, and chain-of-thought scoring.
deepseek-ai/deepseek-r1Qwen 2.5 Coder 32B
Alibaba
Generate, refactor, and document whole repositories in a single overnight job.
qwen/qwen2.5-coder-32bLlama 3.3 70B Instruct
Meta
General-purpose workhorse for classification, summarization, and labeling at scale.
meta-llama/llama-3.3-70b-instructBatch prices are 50% below the realtime per-token catalog rates. Need a model that is not listed? Request it.
Realtime, batch, or your own GPUs
Pick the right inference mode.
Synchronous, asynchronous, or serverless: every mode shares one network, one catalog, and one API key. Batch is the on-demand path that trades latency for throughput and a 50% discount.
Batch
/v1/batchesAsynchronous, on-demand jobs for millions of prompts. JSONL in/out, 24h window, billed at 50% off. Same model, deferred and discounted.
Evals, embeddings, labeling, bulk generation
How it compares
Batch inference, without the trade-offs.
OpenAI and Claude batch are great if you only need their models. Bedrock and RunPod mean cloud accounts and S3 or per-second GPU workers. OpenRelay gives you open models on a self-serve, OpenAI-compatible batch API.
| Provider | Open models | OpenAI-compatible | Batch 50% off | Self-serve | Pricing |
|---|---|---|---|---|---|
| OpenRelayYou are here | DeepSeek · Qwen · Llama | 24h | Per token | ||
| OpenAI Batch API | GPT only | 24h | Per token | ||
| Anthropic / Claude Batch | Claude only | Messages API | 24h | Per token | |
| Bedrock Batch Inference | Select models | AWS + S3 | Per token | ||
| Together Batch | Per token | ||||
| RunPod Serverless | Bring your own | Worker-defined | Per GPU-sec |
Reflects publicly documented capabilities as of June 2026; verify current features with each provider. OpenRelay also lets you rent GPUs directly for fully custom inference.
Quickstart
From JSONL to results.
{"custom_id":"row-1","method":"POST","url":"/v1/chat/completions","body":{"model":"meta-llama/llama-3.3-70b-instruct","messages":[{"role":"user","content":"Classify the sentiment of this review"}]}}
{"custom_id":"row-2","method":"POST","url":"/v1/chat/completions","body":{"model":"deepseek-ai/deepseek-r1","messages":[{"role":"user","content":"Grade this answer 1-5 and explain why"}]}}curl https://inference.openrelay.inc/v1/batches \
-H "Authorization: Bearer vl_••••••••" \
-H "Content-Type: application/json" \
-d '{
"input_file_id": "file_abc123",
"endpoint": "/v1/chat/completions",
"completion_window": "24h"
}'FAQ
Batch inference, answered.
What is a batch inference API?
A batch inference API lets you submit many model requests as one asynchronous job instead of one HTTP call at a time. You upload a JSONL file (or pass requests inline), the platform runs them within a completion window, and you download a results file. It is the right tool for evals, embeddings, classification, summarization, and synthetic-data generation where latency does not matter but cost and throughput do.
Is OpenRelay an OpenAI Batch API alternative?
Yes. The surface mirrors the OpenAI Batch API: POST /v1/batches with an input_file_id, an endpoint, and a 24h completion window, returning request_counts and output/error files, so you keep the OpenAI SDK and only change the base URL to https://inference.openrelay.inc/v1. The difference is the models: you run open models like DeepSeek, Qwen, and Llama instead of GPT, at a fraction of the per-token cost.
Which models can I run batch jobs on?
Any model in the inference catalog (DeepSeek R1, Qwen 2.5 Coder, Qwen3 VL, Llama 3.3 70B, Llama 3.1 8B, Gemma, and more) through one OpenAI-compatible endpoint. Each JSONL line names its own model, so a single batch can fan out across several models at once.
How much does batch inference cost?
Batch jobs bill at the same per-token rates as realtime inference, minus 50%. Llama 3.3 70B runs at $0.28 / $0.40 per 1M input/output tokens in batch, and Llama 3.1 8B at $0.015 / $0.03. There is no per-hour GPU rental and no minimum, so you pay only for the tokens your job actually processes.
How is batch different from async and serverless inference?
Realtime inference streams tokens back over /v1/chat/completions for interactive traffic. Batch is the asynchronous, on-demand path: deferred jobs that trade latency for a 50% discount and massive throughput. Serverless GPUs are for when you want to run your own image or fine-tuned weights without managing VMs. All three share one network and one API key.
What file format and limits does a batch use?
JSONL in, JSONL out. Each input line is {custom_id, method, url, body}, where body is a normal chat-completions request. Each output line is {custom_id, response} (or an error), so you can match results back to inputs. Jobs accept up to 50,000 records and 200MB per file.
Do I get webhooks and automatic retries?
Yes. Jobs are idempotent per (batch, custom_id) with exponential-backoff retries, failed rows are written to a separate error file so one bad request never sinks the job, and you can poll status or receive a completion webhook. Unfinished work past the 24h window is marked expired rather than billed.
Can I migrate from OpenAI, Bedrock, or RunPod?
If you already build JSONL batches for the OpenAI Batch API, the file format and SDK calls carry over directly. Coming from Bedrock batch inference or RunPod serverless, you drop the S3 wiring and per-GPU-second workers in favor of a self-serve, per-token API with open models.
Ship your first batch today.
Grab an API key, upload a JSONL file, and run open models at half the realtime cost. No contract, no minimums. Deposit $5 to get $10.