Agentic Batch API
Agents multiply model calls. Batch is how you pay for that.
One user request to an agent becomes ten model calls; a pipeline over ten thousand work items becomes millions. Most of those calls have no user waiting on them. Run them as batch jobs: JSONL waves in, results folded back into agent state, at 50% off realtime rates.
- Fan-out agent steps
- Wave-by-wave loops
- Judge passes built in
- 50% off realtime
Plan in your orchestrator. Execute in waves.
An agentic batch pipeline splits the agent loop at its natural seam. The orchestrator owns state, tools, and control flow, exactly as it does today. The model calls, which are the expensive part, are collected across every work item and submitted as one job per step.
Fan-out steps (research every lead, read every document) are a single wave. Sequential steps alternate: model wave, tool execution, next wave. The custom_id field carries the work-item and step key, so results join back to state without bookkeeping.
Nothing about the surface is proprietary: it is the standard OpenAI-compatible Batch Inference API driven by an agent planner instead of a static file.
{"custom_id":"lead-0041:research","method":"POST","url":"/v1/chat/completions","body":{"model":"openrelay/gpt-oss-120b","messages":[{"role":"system","content":"You are the research step of a sales agent. Given a company profile, produce {\"summary\", \"signals\": [], \"disqualifiers\": []} as JSON."},{"role":"user","content":"Company: Meridian Freight, 240 employees, WMS migration announced in June..."}]}}
{"custom_id":"lead-0042:research","method":"POST","url":"/v1/chat/completions","body":{"model":"openrelay/gpt-oss-120b","messages":[{"role":"system","content":"You are the research step of a sales agent. Given a company profile, produce {\"summary\", \"signals\": [], \"disqualifiers\": []} as JSON."},{"role":"user","content":"Company: Halcyon Labs, 45 employees, hiring two ML platform engineers..."}]}}# wave N: run one agent step across every work item, as one job
plan = [agent.next_request(item) for item in items if not item.done]
write_jsonl("wave.jsonl", plan) # custom_id = item:step
f = client.files.create(file=open("wave.jsonl", "rb"), purpose="batch")
batch = client.batches.create(
input_file_id=f.id,
endpoint="/v1/chat/completions",
completion_window="24h",
)
# when the job completes, fold results back into agent state
for row in read_jsonl(client.files.content(batch.output_file_id)):
items[row["custom_id"]].apply(row["response"])
# tool calls requested by the model run in your orchestrator,
# and their results become wave N+1's JSONLSix agent workloads that batch.
Where the fan-out is wide and nobody is watching a spinner, the discount and the missing rate-limit orchestration both land in your favor.
Fan-out steps
The map phase of an agent pipeline: research every lead, read every file, triage every ticket. One agent design, thousands of independent executions. This is 90% of agent token spend and none of it needs a streaming connection.
Wave-by-wave loops
Sequential agent steps become alternating turns: submit a wave of model calls, run the requested tool calls in your orchestrator, fold results into state, submit the next wave. Latency per step stops mattering when the pipeline runs overnight anyway.
Judge and gate passes
Score every agent trajectory with an LLM-as-a-judge batch before results reach production or a human queue. The judge pass typically costs a rounding error compared to the run it audits.
Memory compaction
Summarize long agent histories, distill scratchpads, and refresh entity memory across every active session in one nightly job instead of inline during user interactions.
Backfills and replays
New agent version, old inputs: replay last quarter's tickets through the new pipeline to compare outcomes before cutover. Batch pricing makes full-history replays routine instead of exceptional.
Eval sweeps
Grid-search prompts, models, and tool configurations by expanding the matrix into JSONL records. A 10x20 sweep over 500 tasks is one file, not a week of rate-limited loops.
Batch the pipeline, stream the conversation.
The split is not which agent, it is which call: the same product usually has both kinds.
- Pipelines where each item is independent (leads, documents, tickets, repos)
- Agent steps that tolerate hours of latency: enrichment, triage, research, review
- Anything you would run nightly, weekly, or per-release
- A user watching the agent work: keep that on realtime chat completions
- Deep sequential chains over a single item with no fan-out to amortize
The steps your agents are already doing.
Every workload page ships a runnable JSONL example and model picks with batch pricing.
Agentic batch, answered.
What is an agentic batch API?
A batch API used as the execution layer for AI agent pipelines. Instead of an orchestrator calling a realtime endpoint once per agent step, it collects the current step for every work item into a JSONL file, submits one job, and folds the results back into agent state. Fan-out steps run as one wave; sequential steps run wave by wave. The economics are the point: agent pipelines multiply request volume, and batch cuts the per-token price in half while removing rate-limit orchestration.
How do tool calls work in a batch job?
The model side of a tool loop batches; the tools run where they always ran, in your orchestrator. A wave's responses include the tool calls each agent requested; your code executes them and writes the tool results into the next wave's messages. Batch executes model requests, so anything the model needs mid-request has to be in the request.
When should an agent use batch instead of realtime?
Split by who is waiting. A user watching the agent needs realtime streaming. A pipeline processing a queue (enrich these 10,000 leads, triage this backlog, review every PR from last sprint) is batch-shaped: independent items, no latency requirement, and enough volume for the 50% discount to be a line item.
Does this work with my agent framework?
If the framework separates planning from execution, yes: anything that can emit its next model request as JSON instead of firing it can batch. Teams typically add a batch executor alongside the realtime one and route by workload. The surface is the standard OpenAI batches API, so the SDK plumbing already exists.
How do I evaluate agents at scale?
Serialize each trajectory (messages, tool calls, outcomes) into one grading record and run an LLM-as-a-judge batch over all of them. The same pattern gates deploys: replay a fixed task set through the new agent version, judge both runs, and diff the scores. See the LLM-as-a-judge workload page for prompts and costs.
Point your orchestrator at a batch endpoint.
Same OpenAI SDK, same JSONL, half the price. If your agent pipeline has an unusual shape, tell us what it does and we will help you wave it.