LLM-as-a-judge evals, 50,000 grades per job
Score model outputs, agent transcripts, and RAG answers with a judge model, in bulk. Upload one JSONL file of grading prompts, get back a JSONL of verdicts, and pay half the realtime per-token rate.
- Judge 50k outputs per job
- Rubric prompts in JSONL
- Failed rows isolated
- 50% off realtime
Built for this shape of work.
Independent records, no user waiting, and a real budget: the profile the 50% batch discount is priced for.
Evals are the canonical batch workload
Nothing about grading yesterday's outputs needs a streaming connection. A judge run is thousands of independent requests with no latency requirement, which is exactly the shape the 50% batch discount is priced for.
Reproducible eval runs
One input file, one output file, one job id. Re-running an eval after a prompt change means re-submitting the same JSONL, and diffing two result files gives you the regression report.
A bad row never sinks the run
Malformed records and model errors land in a separate error file with their custom_id. The other 49,990 grades complete, and you re-submit only the failures.
One JSONL file, one job.
Each line is a complete OpenAI-compatible request with a custom_id that is echoed on the matching output row. Up to 50,000 records and 200MB per job.
{"custom_id":"eval-001","method":"POST","url":"/v1/chat/completions","body":{"model":"openrelay/glm-5.2","messages":[{"role":"system","content":"You are an evaluation judge. Score the answer 1-5 against the rubric and reply as JSON: {\"score\", \"rationale\"}."},{"role":"user","content":"Rubric: factual accuracy.\nQuestion: When did the Apollo 11 mission land?\nAnswer to grade: July 20, 1969."}]}}
{"custom_id":"eval-002","method":"POST","url":"/v1/chat/completions","body":{"model":"openrelay/glm-5.2","messages":[{"role":"system","content":"You are an evaluation judge. Given two answers, reply with the better one: A or B, plus one sentence of rationale."},{"role":"user","content":"Question: Summarize the refund policy.\nA: ...\nB: ..."}]}}from openai import OpenAI
client = OpenAI(
base_url="https://inference.openrelay.inc/v1",
api_key="vl_••••••••", # same SDK, new base URL
)
# grade 50,000 model outputs against a rubric overnight
f = client.files.create(file=open("batch.jsonl", "rb"), purpose="batch")
batch = client.batches.create(
input_file_id=f.id,
endpoint="/v1/chat/completions",
completion_window="24h", # billed at 50% of realtime
)
# poll: validating → in_progress → completed
batch = client.batches.retrieve(batch.id)
print(batch.status, batch.request_counts)
# JSONL of {custom_id, response}; failures land in error_file_id
results = client.files.content(batch.output_file_id)The right models for this job.
Batch rates are 50% off the realtime per-token catalog rates, per 1M tokens.
See the full catalogGLM 5.2
Zhipu · 1M context
openrelay/glm-5.2
$0.91 / $2.86
batch input / output per 1M
Reasoning judge for rubric scoring, pairwise comparison, and agent-trajectory review. The 1M context window fits long transcripts without chunking.
GPT-OSS 120B
OpenAI · 128K context
openrelay/gpt-oss-120b
$0.075 / $0.30
batch input / output per 1M
Structured-output judge: return {score, verdict, rationale} as JSON for every row and load results straight into your eval dashboard.
What separates a good run from a re-run.
Prompt and file patterns learned from real jobs, so the first submission is the one that counts.
Pin the judge, version the rubric
Put the rubric in the system message and keep it identical across the whole file. When the rubric changes, that is a new eval version: new file, new job, comparable result sets.
Ask for JSON verdicts
A judge that answers in prose needs a second parsing pass. Constrain the output shape in the prompt ({score, verdict, rationale}) so the results file is machine-readable as-is.
Use pairwise comparison for subtle regressions
Absolute 1-5 scores drift between runs. Putting the old and new answer in one prompt and asking A-or-B is more stable, and each comparison is still just one JSONL line.
Judge agent trajectories, not final answers alone
For agent evaluation, paste the full tool-call transcript and ask the judge whether each step was justified. Long transcripts are cheap when the job is billed at batch rates.
Common questions.
What is LLM-as-a-judge evaluation?
LLM-as-a-judge uses a strong model to score another model's outputs against a rubric: correctness, helpfulness, safety, or a pairwise better-of-two verdict. It replaces slow human review for the bulk of eval work, with humans auditing a sample. Because every grade is an independent request, judge runs map directly onto a batch API.
How do I run LLM evals in batch instead of an eval framework?
Eval frameworks loop over your dataset calling a realtime API. With batch you emit the same grading prompts as JSONL lines, submit one job, and collect verdicts from the output file. You keep your dataset and rubric code and drop the rate-limit handling, retry logic, and the 2x price.
How much does it cost to grade 50,000 outputs?
Cost is per token at half the realtime rate. A typical grading prompt of about 700 input and 150 output tokens on GLM 5.2 runs at $0.91 per 1M input and $2.86 per 1M output tokens in batch, about $54 for the full 50,000 rows. The same run on GPT-OSS 120B is about $5.
Which model should I use as a judge?
Use a model at least as capable as the one being judged. GLM 5.2 is the reasoning option with a 1M window for long transcripts; GPT-OSS 120B handles most rubric scoring at a fraction of the cost and follows JSON output instructions well.
Can I evaluate agents with a batch job?
Yes. Serialize each agent trajectory (messages, tool calls, results) into one grading prompt per line. The judge scores step quality and final outcome. This is the standard pattern for agent evaluation at scale, and it pairs with the agentic batch API for running the agent steps themselves.
More batch workloads
Synthetic data and distillation, a million examples per night
Generate synthetic training data and distill models with batch LLM jobs.
Classify a million documents with a prompt, not a training run
Classify documents, tickets, and text at scale with an LLM batch API.
Free text in, your schema out
Extract structured JSON from documents, emails, and free text with batch LLM jobs.
Running agent pipelines instead of flat request files? See the agentic batch API.
Ship the first job today.
Grab an API key, upload a JSONL file, and run open models at half the realtime cost. No contract, no minimums. Deposit $5 to get $10.