Guides
AI best practices, measured in dollars and retries
Production LLM pipelines succeed or fail on six decisions: which model answers, when work runs, what gets cached, what shape outputs take, how choices get measured, and what happens when a row fails. Each section below is the concrete recipe, with rates you can check.
Six practices, in cost order.
The first three decide the bill; the last three decide whether you can trust the output.
Right-size the model, then route
Most pipeline volume is mechanical work a 20B model handles: classify, score, flag, extract. Screen everything cheap and escalate only the rows the small model marks uncertain. A calibrated two-tier cascade routinely cuts spend several-fold with no quality loss on the easy majority.
The SLM routing guideMove every offline job to batch
Anything nobody is waiting on (backlogs, evals, OCR archives, synthetic data, agent backfills) belongs on a batch API at a flat 50% off: JSONL in, results within 24h, failures isolated per row. The split is not which pipeline, it is which call: stream the conversation, batch everything else.
Batch Inference APICache the shared prefix
Pipelines repeat the same system prompt thousands of times. Cached input bills at a fraction of the fresh rate and applies automatically when the prefix stays byte-identical. Long rubrics and policies become nearly free. The deep dive below covers how it works and how teams accidentally break it.
Jump to the caching sectionConstrain outputs to a schema
Prose answers need a parsing pass that fails a few percent of the time, forever. Pin the output shape in the prompt (field names, types, null rules, no prose), validate mechanically, and retry rejects as a small follow-up job. Two-pass pipelines land above 99% parseable.
Extraction patternsBenchmark on your rows, not on leaderboards
Public benchmarks rank models on other people's tasks. A few hundred hand-labeled rows of your real task, run through the candidate models as one batch job, settles model choice for cents and gives you a regression set to re-run after every prompt change.
LLM-as-a-judge evalsIsolate failures so one row never sinks a run
Tag every request with your own id, write failures to a separate error stream, and make retries idempotent. In batch this comes built in (custom_id plus a per-job error file); in realtime pipelines you build the same shape yourself. The difference shows up the first time row 38,412 is malformed.
How batch jobs isolate errorsPrompt caching, properly.
The least-used lever because it looks like magic. It is not: it is a byte-identical prefix and a discount.
How it works
Serving engines keep a KV cache: the attention keys and values for tokens already processed, so generating each new token does not recompute the sequence from scratch. Prompt caching persists that work across requests. When your next request starts with the same bytes as a previous one, the engine reuses the stored computation and the provider bills those tokens at the cached rate.
On OpenRelay the discount is 10x on GPT-OSS models: GPT-OSS 20B input drops from $0.05 to $0.005 per 1M tokens when served from cache, and GPT-OSS 120B from $0.15 to $0.015. It applies automatically; there is nothing to configure and no write premium.
How teams break it
The cache matches byte-for-byte from the start of the prompt, so the classic mistakes are interpolating a timestamp, request id, or user name into the system prompt, or rotating prompt variants per request. One changing character invalidates everything after it. The rule that fixes all of them: stable content first, volatile content last, and one canonical prompt per pipeline.
# Structure prompts so the shared prefix stays byte-identical:
# stable content first, per-request content last.
SYSTEM = load("moderation_policy_v4.md") # 6,000 tokens, same every call
resp = client.chat.completions.create(
model="openrelay/gpt-oss-20b",
messages=[
{"role": "system", "content": SYSTEM}, # cached after first call
{"role": "user", "content": post_text}, # the only varying part
],
)
# usage.prompt_tokens_details.cached_tokens shows what the cache servedWorked example: a moderation pipeline with a 6,000-token policy and 200-token posts is 97% shared prefix. At the 10x cached rate, input spend falls roughly 8x, on top of whatever routing and batch already saved. Elsewhere the mechanics differ (OpenAI caches automatically at a discount; Anthropic uses explicit cache_control breakpoints with a write premium and ~10x reads, as documented August 2026), but the prompt-structure discipline transfers unchanged.
Best practices, answered.
What is prompt caching?
Prompt caching reuses the computation for a prompt prefix the provider has already processed. When thousands of requests share the same system prompt, the provider processes it once and serves it from cache afterward, billing those tokens at a steep discount. On OpenRelay's GPT-OSS models the cached rate is one tenth of fresh input, applied automatically with no configuration.
KV cache vs prompt caching: what is the difference?
The KV cache is the serving-engine mechanism: attention keys and values computed for earlier tokens are kept in GPU memory so each new token does not recompute the whole sequence. Prompt caching is the product built on top of it: persisting that computation across requests that share a prefix and passing the saving through as a cheaper rate. Every LLM uses a KV cache internally; prompt caching is what shows up on your bill.
How much does prompt caching actually save?
It scales with how much of your prompt is shared prefix. A moderation pipeline with a 6,000-token policy and 200-token posts is about 97% prefix; at a 10x cached-input discount that cuts input spend roughly 8x. A chat app with unique history per user saves far less. Measure it: usage reports cached tokens per request.
Do OpenAI and Anthropic have prompt caching too?
Yes, with different mechanics as of August 2026: OpenAI applies caching automatically and discounts cached input; Anthropic uses explicit cache_control breakpoints with a premium on cache writes and roughly 10x cheaper reads. The portable skill is the same everywhere: keep the shared prefix byte-identical and put volatile content last.
What breaks prompt caching?
Anything that makes the prefix differ between requests: a timestamp or request id interpolated into the system prompt, per-user values early in the prompt, reordered tool definitions, or A/B prompt variants. The cache matches byte-for-byte from the start of the prompt, so one changing character at position 100 discards everything after it.
What are the most important best practices for LLMs in production?
In cost order: route requests to the smallest model that passes your benchmark, run offline work through a batch API, structure prompts for caching, and constrain outputs to schemas. In reliability order: benchmark on your own labeled rows, isolate per-row failures, and re-run your regression set after every prompt or model change. The six sections on this page cover each with the concrete recipe.
Practice on real rates.
Every recipe on this page runs against the same OpenAI-compatible endpoint and one vl_ key. Deposit $5 to get $10 and benchmark the whole stack in an afternoon.