AI Best Practices: Cutting LLM Pipeline Costs Without Losing Quality
Most teams pay frontier prices for work that is not frontier work. Two changes fix the majority of that: send the easy rows to a smaller model, and stop paying full price for the same system prompt thousands of times. Neither costs you quality on the work that matters.
TL;DR
- →Right-size the model: screen everything on a 20B, escalate only what it flags. Typically 3-10x cheaper with no quality loss on the easy majority.
- →Cache the shared prefix: repeated system prompts bill at $0.005 per 1M on the 20B, one tenth the fresh rate. Automatic, no configuration.
- →Together: a 1M-row pipeline drops from $2132 to $278 before caching, and the long rubric that makes the small model reliable becomes nearly free after it.
1. Right-size the model
Most pipeline volume is mechanical work a 20B model handles: classify, score, flag, extract. These tasks have a narrow output space and a clear right answer, which is exactly where a small model is strongest and where a frontier model is paying for capability you never use.
The pattern is to screen everything cheap and escalate only the rows the small model marks uncertain. Have it return a confidence alongside its answer, then route the low-confidence tail to the larger model. The easy majority never touches the expensive path, and the hard rows still get the better model, so quality holds where it matters.
| Escalated to GLM 5.2 | Cost, 1M rows | vs all-GLM 5.2 |
|---|---|---|
| 0% (everything on GLM 5.2) | $2132 | baseline |
| 5% | $172 | 12.4x cheaper |
| 10% | $278 | 7.7x cheaper |
| 15% | $385 | 5.5x cheaper |
| 25% | $598 | 3.6x cheaper |
700 input and 150 output tokens per row at list rates, screening on GPT-OSS 20B ($0.05 / $0.2 per 1M) and escalating to GLM 5.2 ($1.82 / $5.72).
Two things set where you land in that range, and both are worth knowing before you design around a number.
Your escalation rate. A clean classification task might flag 5% as uncertain. A messy extraction task might flag 25% and still come out well ahead. Measure it on a sample rather than assuming it.
The price gap you escalate across. The saving is bounded by it. Escalating from GPT-OSS 20B to GLM 5.2 spans a 36x gap, which is what puts the range at 3-10x. Escalating to GPT-OSS 120B instead spans only 3x, so it tops out nearer 2.6x however good your screening is. Use the small model to avoid the reasoning model, not just the next size up.
2. Cache the shared prefix
Pipelines repeat the same system prompt thousands of times. The rubric, the policy, the output schema, the few-shot examples: identical on every row, and on most platforms billed fresh every time. That is usually the single largest line item in a classification pipeline, and it is pure waste.
Cached input bills at one tenth the fresh rate on GPT-OSS models, $0.005 per 1M on the 20B, automatically, with no configuration. Long rubrics and policies become nearly free.
| Model | Input / 1M | Cached input / 1M | Saving |
|---|---|---|---|
| GPT-OSS 20B | $0.05 | $0.005 | 10x |
| GPT-OSS 120B | $0.15 | $0.015 | 10x |
To benefit, keep the repeated content at the front of the prompt and the row-specific content at the end. A cache hit depends on a shared prefix, so interleaving the variable part early breaks it. Put the system prompt, rubric and examples first, then the row.
Why these two compound
Right-sizing cuts the number of expensive calls. Caching cuts the cost of every remaining call that shares a prefix. A long rubric is the thing that makes a small model reliable enough to screen with, and caching is what makes a long rubric affordable. Each one makes the other cheaper to adopt.
Keep going
- →Inference pricingcurrent rates for every model, including cached input.
- →How to reduce LLM inference coststhe infrastructure side: right-sizing GPUs, quantization, batching.
- →Batch inference APIhalf the realtime rate when the work is not latency-sensitive.