Small language models do 90% of the work
Most pipeline volume is mechanical work an SLM handles: classify, score, flag, extract. This guide covers what small models are genuinely good at, where they lose to large ones, and the two-tier routing pattern that lets you screen everything cheap and escalate only the rows the SLM marks uncertain.
- SLM vs LLM, by task
- Two-tier cascade with code
- Real per-token math
- Calibrated, not guessed
SLM vs LLM: the split is consistent.
A small language model is one compact enough to serve cheaply and fast: here, GPT-OSS 20B and Gemma 4 31B NVFP4, versus the 120B-and-up large tier. What each side wins barely moves across domains.
Small language models hold their own on
- Closed-set classification and routing (pick one of N labels)
- Sentiment and scoring on a fixed scale
- Policy screening against numbered rules
- Short extraction with a simple, typed schema
- Paraphrase, formatting, and normalization
Escalate to the large model for
- Multi-step reasoning and judgment calls
- Strict schemas over messy, inconsistent source text
- Nuance the small tier flattens: sarcasm, mixed signals, edge cases
- Open-ended generation where quality is the product
- Anything the small model itself flags as uncertain
The boundary is empirical, not doctrinal: benchmark a few hundred labeled rows on GPT-OSS 20B and GPT-OSS 120B and let the agreement rate draw the line for your task. That experiment costs cents as a batch job.
The two-tier cascade: how the SLM earns its keep.
One small model, one big model, one threshold. Every request runs on the SLM with a prompt that demands a confidence value alongside the answer. High-confidence rows are done. Low-confidence rows run again on the big model, whose answer wins.
That is the entire architecture. No learned router, no feature engineering, no separate service: the small model is its own router, and the threshold is one number you calibrate against labeled data. Escalated rows pay for both passes, which sounds wasteful until you notice they are 5 to 15% of traffic while the other 85 to 95% just got 3 to 30x cheaper per row, depending on the big model's rate.
Because both models sit behind the same OpenAI-compatible endpoint here, the router is a model-string swap, not an integration. The same pattern runs as two waves through the Batch Inference API for offline backlogs.
import json
from openai import OpenAI
client = OpenAI(
base_url="https://inference.openrelay.inc/v1",
api_key="vl_••••••••",
)
SCREEN = "openrelay/gpt-oss-20b" # the SLM: $0.05/$0.20 per 1M
ESCALATE = "openrelay/gpt-oss-120b" # for the rows the SLM flags
def classify(ticket: str) -> dict:
r = ask(SCREEN, ticket) # returns {"label", "confidence"}
if r["confidence"] >= 0.85:
return r # ~90% of traffic stops here
return ask(ESCALATE, ticket) # the hard 10% gets the big model
def ask(model: str, ticket: str) -> dict:
resp = client.chat.completions.create(
model=model,
messages=[
{"role": "system", "content":
"Classify into exactly one of: billing, bug, feature_request. "
'Reply as JSON: {"label", "confidence": 0..1}.'},
{"role": "user", "content": ticket},
],
)
return json.loads(resp.choices[0].message.content)The math, on real rates.
100,000 requests at 500 input / 50 output tokens each, realtime rates, 10% escalation. Escalated rows pay both passes. Batch halves every number below.
| Big model | Everything on the big model | Routed via GPT-OSS 20B | Savings |
|---|---|---|---|
| GPT-OSS 120B$0.15 / $0.60 per 1M | $10.50 | $4.55 | 2.3x |
| GLM 5.2$1.82 / $5.72 per 1M | $120 | $15.46 | 7.7x |
Computed from the live catalog rates. The gap widens with the price of the big model: route away from a frontier-priced model and the factor climbs toward an order of magnitude.
What separates a router that saves money from one that saves face.
Four practices from pipelines that run this pattern in production.
Calibrate the threshold on labeled data
Run the SLM over about a thousand hand-labeled rows, plot accuracy against reported confidence, and place the threshold where accuracy crosses your bar. A guessed threshold either burns money (too low a bar for escalation) or ships errors (too much trust).
Keep the prompt identical across models
Same system prompt, same output schema on both. Escalation becomes a pure model-string swap, results stay comparable, and the shared prefix keeps earning cached-input rates on both tiers.
Audit the escalations, not only the answers
The escalated slice is a free curriculum: it tells you what the small model finds hard. Reviewing a sample weekly catches distribution drift long before aggregate accuracy moves.
Re-run calibration as a scheduled batch job
Prompts change, traffic changes, models improve. A monthly batch job over the fixed calibration set re-validates the threshold for pennies and turns silent drift into a diff between two result files.
Small language models, answered.
What is a small language model?
A small language model (SLM) is a model compact enough to serve cheaply and fast, typically a few billion to a few tens of billions of parameters, versus the hundreds of billions in frontier LLMs. On this platform the SLM tier is GPT-OSS 20B and the Gemma 4 31B NVFP4 serving; the large tier is GPT-OSS 120B, DeepSeek V3.1 Terminus, and GLM 5.2. The distinction that matters in production is not the parameter count, it is the per-token price and latency you pay for capability you may not need.
SLM vs LLM: when is the small model enough?
SLMs track LLMs closely on closed-set classification, sentiment, moderation, routing, and short extraction, and fall behind on multi-step reasoning, strict schemas over messy input, and open-ended generation. The practical answer is empirical: run a few hundred labeled rows through both and compare. On tasks where the SLM lands within a point or two, routing to it is free money.
What is LLM routing?
LLM routing sends each request to the cheapest model that can handle it instead of one model for everything. The workhorse pattern is a two-tier cascade: the SLM processes every request and reports confidence, and only low-confidence rows escalate to the big model. Most pipeline traffic is mechanical, so the small tier absorbs the volume and the big tier gets the genuinely hard remainder.
What is model cascading and how is it different from a router?
Cascading is routing where the decision comes from the small model itself: it always runs first, and its own confidence decides whether the request continues up the ladder. A standalone router classifies the request before any model runs. Cascades are simpler to build and calibrate, which is why they are the right starting point; dedicated routers earn their complexity only when the tiers differ in kind (for example text vs vision) rather than in capability.
How do I pick the confidence threshold?
Calibrate against a hand-labeled sample: run the SLM over about a thousand rows, plot its accuracy by reported confidence, and set the threshold where accuracy drops below your bar. Recheck it whenever the prompt or model changes. Thresholds around 0.8 to 0.9 are typical for closed-set tasks, sending 5 to 15% of traffic to the big tier.
Does routing to a small model hurt quality?
Not when the escalation path exists and the threshold is calibrated: easy rows get answers a big model would also give, and hard rows still reach the big model. The failure mode to watch is silent overconfidence on a distribution shift, which is why the calibration sample should be re-run periodically as a cheap batch job.
Both models, one endpoint.
GPT-OSS 20B and 120B sit behind the same OpenAI-compatible API and one vl_ key, so the cascade is an afternoon of work, not a platform project. Deposit $5 to get $10.