Inference delivery network
What is an Inference Delivery Network?
An inference delivery network (IDN) routes AI inference requests across pooled GPU capacity the way a CDN routes content: one endpoint in front, many providers behind it, and every request served by the best available backend. OpenRelay is an IDN.
- One OpenAI-compatible endpoint
- Automatic backend routing
- Failover built in
- Per-token billing
The model
The CDN, applied to inference
CDNs solved content delivery by putting one entry point in front of distributed, interchangeable capacity. An IDN applies the same structure to a harder resource: GPU compute.
| CDN | IDN | Why it matters |
|---|---|---|
| Origin server | The model | The thing being delivered: weights, not files |
| Points of presence | GPU capacity across providers | Pooled supply instead of one vendor's region |
| Request routing | Model-to-backend routing | Every request goes to the best available backend |
| Cache hit | Warm model | A loaded model answers now; a cold one makes you wait |
| Failover to the next PoP | Failover to the next provider | A provider running dry is routine, not an incident |
| Egress billing | Token billing | Pay for what was delivered, not for standing capacity |
The term comes from a 2021 Inria research paper, Towards Inference Delivery Networks, which defined IDNs as networks of computing nodes coordinating to serve ML inference requests. OpenRelay is the production implementation.
Why now
AI development multiplied machine demand. Human attention stayed flat.
Machine demand, human-scale infrastructure
Coding agents burn tokens continuously, not in office hours. A team of five running agents produces the inference load of a team of fifty, against capacity that was bought, priced, and operated for five.
At-desk management
When a provider throttles or runs dry, a person notices, re-routes, and re-provisions. Every system that needs a human at a desk caps how many agents you can actually run. The network has to handle it, or your headcount does.
Single-vendor exposure
One provider means their pricing, their rate limits, and their downtime at 3am. Fragmented GPU supply is the reality; an IDN absorbs it the way a CDN absorbs a traffic spike: route around the constraint, keep serving.
On the network
One network, three outputs
Inference is what the network delivers. The same pooled capacity also rents out raw machines and runs CI, so none of it sits idle.
Frequently asked questions
What is an inference delivery network (IDN)?
An inference delivery network is a network of GPU capacity, pooled across providers and locations, that routes each AI inference request to the best available backend. It applies the CDN model to inference: one endpoint in front, distributed capacity behind it, routing and failover handled by the network instead of by the application.
How is an IDN different from a CDN?
A CDN moves static and cached content close to users; the hard problems are bandwidth and cache placement. An IDN delivers computation: the scarce resource is GPU capacity, and the routing decision weighs model availability, load, and cost rather than geographic distance alone. The operating model is the same: one entry point, many interchangeable backends, automatic failover.
How is an IDN different from a single inference API provider?
A single provider serves models from their own capacity, so their capacity limits are your capacity limits. An IDN sits above providers and treats them as interchangeable supply. When one backend is saturated or down, requests route to another, without the application changing anything.
Where does the term come from?
Researchers at Inria introduced inference delivery networks in a 2021 paper (arXiv:2105.02510, later published in IEEE/ACM Transactions on Networking), defining them as networks of computing nodes that coordinate to serve machine-learning inference requests. OpenRelay is the commercial implementation of that idea for production AI workloads.
Do I need to change my code to use one?
No. OpenRelay's endpoint is OpenAI-compatible: point your existing SDK at the OpenRelay base URL with an OpenRelay API key and requests route across the network. Billing is per token, with no minimum spend.
Point your SDK at the network
One base URL swap and your requests route across pooled GPU capacity, with failover handled for you and billing per token.