Kimi K3.1 Pro is live on OpenRelay
Nov 17, 2026 · 5 min read
Moonshot AI's Kimi K3.1 Pro is live on the OpenRelay API as openrelay/kimi-k3.1-pro. It is a 2.8T-parameter mixture-of-experts with 104B parameters active per token, and we serve it with a 262,144-token context, image input, reasoning and tool calls.
We run it on AMD MI355X: one copy of the model across the eight GPUs of a node, with a small draft model that halves the time per token when the model is answering one request.
Call it
curl https://inference.openrelay.inc/v1/chat/completions \
-H "Authorization: Bearer $OPENRELAY_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "openrelay/kimi-k3.1-pro",
"messages": [
{ "role": "user", "content": "Explain MXFP4 in two sentences." }
]
}'It speaks the Chat Completions API, so the OpenAI SDKs work with the base URL changed. Reasoning, tool calls and image input work as the Chat Completions guide describes, and the inference catalog has the current price.
One copy, eight GPUs
Moonshot trained the model with MXFP4 weights in the loop and ships it that way: four-bit floats with one shared scale per block of 32 values, a format MI355X runs natively. That is still 1,454 GiB. Each MI355X has 288 GB of HBM, so one copy takes a whole eight-GPU node, about 194 GB per GPU once loaded, and the rest of the memory the engine gets goes to the KV cache.
The engine is SGLang with the MI355X recipe from the SGLang cookbook: tensor parallel and decode context parallel across all eight GPUs, on AMD's AITER kernels. The model supports a 1,048,576-token window; we serve 262,144, the length we tested in the setup the API runs.
DFlash halves the time per token
Decoding a model this size is mostly reading weights: every token pulls the active experts out of HBM, and the compute units wait. Speculative decoding spends that idle compute. A small draft model proposes the next five tokens, the full model checks all five in one pass, and the tokens it agrees with are kept. We use RadixArk's DFlash draft for Kimi K3.1 (5 GiB) at a block size of five. On our runs the full model kept 3.0 tokens per pass on average.
Data
| Requests in flight | Without DFlash | With DFlash | Faster by |
|---|---|---|---|
| 1 | 12.9 ms | 6.0 ms | 2.1x |
| 8 | 16.8 ms | 9.1 ms | 1.8x |
| 24 | 22.3 ms | 15.6 ms | 1.4x |
| 48 | 30.4 ms | 21.5 ms | 1.4x |
With one request in flight, time per output token drops from 12.9 ms to 6.0 ms, and output goes from 75 to 155 tokens per second. The gain shrinks as the batch grows, because a full batch already keeps the GPUs busy: at 48 requests it is 30.4 ms against 21.5 ms.
Speculative decoding should not change the answers, and it did not. GSM8K scored 249 of 250 with and without the draft.
Where it does not help
Our test prompts are unique random words, so nothing hits the prefix cache. That is the worst case for prefill. With 8k- and 32k-token prompts under load, prefill sets the pace: one node reads about 15,700 input tokens per second with or without the draft, and DFlash leaves output flat or a few percent lower.
Data
| Prompt | Requests | Without DFlash | With DFlash | Ratio |
|---|---|---|---|---|
| 1k | 1 | 75 tok/s | 155 tok/s | 2.07x |
| 1k | 48 | 1,201 tok/s | 1,385 tok/s | 1.15x |
| 8k | 1 | 61 tok/s | 113 tok/s | 1.85x |
| 8k | 48 | 391 tok/s | 364 tok/s | 0.93x |
| 32k | 1 | 49 tok/s | 77 tok/s | 1.57x |
| 32k | 48 | 129 tok/s | 123 tok/s | 0.95x |
Agent traffic re-sends long shared prefixes that the cache serves without recomputing them, and this test says nothing about that case. We have not measured it here.
Splitting prefill from decode
On one node, long prompts and decoding share the same GPUs, so time per token climbs past 50 ms when 8k-token prompts pile up. We also ran the model across two nodes, prefill on one and decode on the other, with the KV cache sent between them over RDMA by Mooncake. Decode then holds its pace under prefill load: 6 to 8 ms per token with DFlash at every load we tried.
Data
| Setup | 1 request | 8 requests | 24 requests | 48 requests |
|---|---|---|---|---|
| One node | 13.9 ms | 27.9 ms | 54.0 ms | 52.4 ms |
| One node, DFlash | 6.3 ms | 18.4 ms | 19.6 ms | 18.5 ms |
| Split, DFlash | 6.2 ms | 7.6 ms | 6.0 ms | 6.0 ms |
The split costs throughput on prefill-heavy traffic, because its prefill runs on eight GPUs where two separate nodes would prefill on sixteen. Without the draft and with 8k-token prompts, the pair gave 623 output tokens per second at 96 requests; one node gave 391 at 48, so two would give about 780 (two nodes at once were not measured). The API runs the single-node setup with DFlash.
One setting the cookbook does not carry
With the cookbook's prefill/decode flags, every KV transfer failed with transport retry counter exceeded. Each of our nodes has eight 400G RoCE NICs, each on its own network, so a NIC reaches only the same-numbered NIC on the other node, and Mooncake was pairing different ones. Pinning each tensor-parallel rank to one NIC, the same on both nodes, fixed it:
--disaggregation-ib-device '{"0":"ionic_0","1":"ionic_1","2":"ionic_2","3":"ionic_3",
"4":"ionic_4","5":"ionic_5","6":"ionic_6","7":"ionic_7"}'The longest sweep then moved 683 GiB of KV cache between the nodes with no retransmits, and GSM8K still scored 249 of 250 on the split with DFlash.
Built on open work
The API runs upstream software unchanged: SGLang on the lmsysorg/sglang:dev-kimi-k31-pro-mi35x image (the one SemiAnalysis's InferenceX pins for this model on MI355X), AITER kernels, RadixArk's DFlash draft and Moonshot's MXFP4 weights. Thanks to all of them.
How we measured
MI355X nodes with eight GPUs each, held out of serving, running the same SGLang version and flags the API runs. One closed-loop client; every prompt is unique random words, so nothing hits the prefix cache; 256 output tokens forced; temperature 0.6; two rounds per load level (four requests at one in flight). The figures are medians. GSM8K is 250 questions, 48 at a time, at temperature 0.
Try Kimi K3.1 Pro
Create an API key and call openrelay/kimi-k3.1-pro. The Chat Completions guide has every parameter.