Back to blog

Kimi K3.1 Pro is live on OpenRelay

Nov 17, 2026 · 5 min read

Moonshot AI's Kimi K3.1 Pro is live on the OpenRelay API as openrelay/kimi-k3.1-pro. It is a 2.8T-parameter mixture-of-experts with 104B parameters active per token, and we serve it with a 262,144-token context, image input, reasoning and tool calls.

We run it on AMD MI355X: one copy of the model across the eight GPUs of a node, with a small draft model that halves the time per token when the model is answering one request.

Call it

curl https://inference.openrelay.inc/v1/chat/completions \
  -H "Authorization: Bearer $OPENRELAY_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openrelay/kimi-k3.1-pro",
    "messages": [
      { "role": "user", "content": "Explain MXFP4 in two sentences." }
    ]
  }'

It speaks the Chat Completions API, so the OpenAI SDKs work with the base URL changed. Reasoning, tool calls and image input work as the Chat Completions guide describes, and the inference catalog has the current price.

One copy, eight GPUs

Moonshot trained the model with MXFP4 weights in the loop and ships it that way: four-bit floats with one shared scale per block of 32 values, a format MI355X runs natively. That is still 1,454 GiB. Each MI355X has 288 GB of HBM, so one copy takes a whole eight-GPU node, about 194 GB per GPU once loaded, and the rest of the memory the engine gets goes to the KV cache.

The engine is SGLang with the MI355X recipe from the SGLang cookbook: tensor parallel and decode context parallel across all eight GPUs, on AMD's AITER kernels. The model supports a 1,048,576-token window; we serve 262,144, the length we tested in the setup the API runs.

DFlash halves the time per token

Decoding a model this size is mostly reading weights: every token pulls the active experts out of HBM, and the compute units wait. Speculative decoding spends that idle compute. A small draft model proposes the next five tokens, the full model checks all five in one pass, and the tokens it agrees with are kept. We use RadixArk's DFlash draft for Kimi K3.1 (5 GiB) at a block size of five. On our runs the full model kept 3.0 tokens per pass on average.

Without DFlashWith DFlash08162432Milliseconds per output token1 request12.9 ms6.0 ms2.1x faster8 requests16.8 ms9.1 ms1.8x faster24 requests22.3 ms15.6 ms1.4x faster48 requests30.4 ms21.5 ms1.4x faster
One 8-GPU MI355X node, 1k-token prompts with no cache hits, median time per output token. Lower is better. The draft proposes a block of five tokens per step.
Data
Requests in flightWithout DFlashWith DFlashFaster by
112.9 ms6.0 ms2.1x
816.8 ms9.1 ms1.8x
2422.3 ms15.6 ms1.4x
4830.4 ms21.5 ms1.4x

With one request in flight, time per output token drops from 12.9 ms to 6.0 ms, and output goes from 75 to 155 tokens per second. The gain shrinks as the batch grows, because a full batch already keeps the GPUs busy: at 48 requests it is 30.4 ms against 21.5 ms.

Speculative decoding should not change the answers, and it did not. GSM8K scored 249 of 250 with and without the draft.

Where it does not help

Our test prompts are unique random words, so nothing hits the prefix cache. That is the worst case for prefill. With 8k- and 32k-token prompts under load, prefill sets the pace: one node reads about 15,700 input tokens per second with or without the draft, and DFlash leaves output flat or a few percent lower.

1 request48 requests0x0.5x1x1.5x2xOutput rate with DFlash, relative to without1k-token prompts2.07x1.15x8k-token prompts1.85x0.93x32k-token prompts1.57x0.95xNo change
Output tokens per second with DFlash divided by without, one node, no cache hits. Past the dashed line the draft helps; with long prompts under load the node is busy prefilling and it does not.
Data
PromptRequestsWithout DFlashWith DFlashRatio
1k175 tok/s155 tok/s2.07x
1k481,201 tok/s1,385 tok/s1.15x
8k161 tok/s113 tok/s1.85x
8k48391 tok/s364 tok/s0.93x
32k149 tok/s77 tok/s1.57x
32k48129 tok/s123 tok/s0.95x

Agent traffic re-sends long shared prefixes that the cache serves without recomputing them, and this test says nothing about that case. We have not measured it here.

Splitting prefill from decode

On one node, long prompts and decoding share the same GPUs, so time per token climbs past 50 ms when 8k-token prompts pile up. We also ran the model across two nodes, prefill on one and decode on the other, with the KV cache sent between them over RDMA by Mooncake. Decode then holds its pace under prefill load: 6 to 8 ms per token with DFlash at every load we tried.

0 ms15 ms30 ms45 ms60 ms182448Requests in flightOne node, 52.4 msOne node, DFlash, 18.5 msSplit, DFlash, 6.0 ms
8k-token prompts with no cache hits, median time per output token. The split runs prefill on one 8-GPU node and decode on another, with the KV cache sent between them over RDMA.
Data
Setup1 request8 requests24 requests48 requests
One node13.9 ms27.9 ms54.0 ms52.4 ms
One node, DFlash6.3 ms18.4 ms19.6 ms18.5 ms
Split, DFlash6.2 ms7.6 ms6.0 ms6.0 ms

The split costs throughput on prefill-heavy traffic, because its prefill runs on eight GPUs where two separate nodes would prefill on sixteen. Without the draft and with 8k-token prompts, the pair gave 623 output tokens per second at 96 requests; one node gave 391 at 48, so two would give about 780 (two nodes at once were not measured). The API runs the single-node setup with DFlash.

One setting the cookbook does not carry

With the cookbook's prefill/decode flags, every KV transfer failed with transport retry counter exceeded. Each of our nodes has eight 400G RoCE NICs, each on its own network, so a NIC reaches only the same-numbered NIC on the other node, and Mooncake was pairing different ones. Pinning each tensor-parallel rank to one NIC, the same on both nodes, fixed it:

--disaggregation-ib-device '{"0":"ionic_0","1":"ionic_1","2":"ionic_2","3":"ionic_3",
                             "4":"ionic_4","5":"ionic_5","6":"ionic_6","7":"ionic_7"}'

The longest sweep then moved 683 GiB of KV cache between the nodes with no retransmits, and GSM8K still scored 249 of 250 on the split with DFlash.

Built on open work

The API runs upstream software unchanged: SGLang on the lmsysorg/sglang:dev-kimi-k31-pro-mi35x image (the one SemiAnalysis's InferenceX pins for this model on MI355X), AITER kernels, RadixArk's DFlash draft and Moonshot's MXFP4 weights. Thanks to all of them.

How we measured

MI355X nodes with eight GPUs each, held out of serving, running the same SGLang version and flags the API runs. One closed-loop client; every prompt is unique random words, so nothing hits the prefix cache; 256 output tokens forced; temperature 0.6; two rounds per load level (four requests at one in flight). The figures are medians. GSM8K is 250 questions, 48 at a time, at temperature 0.

Try Kimi K3.1 Pro

Create an API key and call openrelay/kimi-k3.1-pro. The Chat Completions guide has every parameter.

Get API key