NVIDIA REPORTEDLY REACHES 423 TOKENS PER SECOND PER USER ON KIMI K3 · DIGG

What happened: SemiAnalysis reports Nvidia reached 423 tokens per second per user on the 2.8-trillion-parameter Kimi K3 model through a new upstream vLLM optimization. Key numbers: 423 tokens per second per user; 2.8-trillion-parameter Kimi K3; 1.68 times faster on the same hardware; 12,000-line megakernel. Why it matters: The report credits an Nvidia CentML engineer with writing decode-specialized kernels in Cute DSL, and says Nvidia is working on DeepSeek v4.1 and other models for vLLM.

SemiAnalysis reports Nvidia reached 423 tokens per second per user on the 2.8-trillion-parameter Kimi K3 model through a new upstream vLLM optimization.

The report credits an Nvidia CentML engineer with writing decode-specialized kernels in Cute DSL, and says Nvidia is working on DeepSeek v4.1 and other models for vLLM.