vLLM Boosts Kimi K3 Speed on NVIDIA Blackwell GPUs

Inference engine vLLM has introduced a specialized decode path tailored for Moonshot AI's Kimi K3 model running on NVIDIA GB300 systems. Written in NVIDIA's low-level CuTe DSL, the optimized backend fuses key attention, mixture-of-experts routing, and inter-GPU communication tasks. Running on a cluster of 16 GB300 GPUs with DSpark speculative decoding and a synthetic acceptance length of six tokens, the system achieved 842 tokens per second at batch size 1, marking a 1.73x speedup. Even without speculative decoding, the update delivers 1.5x to 2x higher decode throughput across TP8 and TP16 tensor-parallel setups.

The performance gain stems from trimming execution overhead. Within a single Kimi K3 layer, kernel launches dropped from 23 to 13, cutting execution runtime from 130 to 69 microseconds. For real-world coding workloads on SemiAnalysis AgentX, the build yielded a 1.8x boost in P90 interactivity, jumping from 223.7 to 403.2 tokens per second per user at concurrency 1. Technical enhancements include shape-aware matrix multiplication, fused NVLink all-reduce projections, sharded MoE routing on separate CUDA streams, and weight prefetching.

For practitioners, activating the specialized path requires setting the decode backend configuration to latency_specialized for TP8 or TP16 NVLink setups handling decode batches between 1 and 64. The release also acts as a testbed for a proposed fast-track RFC governing agent-written GPU kernels, which enforces strict isolation and reproducible benchmarks after resolving edge-case compiler bugs during MRCR and AIME25 evaluations. Similar optimizations are expanding across the ecosystem: a pull request adds the latency_specialized path to DeepSeek-V4.1-Flash, while early tests for Meta's Muse Spark Open reached over 950 tokens per second per user with seven draft tokens, with GLM 5.3 support planned next.