The vLLM team details engineering work to serve Moonshot AI’s Kimi K3 model at production scale, including a prefix-caching design that separates the physical KDA state-block size from prefix-match granularity, enabling partial cache hits without storing recurrent state at every attention block. Additional optimizations include fused kernels for Kimi Delta Attention, an optimized Attention Residual implementation, and MXFP4 quantization integrated with SiTU activation across both NVIDIA and AMD hardware, addressing serving challenges from the model’s hybrid attention design, extreme sparsity (896 routed experts), and native multimodal capabilities.
