The Arm team collaborated with vLLM and related communities to optimize large language model serving on Arm Neoverse CPUs through improvements across the full inference stack. Key optimizations included switching to mimalloc for memory allocation, enabling hardware atomic instructions in OpenMP, implementing weight prepacking for dense layers, and accelerating paged attention with custom kernels. These enhancements delivered up to 2.7× throughput gains for BF16, 4.8× for INT8 W8A8 quantization, and 6.2× for INT8 W4A8, while also expanding feature support including chunked prefill and prefix caching.
