Researchers propose ‘disaggregated quantization,’ which applies distinct quantization schemes to the prefill and decode phases of LLM inference instead of one uniform format, since prefill benefits from low-precision compute while decode benefits from compact weights that cut memory traffic. They show removing activation quantization during decode improves accuracy on decode-heavy tasks at no extra inference cost, and that training prefill-specific weights preserves accuracy at 2-3 bit decode precision. On a 27B model, their NVFP4 prefiller substantially improved MMLU-Pro and MMMU-Pro accuracy without altering the decode checkpoint, and an offloaded disaggregated-prefill setup delivered a 1.78x time-to-first-token speedup at 8K prompt length; the method was validated on models up to 2.8T parameters.
