Researchers introduce WUSH-KV, a method for low-bit quantization of the key-value cache used in long-context LLM inference. The technique builds a data-aware transform from the second-order statistics of keys and values, applying the value transform within the model weights and the key transform after rotary position embeddings. Integrated into the SGLang inference engine and evaluated with a clipped quantizer called QuEST INT, WUSH-KV reduces layerwise reconstruction error and matches or outperforms the existing OSCAR transform across tested models at 2-bit precision. The approach targets the memory and bandwidth bottleneck that limits efficient long-context serving as context length and batch size grow.
