HyQuant is a hybrid-precision quantization framework that addresses accuracy loss in low-bit quantization of LLM attention by selectively keeping high precision for accuracy-critical regions, specifically vertical-line tokens and local-window states, while quantizing the rest of attention to lower bit-widths. During prefill it uses a specialized attention operator that preserves full precision for critical tokens and a sliding window; during decode it applies the same principle to KV-cache compression and fuses dequantization with attention computation. The technique reports near-lossless accuracy across tasks and models with minimal added computational overhead, targeting practical inference cost reduction.
