This paper shows that Sliding Window Attention (SWA) with attention sinks delivers comparable or superior performance to post-trained Linear Attention models across multiple large language models and downstream tasks. On long-context reasoning benchmarks such as Needle-in-a-Haystack and BABILong, SWA outperforms linear attention by 2-10 times. The authors argue that SWA requires no post-training, is extremely fast, and needs low memory, making it a more practical choice than retrofitting models with linear attention mechanisms, which may require training from scratch.
