A new study examines why hybrid attention architectures, which combine full attention with either sliding-window attention or gated linear-attention variants, behave differently when models are extended to longer context lengths. The researchers identify a “seesaw effect”: linear-attention hybrids benefit more from long-context continual pretraining, while sliding-window hybrids perform better at length extrapolation, and they trace this difference to the distinct positional inductive biases of each attention mechanism. Building on this analysis, they propose Sliding-Window Linear Attention, which achieves 16x training-free length extrapolation while maintaining full accuracy on a needle-in-a-haystack retrieval test at 64k context length.
