The paper studies long-context extension of Gated DeltaNet, a linear-attention architecture, through the spectral properties of its state transition matrix rather than naive continued pretraining. It identifies two governing factors for long-range retrieval — a broad slow spectral band matching the target dependency length, and preserved fast-decaying modes for state clearing — and proposes SpectralShift, which reparameterizes the alpha-projection initialization and applies a tailored learning-rate scaling. Experiments show SpectralShift consistently improves long-context capability during continual pretraining, and the code is open-sourced.
