This research examines internal activation patterns in hybrid linear-attention language models and identifies two distinct phenomena: sharp pre-attention spikes that appear immediately before full-attention layers, and inter-spike plateaus that persist through the intervening linear-attention layers. Testing models ranging from 1.2B to 397B parameters across multiple architectures and datasets, the authors find both patterns emerge early in training and respond asymmetrically to output gating. Their mechanistic analysis traces the difference to timing in activation cancellation, with pre-attention spikes following localized cancellation and plateaus reflecting delayed cancellation consistent with full-attention behavior.
