Maglev proposes a way for transformers to maintain persistent, token-wise memory while keeping both inference cost bounded and training parallelizable. It splits the model into a causal prefiller that generates memory targets in parallel during training, and a sliding-window decoder that learns to predict both tokens and matching memories via a consistency loss, letting the decoder run independently at inference time. On a 435M-parameter model trained on 43.5 billion tokens, Maglev outperformed sliding-window and latent-recurrent baselines, improving validation loss from 0.7413 to 0.7251 bits-per-byte and average downstream task accuracy from 54.1% to 56.4%.
