The paper identifies a bottleneck in autoregressive transformers where only sampled output tokens are fed back into the model, discarding the richer hidden-state representations computed at the top layer. It proposes latent feedback, which fuses the previous top-layer hidden state with the current token embedding through a gated mechanism before it re-enters the stack, combined with a scheduled multi-pass training objective that preserves parallel training efficiency. On 1B-parameter models, the resulting full-bandwidth transformer matches the performance of a standard transformer trained on roughly 1.5 times more data, while producing more concise reasoning traces without added decoding overhead.
