Gated Recurrent Transformer is a new architecture that brackets a single shared transformer core, iterated multiple times, between fixed prelude and coda blocks, using a lightweight gating mechanism inspired by gated recurrent neural networks to modulate each recurrent update. This lets the model specialize its input to the same few layers across iterations rather than requiring many unique layers for functional diversity. Under matched compute, a 3-layer Gated Recurrent Transformer matches the accuracy of a 12-layer GPT-2 Small baseline, and at large scale the approach cuts parameters by 63% and peak decoding memory by 59% for only a 10% increase in generation latency.
