Researchers studied how far pretrained transformers actually propagate reference chains through their depth, finding that thirteen base models reliably followed only 1.4 to 3.6 lines of a program before losing track, with additional pretrained loops adding little improvement. A small rank-8 LoRA adapter trained at a single early layer, with all other weights frozen, substantially extends this capacity: Qwen3-8B’s exact-match accuracy on 24-line reference chains rose from 15.5% to 99%, and a longer-trained LoRA reached 50-line chains, with Ouro-1.4B reaching 60 lines after four loops. Analysis traced the effect to a relay mechanism in which program-line identities pass through a narrow band of middle layers, with frozen attention heads reading progressively further up the chain until parent-line attention is removed. The same task-specific LoRA approach also improved performance on the MuSiQue multi-hop QA benchmark.
