The paper studies ‘long-horizon memorization,’ where a model must learn 100 sequential query-answer tasks via continual fine-tuning without retaining past examples, finding that no single continual-learning mechanism prevents catastrophic forgetting at this scale. The authors compose complementary mechanisms along two axes (data/function/weight anchors and low-rank allocation rules), using task-level successive halving and factorial experiments to measure interaction effects. Their best combination (all three anchors plus merged LoRA) raises average final retention from 1.2% under naive fine-tuning to 34.9%, a 28-fold improvement, with data anchoring and merged LoRA interacting super-additively across three constructed datasets.
