The paper studies how 13 different language model checkpoints degrade under recursive training on model-generated text, using a shared corpus across five generations and measuring output diversity via unique 4-gram counts, finding a five-fold spread in collapse rates across models. The authors show parameter scale alone does not explain the differences, and that a cheap two-to-three generation self-iteration test can predict a given model’s fragility to collapse. They test interventions and find that tightening top-p sampling nearly stops collapse within three generations, whereas data filtering only slows degradation, giving practitioners a concrete lever for guarding synthetic-data training pipelines.
