The paper addresses exploration-strategy management for recursive self-improvement in autonomous AI agents by building a framework that treats historical discovery data as a replay simulator, enabling off-policy evaluation of candidate strategies without repeated expensive online testing. The system cycles between refining exploration policies inside this simulator and deploying improved strategies to discover novel solutions in the real environment. Across algorithm engineering, mathematical optimization, and GPU kernel design tasks, the approach achieves discovery quality competitive with or better than baselines while substantially reducing evaluation cost.
