This study systematically compares Evolution Strategies (ES) against Group Relative Policy Optimization (GRPO) as post-training paradigms for LLM reasoning. It finds that ES achieves broader Pass@K reasoning coverage than GRPO, which suffers from entropy collapse, while GRPO retains an edge on Pass@1 — leading the authors to propose a sequential GRPO-then-ES training strategy that combines both strengths. The paper also shows ES’s gains come from a sparse subset of large-magnitude parameter updates rather than uniform drift across the model.
