The paper studies expert routing as an underused source of rollout diversity in reinforcement learning post-training of Mixture-of-Experts language models, finding that perturbing expert selection increases output diversity similarly to raising decoding temperature but can degrade rollout quality if applied naively. The authors introduce Expert-Space Exploration RL (ESRL), which preserves high-confidence anchor experts, restricts stochastic routing to a plausible candidate pool, adapts perturbation strength to router entropy, and replays recorded expert paths during policy optimization to avoid routing mismatch. Across MoE backbones with top-K, top-1, and shared-expert routing on math, science, and code tasks, ESRL improves average Pass@1 and Pass@8 over standard GRPO by 3.2 and 4.5 percentage points respectively on Qwen3-30B-A3B, without extra sampling or compute cost.
