The authors identify ‘Value Flattening,’ a failure mode in PPO critics for LLM RL where true state values shift sharply across intermediate states while critic predictions stay comparatively flat, worsening as the state space grows. They trace it to an implicit variance penalty in the critic loss and redundant updates from correlated states, and propose SP3O, which applies the value loss to only a few well-separated states per response. On Qwen3-Base, supervising just three states per response mitigates Value Flattening and consistently improves the learned policy across model sizes and evaluation suites.
