Bellman Policy Optimization (BPO) is a critic-free reinforcement learning method for LLM reasoning that reformulates Policy Mirror Descent as a trajectory-level objective using the Bellman equations, avoiding the need to estimate state values at intermediate generation steps. The authors prove this reformulation shares the same unique optimal solution as the original objective and derive a practical loss whose correction term is a smoothed ratio of complementary token probabilities. Experiments on mathematical reasoning benchmarks demonstrate BPO’s effectiveness for reinforcement learning with verifiable rewards.
