Researchers propose Comparison-based Preference Optimization (ComPO), a zeroth-order method for aligning large language models with human preferences that extracts directional information from preference pairs via comparison oracles instead of optimizing a differentiable preference loss directly, aiming to mitigate likelihood displacement seen in direct alignment methods. The paper establishes a convergence guarantee for an offline variant under smoothness and gradient-sparsity assumptions, and introduces an online variant that adds reverse-KL control using unlabeled policy generations. Experiments on Mistral, Llama, Gemma-2, Qwen3, and Gemma-3 models show improvements over existing direct alignment baselines, including length-controlled win rates.
