Researchers introduce OPID, an on-policy skill distillation framework for training language agents that addresses the sparse-reward problem in long-horizon reinforcement learning tasks. The method extracts hierarchical skill supervision, covering episode-level workflows and step-level decisions, from an agent’s own completed trajectories and feeds it back as a token-level self-distillation signal alongside the standard outcome-based reward. By combining self-distillation with outcome advantages, OPID keeps reinforcement learning as the primary training objective while adding denser guidance, and the paper reports improvements on agent benchmarks including ALFWorld and WebShop.
