The paper tackles a known failure mode in knowledge distillation where student models gain single-attempt accuracy but lose the output diversity of their teacher. The authors introduce First-Order Local Entropy Influence, a metric that flags which training updates harmfully contract entropy, and build Influence-Directed Adaptive On-Policy Distillation, which preserves entropy-expanding updates while replacing entropy-contracting ones, using only sampled-token information rather than full-vocabulary data. On reasoning benchmarks, the method improves multi-attempt performance while retaining single-attempt gains, matching stronger baselines at lower computational cost.
