Published July 8, 2026, this article surveys how frontier AI labs implement knowledge distillation through three approaches: training smaller student models on a larger teacher’s soft labels or generated text, training specialized domain experts via reinforcement learning before merging them through on-policy distillation, and self-distillation where a model learns from an enhanced version of itself. It notes labs increasingly favor the RL-then-merge approach, where the student writes and the teacher grades every token, and shows how self-distillation supports continual learning without catastrophic forgetting.
