Researchers at Multiverse Computing introduce two systems-level optimizations for LLM knowledge distillation: caching the teacher model’s top-100 logits offline so the teacher never has to be held in GPU memory during training, and a ‘fused chunked KL loss’ that processes and discards one sequence chunk at a time instead of materializing the full vocabulary-by-sequence matrix. Together the techniques cut peak GPU memory usage by up to 15.6x on long contexts, allowing distillation runs that previously required hundreds of GPUs to complete on a single GPU.
