Researchers identify four unbounded memory peaks in standard MoE training pipelines — expert dispatch, vocabulary projection, gradient checkpointing, and optimizer state — and show that lowering the largest only exposes the next. They introduce four matched scheduling techniques (PipelinedLLEP, Ring-DTP, selective checkpoint offload, and OffloadStreamAdamW) that cap each peak without changing the loss or gradients. Composed together on MoE models from 120B to 667B parameters, the techniques enable training at 1M-token context length, 8-32x the reach of a tuned FSDP2 baseline, with up to 10.4x higher throughput.
