Researchers demonstrate that layer dropout, largely abandoned in modern LLM pretraining recipes, actually improves training efficiency and enables valuable post-training optimizations when properly tuned. Across more than 2,400 experiments spanning models from 271M to 8.2B parameters, the study found that layer dropout with optimal distribution, scheduling, and hyperparameters achieves lower validation loss at the same training FLOPs, saving up to 25% of compute. The technique also enables early exit, intermediate-layer skipping, and self-speculative decoding, yielding up to 1.5x inference speedup with negligible accuracy loss. All experiments were run on Cerebras CS-3 systems.
