Meta details the hardware-software co-design behind doubling training efficiency for GEM, its LLM-scale ads recommendation foundation model, including custom recommendation kernels such as Jagged Flash Attention and Generalized Dot-Product Attention alongside mixed ultra-low-precision training with MXFP8 to raise per-GPU utilization. For distributed training across thousands of GPUs, the team deployed topology-aware 5D parallelism that combines FSDP and expert parallelism for dense parameters with fully sharded 2D model parallelism for sparse parameters, plus SM-free communication and advanced activation checkpointing. These changes reportedly reached 20-25% Model FLOPs Utilization while scaling total training compute 4x over twelve months.
