A PyTorch engineering post presents kernel fusion techniques for normalization operations such as LayerNorm and RMSNorm that reduce memory-IO overhead. It introduces three approaches: a naive fusion strategy suited to small tensor dimensions, Lazy Pre-Norm, which fuses pre-RMSNorm into GEMMs by deferring elementwise multiplication to the epilogue, and Multi-CTA Norm, which uses CTA clusters and distributed shared memory to enable post-norm fusion. The work culminates in FlashNormAttention, a kernel that fuses both pre- and post-norm operations into an attention kernel, achieving up to a 35% speedup through optimized memory buffers, warp specialization, and pipeline efficiency.
