The third installment in Hugging Face’s “Profiling in PyTorch” series examines how different attention mechanisms perform under GPU profilers. The authors profile several implementations, from naive hand-written attention to PyTorch’s optimized backends (math, efficient, flash, and cuDNN), finding that specialized kernels such as FlashAttention-2 outperform the math backend by keeping intermediate matrices on-chip rather than moving them to main memory. The piece encourages developers to form predictions before examining profiler traces in order to build intuition about unexpected performance behavior.
