PyTorch engineers and AMD collaborators upstreamed FP8 training optimizations for AMD Instinct GPUs into TorchAO and TorchTitan, building on techniques first demonstrated with the Primus-Turbo library at PyTorch Conference 2025. The work added native support for AMD’s e4m3fnuz FP8 format, enabled FP8 grouped GEMM for Mixture-of-Experts architectures on ROCm, and built a Triton kernel fusion pipeline that reduced quantization overhead. On a dense Llama3-8B model, rowwise FP8 delivered a 13.4% throughput gain over BF16, while fused Triton kernels recovered 89% of the FP8 quantization overhead on DeepSeek-V3 671B MoE shapes, with individual kernel optimizations reaching up to 6.2x speedups. All of the contributions have been merged into the mainline PyTorch AO and TorchTitan repositories.
