This work tackles a performance bottleneck in attention computation on Blackwell GPUs’ 4-bit floating-point tensor cores, where softmax conversion and on-chip dependencies dominate runtime once matrix products shrink. The authors introduce Direct-P for noncausal inference and a causal training path that forwards quantized values through backpropagation, achieving up to 2.13x throughput improvement over bfloat16 during inference on NVIDIA GB200 hardware. For full training updates on 8-billion-parameter models, the causal approach using FP8 gradients delivers a 1.14x speedup while maintaining numerical stability, where alternative mixed-precision schemes diverge during distributed training.
