AMD and Meta ported PyTorch Monarch, a distributed training framework, to AMD Instinct GPUs via ROCm, enabling fault-tolerant distributed training without traditional checkpoint-restart mechanisms. The port converted CUDA code to HIP using hipify_torch, integrated RCCL for collective communications, and used Rust compatibility shims to keep the code platform-agnostic. The system was validated on 128-GPU and 256-GPU clusters training Llama models, demonstrating that healthy nodes can continue training through injected failures via dynamic peer checkpoint transfers and quorum-based synchronization, improving GPU utilization in large-scale training.
