The vLLM project detailed Distributed Layerwise Offload, a technique that lets large diffusion transformer models run across multiple GPUs or NPUs by combining meta-device initialization, AllGather-based weight sharding, double-buffered prefetching, and data-parallel multi-concurrency, keeping only two weight layers on-device at a time. For a 33GB model split across four devices, the technique cut cold-start memory from 178GB to 47GB, and on NVIDIA B300 GPUs with four concurrent requests it achieved 1.39x the throughput of HSDP while using only 30% of the HBM, a step toward efficiently scaling models beyond 200 billion parameters.
