This paper shows that common off-policy RL stabilizers behave differently depending on data regime: parameter normalization helps under narrow replay coverage but hurts when data is abundant from massively parallel simulation, while clipped double-Q can be relaxed under high-throughput training. Building on these findings, the authors propose WarpSAC, a regime-aware family of off-policy RL algorithms with variants tuned for CPU-scale and GPU-parallel training. WarpSAC raises a robot box-transport task’s success rate from 19.8% to 96.4% and achieves 36.4% faster sim-to-real deployment on a Unitree G1 humanoid than the prior FlashSAC method.
