StableVQ addresses training instability in vector-quantized visual tokenizers by arguing the root cause is entanglement between encoder-decoder and codebook training, where each module only functions correctly when the other happens to cooperate. It introduces three lightweight fixes — Dynamic STE to stabilize the encoder’s learning objective, a Region VQ Loss letting the codebook independently track the encoder’s output distribution, and a Decoupled Schedule giving each subsystem its own learning-rate schedule — none of which add learnable parameters. On ImageNet, StableVQ delivers consistent gains in training stability, codebook utilization, and reconstruction quality across codebook sizes and initializations.
