Normalized Low-Rank Adaptation (NoRA) normalizes the down-projection matrices during LoRA training to improve optimization stability, and can also be applied only at initialization to enhance standard LoRA without repeated normalization throughout training. Across pretraining, supervised fine-tuning, and reinforcement learning experiments, NoRA accelerates convergence, improves performance and training stability, and mitigates catastrophic forgetting with no added computational overhead.
