This paper studies the training-inference mismatch in reinforcement learning with verifiable rewards (RLVR), where the inference engine that samples rollouts and the training engine that computes gradients assign different probabilities to identical tokens. It characterizes this mismatch as an additive log-odds displacement whose distribution is approximately invariant to token confidence, and introduces calibrated importance sampling (CIS), which applies confidence-aware truncation so the importance-ratio cap tightens as token confidence increases. Theoretically, CIS bounds the second-moment error that otherwise governs exact importance sampling, at the cost of a controlled truncation bias. Across three mixture-of-experts models and five mathematical reasoning benchmarks, CIS achieves the highest five-benchmark average among evaluated baselines and places less truncation bias on low-confidence tokens than standard truncated importance sampling.
