Researchers introduce a lightweight, training-free audit for prefix invariance — the property that a sequence model’s representation at any position should not depend on future tokens — requiring only two forward passes to produce per-layer causality scores. The paper shows that the standard verification method, inspecting attention masks, fails to catch violations introduced through scans, aggregations, or normalization layers: across 192 injected faults in eight model checkpoints, mask inspection found none while the new audit localized all 192. Applying the method to production models, the authors uncovered genuine inter-chunk axis defects in Zamba2 and Nemotron-H’s chunked-scan implementations, and note the audit fits on a single page of code and runs in seconds.
