This paper proposes the Spatial Memory Agent (SMA), a framework that improves spatial reasoning in frozen vision-language models without any parameter updates or external spatial tools. SMA converts verified spatial experiences into reusable lessons through verifier-guided reflection, and tracks each lesson’s usefulness over time with a Transfer Reliability Score that updates as it is deployed. Across five spatial reasoning benchmarks and four different base models, SMA achieved the highest macro-average performance in every base-model group tested, showing that experience-based memory can substitute for retraining.
