This paper introduces CASTER, a gradient-free test-time adaptation method for frozen neural network models that stores source class statistics and estimates affine transformations from target batches to adapt frozen features without parameter updates. The authors also propose an empirical transportability certificate that predicts when affine adaptation will fail. Across four backbones and seven datasets, the method uses 18 times less memory overhead than k-NN baselines.
