This paper introduces the Falcon family of fast-weight attention mechanisms for continual learning, compressing expanding context into fixed-size recurrent states through online learning rules. The authors derive normalized first-order updates for squared-error regression and inner-product objectives, presenting variants Falcon-1 (a scalar NLMS update) and Falcon-2 (a per-column extension) with numerically stable implementations, showing competitive language-modeling performance and improved length extrapolation on variable-digit addition tasks.
