A new paper proposes replacing a small language model’s learned embedding table, often its largest set of parameters, with a frozen table of 512-bit codes derived once via principal-component projection and sign binarization of a pretrained embedding. The resulting table is about 16 MB and contains no trainable parameters. In controlled from-scratch training, the frozen-table model matched or beat a learned-table model on most benchmarks, and when paired with a ternary model body produced a 46 MB model that ran several times faster than an 8-bit float model on a laptop CPU while using about one-seventh the memory.
