The paper presents the Functionalizer, a pre-tokenization framework that decomposes orthographic variations (capitalization, diacritics, etc.) into reversible opcode/operand pairs encoded in the Unicode Private Use Area before tokenization, enabling smaller vocabularies without information loss. Tested at 98M parameters, it shows improved code syntax validity and reduced n-gram repetition versus standard tokenization.
