This paper introduces the Superposition Linearity Hypothesis: when inputs from distinct text streams are linearly combined and fed to an LLM, the output is approximately a superposition of the next-token distributions the model would produce for each stream individually. The authors present evidence this linearity is an intrinsic property of the Transformer architecture rather than something learned during training — it actually tends to diminish as pretraining progresses — but can be substantially restored via lightweight fine-tuning. They introduce a guided decoding procedure that exploits this property to disentangle the superposed outputs back into their component predictions.
