Researchers have proposed Deep Fusion Memory (DFM), a technique that lets large language models draw on external knowledge without the inference-time overhead of retrieval or the catastrophic forgetting risk of continued pretraining. DFM uses a causal predictor to generate a memory from preceding context, supervised by retrieved chunk embeddings, then fuses that memory into the model’s intermediate hidden states through gated cross-attention, trained end to end on a frozen backbone. The method needs 32 times fewer retrieval queries and 2,048 times fewer retrieved items to build its training data than prior approaches, and outperforms the strongest baseline by 12.71% on Mistral-7B-v0.3 and 11.19% on Qwen3-8B-Base across six QA tasks, with average domain-adaptation gains of 6.51 and 4.38 points on Qwen3-4B-Base and Qwen3-8B-Base respectively.
