The paper introduces RAG-Safety-Bench, a benchmark isolating how retrieval-augmented generation affects LLM safety by testing four controlled conditions: no-RAG baseline responses, RAG with documents containing harmful answers, RAG with thematically related but safe documents, and RAG with random safe documents. Applying this methodology across five open-source LLMs, the authors find that a model’s baseline safety behavior does not reliably carry over once retrieval is introduced, and that even benign retrieved documents can inadvertently trigger unsafe outputs. The benchmark is proposed as a diagnostic tool for teams building production RAG pipelines to test safety regressions introduced specifically by retrieval.
