Retrieval-Augmented Generation (RAG) poisoning
- Attacks
- Standard term
RAG poisoning is planting false information or hidden instructions in the documents an AI searches before answering, or tampering with how they are ranked, so the AI picks up what an attacker chose.
Retrieval-augmented generation (RAG) means the AI looks up passages in a document collection and answers from them. It picks passages that match the question and tends to treat them as fact. So anyone who can add to the collection can get words into answers, and a passage worded like common questions is likelier to be picked.
Where you’ll meet it
Your company's support chatbot answers customers from your help articles and the public community forum. A new forum post says refunds now go through a “verification page,” with a link. That afternoon, the chatbot starts giving customers the link.
What to do
Remove the post, and have whoever runs the chatbot drop the forum from its sources until forum posts are reviewed before use. Then ask them to search the chatbot's past conversations for the link and list each customer who got it, so you know whom to warn.
Easy to mix up
Prompt injection works by telling the AI what to do. RAG poisoning can do damage with no orders at all, since a planted false claim that the AI repeats is enough.
Check yourself
Your company's HR assistant answers from a shared drive that every employee and dozens of outside contractors can edit. A manager proposes showing a link to the source document under every answer. How much does that protect against planted text?
Go deeperA few planted passages can be enough, and tools raise the stakes
Attacks need very little text. In the 2024 PoisonedRAG study, Zou and colleagues showed that a handful of crafted passages, added to a collection of millions, could steer the answers to chosen questions most of the time.
OWASP, a nonprofit that publishes lists of common security risks, files this one under Vector and Embedding Weaknesses, its entry for flaws in how documents are stored and matched for search. When the AI can also use tools, such as sending email or editing records, an instruction inside a retrieved passage can set them off, and a wrong answer becomes a wrong action.