[HN Gopher] Document poisoning in RAG systems: How attackers cor...
___________________________________________________________________
Document poisoning in RAG systems: How attackers corrupt AI's
sources
I'm the author. Repo is here: https://github.com/aminrj-labs/mcp-
attack-labs/tree/main/lab... The lab runs entirely on LM Studio +
Qwen2.5-7B-Instruct (Q4_K_M) + ChromaDB -- no cloud APIs, no GPU
required, no API keys. From zero to seeing the poisoning succeed:
git clone, make setup, make attack1. About 10 minutes. Two things
worth flagging upfront: - The 95% success rate is against a
5-document corpus (best case for the attacker). In a mature
collection you need proportionally more poisoned docs to dominate
retrieval -- but the mechanism is the same. - Embedding anomaly
detection at ingestion was the biggest surprise: 95% - 20% as a
standalone control, outperforming all three generation-phase
defenses combined. It runs on embeddings your pipeline already
produces -- no additional model. All five layers combined: 10%
residual. Happy to discuss methodology, the PoisonedRAG
comparison, or anything that looks off.
Author : aminerj
Score : 31 points
Date : 2026-03-12 13:40 UTC (9 hours ago)
(HTM) web link (aminrj.com)
(TXT) w3m dump (aminrj.com)
| sidrag22 wrote:
| > Low barrier to entry. This attack requires write access to the
| knowledge base,
|
| this is the entire premise that bothers me here. it requires a
| bad actor with critical access, it also requires that the final
| rag output doesn't provide a reference to the referenced result.
| Seems just like a flawed product at that point.
| sandermvanvliet wrote:
| If you think about this in the context of systems that ingest
| content from third party systems then this attack becomes more
| feasible.
|
| But then, if you're inside the network you've already overcome
| many of the boundaries
| SlinkyOnStairs wrote:
| > it requires a bad actor with critical access
|
| This isn't particularly hard. Lots and lots of these tools take
| from the public internet. There's already plenty of documented
| explanes of Google's AI summary being exploited in a
| structurally similar way.
|
| For what it concerns internal systems, getting write access to
| documents isn't hard either. Compromising some workers is easy.
| Especially as many of them will be using who knows what AI
| systems to write these documents.
|
| > it also requires that the final rag output doesn't provide a
| reference to the referenced result.
|
| RAG systems providing a reference is nearly moot. If the
| references have to be checked; If the "Generation" cannot be
| trusted to be accurate and not hallucinate a bunch of bullshit,
| then you need to check _every single time_ , and the generation
| part becomes pointless. Might as well just include a verbatim
| snippet.
| zenoprax wrote:
| "bad actor" can now be "ignorant employee running AI agents on
| their laptop".
|
| Threats from incompetence or ignorance will be multiplied by
| 'X' over 'Y' years as AI proliferates. Unsupervised AI agents
| and context poisoning will spiral things out of control in any
| environment.
|
| I'm interested in the effect of this with respect to AI-
| generated/assisted documentation and the recycling of that
| alongside the source-code back into the models.
| malfist wrote:
| Almost like defense in depth is key to good security. GP is
| ignoring that a truffle defense is only good until the first
| person is tricked
| robutsume wrote:
| The "requires write access" framing undersells the risk. Most
| production RAG pipelines don't ingest from a single curated
| database -- they crawl Confluence, shared drives, Slack exports,
| support tickets. In a typical enterprise, hundreds of people have
| write access to those sources without anyone thinking of it as
| "write access to the knowledge base."
|
| The PoisonedRAG paper showing 90% success at millions-of-
| documents scale is the scary part. The vocabulary engineering
| approach here is basically the embedding equivalent of SEO --
| you're just optimizing for cosine similarity instead of PageRank.
| And unlike SEO, there's no ecosystem of detection tools yet.
|
| I'd love to see someone test whether document-level provenance
| tracking (signing chunks with source metadata and surfacing that
| to the user) actually helps in practice, or if people just ignore
| it like they ignore certificate warnings.
| alan_sass wrote:
| I've seen these data poisoning attacks from multiple perspectives
| lately (mostly from): SEC data ingestion + public records across
| state/federal databases.
|
| I believe it is possible to reduce the data poisoning from these
| sources by applying a layered approach like the OP, but I believe
| it needs many more dimensions with scoring to model true
| adversaries with loops for autonomous quarantine->processing->ing
| esting->verification->research->continue to verification or
| quarantine->then start again for all data that gets added after
| the initial population.
|
| Also, for: "1. Map every write path into your knowledge base. You
| can probably name the human editors. Can you name all the
| automated pipelines -- Confluence sync, Slack archiving,
| SharePoint connectors, documentation build scripts? Each is a
| potential injection path. If you can't enumerate them, you can't
| audit them."
|
| I recommend scoring for each source with different levels of
| escalation for all processes from official vs user-facing
| sources. That addresses issues starting from the core vs allowing
| more access from untrusted sources.
| alan_sass wrote:
| I think an interesting thing to pay attention to soon is how
| there are networks of engagement farming cluster accounts on X
| that repost/like/manipulate interactions on their networks of
| accounts, and X at large to generate xyz.
|
| There have been more advanced instances that I've noticed where
| they have one account generating response frameworks of text from
| a whitepaper, or other source/post, to re-distribute the content
| on their account as "original content"...
|
| But then that post gets quoted from another account, with another
| LLM-generated text response to further amplify the previous
| text/post + new LLM text/post.
|
| I believe that's where the world gets scary when very specific
| narrative frameworks can be applied to any post, that then gets
| amplified across socials.
| ineedasername wrote:
| Any document store where you haven't meticulously vetted each
| document-- forget about actual bad actors-- runs this risk. A
| size org across many years generates a lot of things. Analysis
| that were correct at one point and not at another, things that
| were simply wrong at all times, contradictory, etc.
|
| You have to choose model suitably robust is capabilities and
| design prompts or various post training regimes that are tested
| against such, where the model will identify the different ones
| and either choose the correct one on surface both with an
| appropriately helpful and clear explanation.
|
| At minimum you have to start from a typical model risk
| perspective and test and backtest the way you would traditional
| ML.
___________________________________________________________________
(page generated 2026-03-12 23:00 UTC)