Number Research
← Back to Explore
Information Systems Knowledge Management Retrieval-Augmented Generation

RAG Doesn't Cure Hallucination, It Relocates It

Stanford caught LexisNexis and Westlaw's retrieval tools inventing law — with confident citations attached

S.J. Nam 6 min read
RAG Doesn't Cure Hallucination, It Relocates It

In the spring of 2020, a doctoral student named Patrick Lewis was splitting his time between University College London and a new Meta AI lab in the same city, trying to solve a problem that sounds almost quaint now: how do you make a language model stop guessing?
The paper he and his co-authors published that year introduced the term "retrieval-augmented generation," presented at NeurIPS, and demonstrated that RAG models generated more specific, diverse, and factual language than outputs based purely on the original model's parameters.
Lewis has since admitted the acronym was never meant to carry the weight it now does. Five years later, "RAG" is a product category, a job description, and a line item in enterprise budgets from Oracle to Pinecone.

The standard explanation you'll read everywhere is comforting and mostly wrong. It goes: language models hallucinate because they only know what they memorized, so if you let them look things up, they stop making things up. That framing treats RAG as a cure. It is closer to a transplant. RAG does not abolish the hallucination problem; it relocates it — from the opaque interior of a neural network to a retrieval pipeline that most teams underestimate and few audit. And a system that fails while citing sources is, in some ways, more dangerous than one that fails while obviously bluffing.

The Open-Book Exam That Isn't Automatically Passed

The original architecture is elegant, and worth understanding precisely because the elegance hides where things break.
Lewis's framework combined two kinds of memory: parametric memory, the knowledge baked into a pre-trained sequence-to-sequence model — BART, in their case — and non-parametric memory, an external corpus such as Wikipedia, indexed as dense vectors using Dense Passage Retrieval.

At inference time, the system retrieves relevant documents from that external memory and conditions the generator on both the query and the retrieved context.
The metaphor everyone reaches for is the open-book exam: a model that can look things up rather than recall them from memory.

But an open book helps only if you turn to the right page. The retrieval step — embedding your question, searching a vector store, pulling back the "relevant" chunks — is where the entire enterprise lives or dies.
In practice this means documents are chunked, embedded into vectors, and stored in a vector database at build time, and at query time the user's question is embedded into the same vector space.
If your chunking splits a contract clause in half, if your embedding model misreads a domain term, if the passage that actually answers the question ranks twelfth instead of third — the model never sees the truth. It sees plausible-looking neighbors and writes a confident answer grounded in them. The intelligence was never the bottleneck. The plumbing was.

Why Everyone Bought It Anyway

The commercial rush was not irrational. RAG solved three genuine, expensive problems at once.
Language models freeze at their training cutoff while the world keeps moving; they can't touch your internal policies, CRM data, or proprietary documents; and when uncertain, they fabricate confident-sounding answers.
Fine-tuning a model on your data is slow, costly, and goes stale the moment your data changes. RAG lets you swap the knowledge base without retraining anything — a modularity that made it irresistible to enterprises drowning in documents they couldn't search.

So the adoption curve went vertical.
By 2025, companies including AWS, IBM, Google, Microsoft, NVIDIA, Oracle, and Pinecone were publicly offering RAG-as-a-service — a remarkable transformation from research paper to global infrastructure in just five years.
Meanwhile the technique quietly fragmented. Lewis's paper alone described two variants — RAG-Sequence and RAG-Token — and the field has since piled on hybrid search, rerankers, and knowledge graphs. The word "RAG" on a slide now tells you almost nothing about what's actually running underneath.

The Detail the Explainers Skip

Here is the fact that should sober anyone selling RAG as a hallucination cure. When Stanford researchers stress-tested the most sophisticated commercial RAG systems in the world — the AI legal-research tools from LexisNexis and Thomson Reuters' Westlaw, products built by companies with enormous document corpora and every incentive to get retrieval right — the tools still hallucinated. Not occasionally. In a meaningful fraction of queries, they produced answers that were flatly wrong or cited authorities that didn't support the claim, often while presenting a confident, well-formatted citation. These were purpose-built RAG systems, and they were caught inventing law.

That is the counterintuitive core of the whole subject. A bare language model that hallucinates announces its unreliability; you learn to distrust it. A RAG system that hallucinates hands you a footnote. The citation is a trust signal, and when the retrieval underneath is subtly wrong, that signal becomes a liability — it makes a wrong answer look verified. Retrieval augmentation raises the ceiling on accuracy and simultaneously raises the stakes on failure. The better the packaging, the more expensive the mistake nobody checks.

What RAG Is Actually Good At

None of this is an argument against RAG. It's an argument for putting it where its failure mode is survivable. RAG shines in exactly the situations Lewis targeted: open-domain question answering, fact verification, and grounding answers in a corpus that changes faster than any model can be retrained. A support agent that pulls the current refund policy and paraphrases it, a research assistant that surfaces the three most relevant internal memos, a system that answers "what did we ship in Q3" from live documents — these are cases where a human can glance at the retrieved source and confirm it in seconds. The retrieval is auditable at the point of use.

The trouble starts when organizations treat RAG as an oracle rather than a research assistant with a filing cabinet — when the retrieved sources are hidden, the answers are trusted wholesale, and no one budgets for evaluating whether the right documents came back. The uncomfortable truth from five years of deployment is that most of the engineering value in a RAG system sits in the least glamorous layer: how you chunk, how you embed, how you rank, and how you measure retrieval quality before the model ever opens its mouth.

Lewis built a machine for looking things up. What the industry keeps forgetting is that looking something up and finding the right thing are not the same act — and that a model reciting the wrong page in perfect prose is not a smarter machine. It's a more persuasive one.

References