Number Research
← Back to Explore
Information Systems Technology Adoption and Management Artificial Intelligence in Agriculture

The Guessing Instinct That Retrieval Never Touches

OpenAI's own researchers found a leading model invented three different wrong birthdays rather than admit it didn't know — and better search results won't fix that reflex.

S.J. Nam 7 min read
The Guessing Instinct That Retrieval Never Touches

Three Wrong Birthdays

Ask a leading open-source language model a simple factual question — what is researcher Adam Tauman Kalai's birthday — and tell it plainly to answer only if it actually knows, using a DD-MM format. Do this three separate times. In a test run described in a September 2025 OpenAI research paper, the model produced three different, confidently stated, entirely wrong dates: "03-07," "15-06," and "01-01."
a state-of-the-art open-source language model output three incorrect dates: "03-07", "15-06", and "01-01," even though a response was requested only if known.
The real date falls in autumn. None of its three guesses came close, and it never once said the three words that would have made it accurate: I don't know.

That small experiment is more useful than most explainers on "AI hallucination," because it isn't really a story about broken software. It's a story about incentives. The model wasn't malfunctioning. It was doing exactly what its training taught it to do — and that distinction is the key to understanding both why hallucinations happen and why retrieval-augmented generation, the technique of feeding a model retrieved documents before it answers, helps as much as it does without ever fully solving the problem.

The Test Nobody Told You Was Multiple Choice

The most useful reframing to come out of recent hallucination research is this: a language model behaves less like a liar and more like a nervous student sitting a multiple-choice exam who has learned that a guess might earn credit while a blank space never will. OpenAI's paper, co-authored by Adam Tauman Kalai, Ofir Nachum, Santosh Vempala, and Edwin Zhang, argues that
language models hallucinate because standard training and evaluation procedures reward guessing over acknowledging uncertainty
. The paper's abstract is blunter still:
hallucinations persist due to the way most evaluations are graded -- language models are optimized to be good test-takers, and guessing when uncertain improves test performance
.

This matters because it locates the problem somewhere counterintuitive. We tend to imagine hallucination as a technical glitch that better engineering will eventually stamp out — more parameters, more training data, better architecture. The OpenAI team's argument suggests something closer to a grading-policy problem. If every benchmark that determines whose model "wins" scores a wrong, confident answer no worse than an honest "I'm not sure," then the training process will systematically favor confident guessing, because on average it scores higher. A model that hedges on every uncertain question will look worse on a leaderboard than one that bluffs — even though the bluffer is objectively less trustworthy. The paper's own authors describe this as an "epidemic" that can't be fixed by adding yet another hallucination-specific test; it requires rewriting how the existing, leaderboard-dominating benchmarks assign credit in the first place.

Closed-Book by Design

Underneath the incentive problem sits a structural one. A language model is, mechanically, a system trained to predict the next word in a sequence based on patterns absorbed from enormous quantities of text. It has no built-in ledger of verified facts to consult, no internal flag that lights up when it strays from something it actually "knows" into something merely plausible-sounding. Everything it produces, true or false, comes from the same statistical process. OpenAI is explicit that this is not a solved problem even in its newest systems:
GPT‑5 has significantly fewer hallucinations especially when reasoning, but they still occur.
Notice what that concedes — not that hallucination is a bug being patched out release by release, but that, as the company puts it,
hallucinations remain a fundamental challenge for all large language models
, present by nature rather than by accident.

Think of the default mode of a large language model as a closed-book exam. The student memorized an enormous amount during study — trillions of words' worth — but once the exam starts, there's no dictionary on the desk, no textbook to flip open, nothing but recall and inference. For common, well-attested facts, recall works fine. For the birthday of a specific researcher, or a niche legal citation, or last quarter's revenue figure for a mid-cap company, recall is thin, and thin recall plus a strong incentive to answer anyway produces exactly what we call hallucination: a fluent, structurally correct sentence built around a fact that was never actually there.

Handing the Model an Open Book

Retrieval-augmented generation, first proposed by researchers including Patrick Lewis and collaborators in 2020, attacks this from a different angle than trying to make recall better. Instead of asking the model to answer purely from what it memorized during training, a RAG system first searches an external source — a document database, a company's internal files, the live web — for passages relevant to the question, then hands those passages to the model as part of its prompt before it generates an answer. The closed-book exam becomes an open-book one. The model is no longer required to remember Adam Kalai's birthday; it's required to read a paragraph that states it and report what the paragraph says.

This is a genuinely different task, and it's why RAG has become close to a default architecture for any AI product that needs to answer questions about specific, checkable, or fast-changing information — customer support bots pointed at a company's own documentation, research assistants pointed at a corpus of papers, coding tools pointed at a specific codebase. Grounding the model's output in retrieved text plays to the strength language models actually have: they are very good at summarizing, synthesizing, and rephrasing information that's sitting directly in front of them. They are much less reliable when asked to reconstruct a fact from the fog of statistical association alone.

What the Open Book Doesn't Fix

Here is the part an accessible explainer of RAG often skates past: an open-book exam does not stop a nervous test-taker from guessing. It just changes what the guess is about. If the retrieval step returns several documents and none of them actually contains the answer, the same incentive structure described above is still operating underneath — the model has still been trained in an environment where a wrong, fluent answer often scores no worse than an honest admission that the retrieved material doesn't say. So it fills the gap the same way it always has: plausibly. This is why RAG systems can still hallucinate confidently, sometimes in ways that are harder to catch, because the presence of citations and retrieved snippets on the screen creates an appearance of verification even when the actual leap from source to sentence was invented.

That's not a reason to dismiss retrieval — it measurably narrows the space in which a model has to improvise, and improvisation is where hallucination lives. But it's a reason to be precise about what problem RAG actually solves. It solves the closed-book problem. It does not touch the grading problem. A model can have the correct document sitting right in its context window and still choose to answer the guessing student's way, because nothing in its training ever punished that choice more than it punished honesty.

The Grade We Keep Giving

The uncomfortable implication is that the more durable fix isn't a better retrieval pipeline or a bigger model — it's a willingness to reward "I don't know" in the benchmarks and products that determine which systems get built and sold. That's a much harder sell than shipping a new feature, because a chatbot that frequently declines to answer looks, in a demo, like a worse product than one that always has something confident to say. Until the incentives that grade these systems change, retrieval will keep doing valuable, partial work — handing the guesser better material to guess from — while the guessing instinct itself waits underneath, undisturbed, for the next question nobody thought to check.

References