Number Research
← Back to Explore
Information Systems Information Retrieval Search Evaluation Metrics

Search Engines Quietly Abandoned Recall, Not Just Improved It

Bing's own tests show GPT-4 judges match human raters only about as often as humans match each other — while nobody defines the denominator recall requires

S.J. Nam 7 min read
Search Engines Quietly Abandoned Recall, Not Just Improved It

A Number Nobody Can Compute

Ask a search engineer for the recall of their system and watch the pause. Precision they can give you in a heartbeat — of the ten results shown, how many were relevant. Recall requires knowing the total number of relevant documents that exist anywhere in the collection, a denominator nobody has ever actually counted for the modern web, or for the sprawling, shape-shifting context window of a retrieval-augmented AI answer. This isn't a rounding error or an engineering oversight. It's a structural crack in the foundation of how search quality gets measured, and it has been there since the metric was born in a British aeronautics college in the early 1960s. The crack didn't matter much when Google was returning ten blue links. It matters enormously now that "search" increasingly means a language model synthesizing a paragraph you never asked to fact-check.

The sharpest thing to say about search-quality statistics in 2024 isn't that they're wrong. It's that the industry has quietly stopped even pretending to compute half of them, and has replaced the human judgment that made the other half meaningful with judgments from the very same class of model being judged.

The Cranfield Bargain

Precision and recall come from the Cranfield experiments, a series of test collections built in the 1960s under Cyril Cleverdon to compare rival indexing schemes. The deal Cranfield struck — fixed document collection, fixed set of queries, human experts exhaustively marking every document relevant or not — is what statisticians would call a controlled, closed-world design. It's the reason the field could talk about precision and recall as real, computable quantities at all: someone had actually enumerated the relevant set.

The National Institute of Standards and Technology inherited this bargain when it launched the Text REtrieval Conference (TREC) in 1992, and immediately had to cheat on it a little. The web-scale collections were too large for exhaustive judging, so TREC introduced "pooling" — take the top-k results from every competing system, union them together, and have humans judge only that pool, treating everything outside it as not relevant. It's a clever statistical approximation, and it works reasonably well when every competitor is drawing from the same fixed universe of documents and returning a ranked list from it. It works far less well when one of your "systems" is a language model that doesn't select documents at all — it generates prose that may paraphrase, blend, or hallucinate its way past the boundaries of any candidate pool a human ever judged. You cannot pool what was never retrieved. You cannot compute recall against a denominator that was never defined. The statistical machinery that took three decades to mature was built for a retrieval problem that generative AI search has partially stopped posing.

Grading the Grader

Faced with that gap, the industry's dominant fix has been to stop using human judges for the bulk of relevance labeling and use large language models instead — "LLM-as-judge." The appeal is obvious: a GPT-4-class model can label thousands of query-response pairs in the time a human panel labels dozens, at a fraction of the cost. The practice has real empirical support behind it. Researchers at Microsoft, evaluating Bing search results, found that GPT-4's relevance judgments agreed with expert human labelers about as often as two human labelers agreed with each other — a genuinely surprising result, since inter-annotator agreement among trained human assessors has always been treated as the gold-standard ceiling, not something a model would casually match.

But matching human-to-human agreement is not the same as being an independent arbiter, and this is where the statistical rigor quietly erodes. The Cranfield paradigm's power came from the judge being external to and uncontaminated by the systems under test. An LLM judge scoring the outputs of an LLM-based search system violates that independence in ways researchers have already documented: evaluator models exhibit measurable position bias (preferring whichever answer appears first in a comparison), verbosity bias (rewarding longer answers regardless of content), and self-enhancement bias, where a model rates outputs from its own family more favorably. The paper that first cataloged these effects systematically, on the MT-Bench and Chatbot Arena benchmarks, found GPT-4 agreed with human preferences roughly 80 percent of the time — respectable, but nowhere near the near-total agreement implied when a company reports a single "quality score" without a confidence interval attached. Combine an 80 percent-agreement judge with a system built by the same lab, evaluated internally, reported in a blog post rather than a peer-reviewed track, and you have something that looks like the Cranfield paradigm — same vocabulary, same-looking charts — while lacking the one property that made Cranfield trustworthy: a judge nobody had an incentive to please.

The Vanishing Half of the Metric

Here's the detail that gets lost in most coverage of "AI search quality": recall hasn't been fixed for the generative era — it has essentially disappeared from the vocabulary. Open the evaluation documentation for most retrieval-augmented generation (RAG) systems and you'll find faithfulness scores, answer relevance scores, context precision — metrics from frameworks like RAGAS that ask, in effect, "does the retrieved context support the generated claim, and is the claim relevant to the question." What you generally will not find is any recall analog, because nobody can specify the full set of passages a perfect system should have retrieved from an open-ended web or document corpus. Precision-flavored metrics survive because you only need to judge what was shown. Recall-flavored metrics require judging what wasn't shown, and that problem hasn't gotten easier just because the model got smarter.

The consequence is subtle but real: a search system can score extremely well on every metric currently in wide industry use while systematically missing an entire category of relevant information it never surfaced, because nothing in the measurement pipeline was designed to notice an absence. Precision without recall is a metric that can be gamed by narrowing your ambition, and that's precisely the failure mode users experience as an AI search product that answers confidently while ignoring an inconvenient source.

What Rigor Would Actually Require

None of this means precision, recall, and NDCG are obsolete — Järvelin and Kekäläinen's 2002 formulation of normalized discounted cumulative gain remains the right instrument for graded relevance in a ranked list, discounting a relevant document's value by how deep it sits in the results. The instrument isn't broken. What's broken is the assumption, carried over unexamined from the ranked-list era, that the same statistical apparatus applies unchanged when the output is generated text rather than a selection from a known set. Real rigor would mean publishing confidence intervals alongside every "quality" number the way TREC organizers insist on significance testing before declaring one system better than another; disclosing which judge model scored which system and how often that judge's parent company built the system it graded; and building sampling frames that at least attempt to estimate a recall base, however imperfect, rather than abandoning the concept because it's inconvenient at scale.

Search companies have every incentive to keep the vocabulary of Cranfield — it sounds scientific — while quietly discarding the parts of Cranfield that made it trustworthy. The question worth asking of any AI search product touting its "relevance score" isn't whether the number is precise. It's who computed the denominator, and whether they had any reason to make it small.

References