Number Research
← Back to Explore
Law Legal Technology Embedding Models for Legal Retrieval

The Wrong Benchmark for Legal Search

MTEB's 64.6% score can't tell a general counsel which embedding model finds the controlling precedent — and a 9x price spread rides on it.

S.J. Nam 7 min read
The Wrong Benchmark for Legal Search

The number everyone repeats is 64.6. When OpenAI shipped its new embedding models in January 2024, the headline was that
its score on the MTEB benchmark increased from 61.0% to 64.6%
— a jump of 3.6 points over the aging ada-002. That figure has since been copied into hundreds of comparison tables, calculators, and procurement decks. It is also, for anyone building a legal search tool, close to useless.

That is the uncomfortable thing about the "small vs. large vs. Voyage Law-2" question. It gets framed as a spec-sheet showdown — dimensions, price per million tokens, a leaderboard percentage — when the axis that actually decides the outcome is one almost nobody in these comparisons measures.

The 6.5x Question

Start with the two OpenAI models, because they're the ones most teams reach for by default.
text-embedding-3-small produces 1536-dimension vectors and costs $0.02 per million tokens, while text-embedding-3-large produces 3072-dimension vectors and costs $0.13 per million tokens — 6.5x more expensive.
So the pitch for "large" is: pay almost seven times as much, get double the dimensions, climb the leaderboard.

Climb by how much? Here the marketing and the reality diverge. The small model is no toy —
text-embedding-3-small scores 62.26 on MTEB
. The large model, for all its extra heft,
scores about 2 points higher
. Put differently:
the MTEB retrieval advantage is roughly 2-3 percentage points, which in practical RAG testing translates to marginally better recall on domain-specific technical content and non-English text — rarely worth the 6.5x price premium for most applications.

Two or three points, for 6.5x the bill. That is the trade the spec sheet is really offering, and stated plainly it sounds like a bad one. The vendor's own guidance quietly agrees.
Where large pays off, by this account, is extremely accuracy-sensitive applications — medical, legal, financial — very long documents where the 3072-dimension space captures more nuance, and multilingual corpora with rare language pairs.
Notice that "legal" sits right in that sentence. Which is exactly where the third contender enters.

The Specialist in the Room

Voyage AI took a different bet. Instead of one general model tuned to top a general leaderboard, in April 2024 it released a model trained specifically on legal text —
domain-specific embeddings for retrieval, a "legal edition," marketed as voyage-law-2.
The premise is intuitive: a model that has seen mountains of case law, statutes, and contracts should encode the difference between "indemnify" and "hold harmless," or between a jurisdiction's procedural rules, in ways a generalist blurs together.

It is also priced like a premium product. Among the models in this class,
Voyage AI's large-tier model is the most expensive, at $0.18 per million tokens
— nine times the cost of OpenAI's small model. So a general counsel's office weighing these three is really being asked to rank three theories of value: cheapest-and-good-enough (small), more-dimensions-more-nuance (large), and trained-on-your-actual-domain (Law-2).

And to choose between them, almost everyone reaches for the same MTEB number I opened with. That is the mistake.

Why the Leaderboard Lies to Lawyers

MTEB — the Massive Text Embedding Benchmark — is a general-purpose evaluation. It rewards a model for retrieving Wikipedia passages, clustering news, scoring the similarity of everyday sentences. A 64.6% there tells you the model is a strong all-rounder. It tells you nothing about whether it will surface the controlling precedent when a litigator queries a database of appellate opinions. The skills only partly overlap, and the gap is precisely the region where a legal deployment lives or dies.

The field half-admits this. The reason "legal" keeps getting flagged as a place where you might justify the expensive model is that everyone senses general benchmarks under-serve specialized retrieval — yet the comparison tables keep quoting general benchmarks anyway, because until recently there was no rigorous alternative to quote. That changed only in late 2025, with the arrival of a purpose-built yardstick:
the Massive Legal Embedding Benchmark, MLEB
. Its very existence is the tell. If MTEB scores settled the legal question, no one would have needed to build a legal benchmark from scratch.

Sit with the implication. Voyage-law-2 shipped in 2024. OpenAI's models shipped in 2024. For well over a year, buyers choosing an embedding model for contracts and case law were comparing them on a benchmark that measures the wrong thing — the equivalent of picking a surgeon by their marathon time. The decisions weren't necessarily wrong. They were made blind, then rationalized with a number that didn't apply.

Dimensions Are Not Destiny

There's a second illusion baked into the standard comparison, and it's worth dismantling because it inflates the case for the pricey models. The tables treat 3072 dimensions as strictly better than 1536, and 1536 as better than 1024, as if vector length were a quality dial. It isn't.

OpenAI's own release quietly detonated that assumption.
On the MTEB benchmark, a text-embedding-3-large embedding can be shortened to a size of 256 while still outperforming an unshortened ada-002 embedding of size 1536.
Read that twice. A vector one-sixth the length of the old default beats the old default. The technique — Matryoshka representation — lets you
specify a smaller dimensions value, shortening the embedding down from 3072, trading off some accuracy in exchange for the smaller vector size
, and
both OpenAI and Cohere support truncating to 256 or 512 dimensions with only minor quality loss.

This matters because dimensions are not free to store.
A 1,024-dimension float32 vector takes 4KB
, and that cost scales linearly with every document you index — into real money at ten million documents. So the honest framing isn't "how many dimensions does each model have," but "how few can I get away with." A generalist model truncated to 512 dimensions may retrieve legal text worse than a 1024-dimension model trained on legal text, or it may not — and the only way to know is a legal benchmark, not a dimension count.

What the Comparison Should Have Been

The clean version of this face-off — small is cheap, large is accurate, Law-2 is specialized — is a story told in the wrong units. Price is real and knowable:
$0.02, $0.13
, and up toward
$0.18 per million tokens
is a genuine 9x spread, and for a startup embedding a modest corpus it settles the matter before accuracy enters the conversation. But the moment the corpus is legal and the stakes are a missed clause or a hallucinated citation, the general MTEB gap between small and large — two or three points — is noise, and the interesting question becomes whether domain training earns its premium against a truncated generalist.

That question now has a proper instrument. The people who should be nervous are not the buyers who paid for the specialist; they're the ones who never asked whether specialization survives measurement. A legal benchmark can vindicate voyage-law-2's whole premise — or it can reveal that a cheap generalist, fine-tuned on your own documents, closes the gap for a tenth of the ongoing cost. Either result would be more useful than another table with 64.6 in the corner. The tools to find out finally exist. The only remaining excuse for choosing an embedding model by its general leaderboard score is that reading the general leaderboard score is easier than doing the work.

References