The Massive Legal Embedding Benchmark (MLEB)

AI summary

Overview

Research area: Natural language processing — specifically legal information retrieval and embedding model evaluation.

Technical level: Intermediate. Readers need some familiarity with retrieval evaluation concepts (queries, passages, NDCG@10, zero-shot classification) but the paper's arguments about benchmark quality and jurisdictional coverage are accessible without deep technical background.

Scope: The paper introduces MLEB, a ten-dataset, six-jurisdiction open-source benchmark for evaluating legal embedding models, and reports rankings of 21 embedding models on it.

What This Paper Is About

Embedding models turn legal documents and search queries into numeric vectors so that relevant material can be retrieved, and they power the retrieval step of legal retrieval-augmented generation (RAG) systems. Existing legal retrieval benchmarks are small and narrow — LegalBench-RAG centers on contracts sourced largely from the US, and MTEB-Legal's English datasets have labeling problems and limited topical spread — so benchmark scores do not reliably predict real-world legal retrieval quality. MLEB was built to be larger, more diverse, and higher quality, and to cover jurisdictions and document types the earlier benchmarks miss.

Key Contributions

1. A ten-dataset legal retrieval benchmark. MLEB consists of ten expert-annotated evaluation sets spanning six jurisdictions (the US, UK, EU, Australia, Ireland, and Singapore), five document types (cases, legislation, regulatory guidance, contracts, and literature), and three task types (search, zero-shot classification, and question answering).
2. Seven newly constructed datasets. Seven of the ten datasets are new, built either by having subject matter experts hand-label data or by adapting existing expert-labeled data, in order to fill domain and jurisdictional gaps outside American law and contract law.
3. A public leaderboard and reproducible evaluation. The paper reports NDCG@10 results for 21 embedding models on MLEB, and releases code, results, and data openly, including a GitHub evaluation implementation and Hugging Face dataset repositories.
4. A documented critique of prior benchmarks. The authors document labeling issues in MTEB-Legal's AILA Casedocs and AILA Statutes and argue that LegalBench-RAG's contract-and-US focus limits its usefulness.

Main Findings

• Top model ranking. As of 21 October 2025, Isaacus' Kanon 2 Embedder ranks first on MLEB out of 20 other models with an NDCG@10 tag of 86.03, followed by Voyage 3 Large at 85.71 and Voyage 3.5 at 84.07. Scores are reported both as a task average (mean across all ten evaluation sets) and a domain average (mean across domain types, weighted by number of datasets per domain).
• Domain-adapted models win. The highest-scoring models have all been optimized in some way for the legal domain. Kanon 2 Embedder was pretrained and finetuned largely on legal documents. Voyage Law 2, described as older and likely smaller than most other high-scoring models, ranks 8th (task average 79.63), ahead of OpenAI Text Embedding 3 Large (task average 78.91, rank 9).
• General-purpose retrieval strength does not transfer. Gemini Embedding ranks 1st on MTEB while Voyage 3.5 ranks 23rd there; on MLEB, Gemini Embedding is 7th and Voyage 3.5 is 3rd.
• Regulatory tasks score highest; judicial tasks are hardest. For the top model, the domain breakdown is Regulatory 91.48, Contractual 84.67, and Judicial 82.96.
• Speed varies among commercial models. Total evaluation time (including network latency) was measured using a batch size of 16 for documents and 1 for queries, except that Voyage AI models were run at batch size 1 on datasets with especially long sequences to avoid token rate limits. The paper presents score-versus-time but the specific timing values are not reported in the provided text.
• Prior benchmarks have defects. Manual inspection of MTEB-Legal's AILA Casedocs and AILA Statutes found many query-passage pairs that were totally irrelevant to each other. Those datasets were created with what the authors describe as an "automated methodology" pairing facts from Indian Supreme Court cases with cases and statutes cited by the arguing lawyers, reportedly because involving legal experts would have required significant financial resources and time.
• MTEB-Legal's remaining English coverage is thin. After excluding the two AILA datasets, two of the remaining English datasets deal with consumer terms of service, only one (Corporate Lobbying) deals with legislation, and none deals with case law. The non-English legal split contains two German datasets and one Chinese dataset and no others.
• Coverage gaps in prior benchmarks. LegalBench-RAG evaluates against four pre-existing sets — ContractNLI, CUAD, MAUD, and Privacy QA — all contract-centered and largely US-sourced.
• Evaluation exclusions. No Cohere embedding models could be evaluated because their terms of service expressly forbid benchmarking of any kind. The authors also flag potential data leakage for models from Voyage AI, Jina, and Google, whose terms of service opt a portion or all of their API users by default into sharing data for training purposes.
• Disclosed conflict of interest. Kanon 2 Embedder was created by Isaacus, the same company that sponsored the creation of MLEB.

Methodology in Plain English

The authors first surveyed existing legal retrieval benchmarks and identified what they were missing: document types beyond contracts, jurisdictions beyond the US, and reliable labels. They then assembled MLEB with four stated objectives: high quality in provenance and labeling, real-world utility for legal tech professionals, genuine difficulty requiring legal knowledge and reasoning, and broad representation of jurisdictions, legal areas, and text types.

For the new datasets, several construction patterns recur. Singaporean Judicial Keywords and the Irish and UK legislative datasets were built by collecting publicly available court judgments or laws, converting them to plain text with Inscriptis, cleaning them, removing near duplicates with the simhash algorithm, and extracting catchwords or long titles with regex — then deleting those extracted elements and everything preceding them so that models must represent the core text rather than metadata-rich cover sheets. GDPR Holdings Retrieval used regex to separate facts and holdings sections of GDPRHub articles before plain-text conversion. Australian Tax Guidance Retrieval sampled 8 questions from each of 14 ATO Community forum sub-topics, following hyperlinks in selected answers to pull relevant government passages, extracting questions and passages to Markdown with a purpose-built Chrome extension. Contractual Clause Retrieval paired 45 clause types and NLI-style statements with 2 representative examples each. License TL;DR Retrieval paired tl;drLegal summaries with full license texts.

For the pre-existing sets, Bar Exam QA was used as a challenging retrieval task requiring lawyer-like analysis, and half of SCALR and half of the MTEB version of Consumer Contracts QA were reserved for validation so that test queries do not overlap with validation queries. Evaluation is reported with NDCG@10.

Why This Matters

Impact on research. MLEB gives legal NLP a larger and more jurisdictionally varied target than LegalBench-RAG or MTEB-Legal, and it makes a methodological argument: benchmark quality and domain fit matter more than leaderboard position on general-purpose retrieval benchmarks. The finding that a model ranked 1st on MTEB places 7th on MLEB, while a model ranked 23rd on MTEB places 3rd, is a direct caution against using general retrieval scores to predict legal retrieval performance. Releasing seven new datasets also creates training and evaluation material for jurisdictions that previously had little or none.

Real-world applications.

• Legal research tools that retrieve relevant case law and statutes in response to a lawyer's question.
• The retrieval component of legal RAG systems, where the paper argues low-quality embeddings lead to low-quality search results, low-quality responses, and increased hallucinations.
• Contract review and drafting assistance that identifies clause types such as set-off, termination for cause, or most favored nation clauses from descriptions alone.
• Regulatory compliance workflows, such as matching GDPR fact patterns to relevant decisions, or matching taxpayer questions to government tax guidance.

Industry relevance. Legal tech vendors can use MLEB to select embedding models on evidence rather than general-purpose rankings, and the inclusion of evaluation-time measurements (with batch sizes reflecting real-world query batching) speaks to deployment cost as well as accuracy. The disclosure that Isaacus both created the top-ranked Kanon 2 Embedder and sponsored MLEB is relevant to how readers weigh the leaderboard.

Future Directions

• Counteracting benchmark contamination. The authors flag default data-sharing terms at Voyage AI, Jina, and Google as a possible leakage risk, which raises the question of how legal benchmarks can be kept out of training data.
• Broadening provider coverage. Cohere's terms of service prevented evaluation entirely, so results for that family of models remain unknown and a means of fair third-party benchmarking is unresolved.
• Extending datasets and jurisdictions. The authors state they intend to continue improving MLEB with subsequent expansions, and seven of the ten current datasets were built to fill specific gaps, implying further gaps remain.
• Testing whether legal adaptation generalizes. The paper observes a correlation between MLEB performance and legal domain adaptation, but whether that holds across the six jurisdictions covered and on task types beyond the three evaluated is not established here.

Target Audience

Legal tech engineers and ML practitioners choosing or training embedding models for legal retrieval; benchmark designers and NLP researchers working on domain-specific or multilingual retrieval evaluation; in-house counsel and legal operations teams evaluating legal AI tools; and dataset curators interested in expert annotation methods and jurisdictional coverage. Readers primarily interested in general-purpose embedding models will find the critique of MTEB-Legal and the MTEB-versus-MLEB ranking comparison the most relevant parts.

Authors’ abstract

We present the Massive Legal Embedding Benchmark (MLEB), the largest, most diverse, and most comprehensive open-source benchmark for legal information retrieval to date. MLEB consists of ten expert-annotated datasets spanning multiple jurisdictions (the US, UK, EU, Australia, Ireland, and Singapore), document types (cases, legislation, regulatory guidance, contracts, and literature), and task types (search, zero-shot classification, and question answering). Seven of the datasets in MLEB were newly constructed in order to fill domain and jurisdictional gaps in the open-source legal information retrieval landscape. We document our methodology in building MLEB and creating the new constituent datasets, and release our code, results, and data openly to assist with reproducible evaluations.