LeMAJ (Legal LLM-as-a-Judge): Bridging Legal Reasoning and LLM Evaluation

AI summary

Overview

Research area: Natural Language Processing, specifically LLM evaluation (LLM-as-a-Judge) applied to legal question answering over legal documents.

Technical level: Intermediate. The paper combines LLM prompting, a custom tagging schema, correlation statistics (Pearson correlation, bucketed accuracy), and fine-tuning/cost trade-off experiments. Familiarity with evaluation metrics like BLEU, ROUGE, BERTScore, BARTScore, and LLM-as-a-Judge frameworks such as DeepEval helps, but the core idea is explained without heavy mathematics.

One-sentence scope: The paper introduces LeMAJ, a reference-free evaluation framework that decomposes legal answers into "Legal Data Points" (LDPs) and tags each one as correct, incorrect, irrelevant, or missing, and shows it aligns better with human legal expert judgment than a range of existing baselines on a proprietary dataset and on LegalBench.

What This Paper Is About

Evaluating LLM answers in the legal domain is hard: reference-based methods need expensive expert-written ground truths, and reference-free LLM-as-a-Judge methods tend to be unreliable and inconsistent in legal settings. The authors argue that automated evaluation correlates better with human legal experts when it mirrors the way lawyers actually review answers — reading an answer, breaking it into individual assertions, checking each for factual accuracy and relevance, and spotting critical omissions. LeMAJ operationalizes that process as an automated pipeline and tests whether it beats existing methods.

Key Contributions

1. A new evaluation framework (LeMAJ) for question answering over legal documents that mimics lawyers' reasoning without requiring reference data, based on segmenting answers into Legal Data Points (LDPs) — self-contained units of information — and tagging each one as <correct>, <incorrect>, <irrelevant>, or <missing>. The framework has two components: an automated evaluation producing Correctness and Relevance scores, and a user interface that displays LDPs for annotation by human legal experts.
2. Empirical evidence of superior performance on both a proprietary dataset and an open-source dataset (LegalBench), compared against non-LLM baselines (BLEU, ROUGE, BERTScore, BARTScore) and out-of-the-box LLM-as-a-Judge baselines (DeepEval Answer Relevancy, Faithfulness, Correctness, and Hallucination), with improved alignment to human evaluation and improved inter-annotator agreement.
3. A breakdown of time savings in a commercial use case, using LeMAJ to triage answers so human legal experts review only flagged answers.
4. Open-sourced Legal Data Points for a subset of LegalBench used in the experiments, so the research community can replicate the results.

Main Findings

• Relevance alignment (Table 2): On the proprietary dataset, LeMAJ achieved a Pearson correlation of 0.370 (p = 1.46 × 10⁻²⁹) with a Bucketed Accuracy of 0.50; on LegalBench it achieved 0.354 (p = 2.13 × 10⁻⁶) with Bucketed Accuracy of 0.35. All reported baselines were lower in Pearson correlation on both datasets, including BERTScore (0.174 proprietary; 0.055 LegalBench), BARTScore (0.105; 0.205), and DeepEval Answer Relevancy (0.000; 0.079).
• DeepEval's higher Bucketed Accuracy on relevance is misleading: DeepEval methods reached higher Bucketed Accuracy on relevance (for example, DeepEval Answer Relevancy at 0.37 proprietary and 0.45 on LegalBench, and DeepEval Faithfulness at 0.48 and 0.41), but the authors explain this is because those methods tend to give nearly perfect scores to every answer, and 48.2% of the answers are fully relevant, so they are "correct" around half the time. Their low correlation with human scores (0.079 on LegalBench for Answer Relevancy) confirms this.
• Correctness alignment (Table 3): LeMAJ outperformed all baselines on both datasets, with Pearson correlation of 0.259 (p = 7.54 × 10⁻¹⁵) and Bucketed Accuracy of 0.95 on the proprietary dataset, and Pearson correlation of 0.700 (p = 2.52 × 10⁻²⁶) with Bucketed Accuracy of 0.88 on LegalBench. For comparison, DeepEval Correctness reached 0.077 and 0.018 Pearson correlation, with Bucketed Accuracy 0.43 and 0.24 respectively.
• Inter-annotator agreement improves for correctness (Table 4): Across 150 QA pairs in the LegalBench subset, average inter-annotator agreement on Correctness rose by 11%, from 0.77 with manual review to 0.88 with LeMAJ. Relevance agreement barely moved (0.53 manual versus 0.54 with LeMAJ), which the authors attribute to the inherently subjective nature of relevance in legal work.
• Error analysis: Human reference data contained 2144 LDPs; LeMAJ (Claude Sonnet 3.5 v2) split the same dataset into 1964 LDPs, a difference of less than 10%. Of 212 tagging errors made by LeMAJ, 34 were due to differences in LDP splitting (16% of all errors). The largest error category was LeMAJ being too lenient about the level of detail when grading a data point (57 occurrences, 90 as a sum of occurrences).
• Scaling experiments: The team explored (a) prompt optimization, (b) data augmentation, and (c) an LLM Jury framework with multiple fine-tuned models. The LLM Jury was most performant, but when balancing against cost, fine-tuning with augmentation gave the best trade-offs.
• Un-fine-tuned baselines: Claude 3.5 Sonnet v2 reached a LeMAJ accuracy of 0.716 (0.757 adjusted), while Haiku reached 0.469 (0.447 adjusted), roughly 30% worse. Haiku produced no red (incorrect) tags at all and a disproportionately high share of green tags.
• Commercial triage results: Applying thresholds of a LeMAJ Correctness score of 1 and a Relevance score of at least 0.80 on the proprietary dataset (0.85 on LegalBench) produced time savings of up to 50% on the proprietary dataset and up to 30% on LegalBench.

Methodology in Plain English

The authors first interviewed four lawyers of varying seniority (junior to 5 years Post-Qualification Experience) to learn how they review legal answers. Every lawyer described decomposing an answer into individual assertions, checking each against the source document, judging whether each correct assertion actually answers the question, and looking for anything important that the answer left out.

LeMAJ turns this into a two-stage pipeline. Given a legal document, a question, an answer, and optionally a ground-truth answer, an LLM splits the answer into Legal Data Points. In the same prompt, the LLM tags each LDP: factual errors or hallucinations become <incorrect>; factually correct but off-topic points become <irrelevant>; points that are both accurate and relevant become <correct>; and information that should have been in the answer but was omitted is added as a new LDP tagged <missing>. The authors note the relevance judgment is tunable through prompting, letting the judge be more or less strict, similar to how different lawyers might differ.

Four quantitative scores follow from the tags: Correctness = #Correct / (#Correct + #Incorrect); Precision = #Correct / (#Correct + #Irrelevant); Recall = #Correct / (#Correct + #Missing); and F1 balances Precision and Recall to give the overall Relevance score. The metric is deliberately adaptable, so a task where missing information matters most can weight Recall more heavily.

For evaluation, they compared each method's score against human gold-standard scores using Pearson correlation and Bucketed Accuracy (rounding each score down to 0, 0.25, 0.5, 0.75 or 1 to match the human scale). A third metric, LeMAJ Alignment, maps each LeMAJ LDP to a human LDP using an OpenAI Embedding model and compares tags; it can only be computed for LeMAJ itself.

Datasets: a proprietary internal dataset of 9 contract types (5 contracts per type), described in the paper as totaling 1000 Q&A pairs, with Table 5 reporting 537 training and 422 testing pairs (959 total); and a LegalBench subset of twelve contracts chosen at random with 150 questions, augmented with 20 manually created incorrect or partially incorrect answers, bringing that dataset to 170. For fairness, all LLM-based methods used the same model, Claude 3.5 Sonnet v2, while answers were generated with Claude 3.5 Sonnet v1 to avoid self-enhancement bias.

Why This Matters

Reference-free evaluation matters because legal ground truths are expensive and slow to produce, and because generic LLM judges have been shown to perform poorly without good references. LeMAJ's contribution is showing that imitating the structure of expert legal review — not just asking a model for a score — measurably improves alignment with those experts, and that automated scores can be trusted enough to reduce human review load. It also provides open LDP annotations for a LegalBench subset, giving other researchers a concrete artifact to replicate and build on.

Real-world applications:

• Legal question answering over contracts: evaluating LLM answers for contract types such as Master Service Agreements, Non-Disclosure Agreements, lease agreements, SaaS agreements, and others, where correctness and completeness are critical.
• Triage of LLM outputs: flagging only contentious or at-risk answers for human legal expert review while clearing high-confidence ones, which the paper shows can save up to 50% of review time on the proprietary dataset and up to 30% on LegalBench.
• Improving human annotation quality: using the LeMAJ interface to pre-segment answers into LDPs, which improved inter-annotator agreement on Correctness by 11% and makes relevance judgments more transparent and auditable.
• Building training and iteration data: the paper notes that running LeMAJ against production tools lets organizations bucket information for training and iteration and gives users confidence about what to review.

Industry relevance: the work is a collaboration between Robin AI and Amazon Web Services, and directly targets the legal industry's pressure to "do more for less" while relying on LLM outputs. It speaks to any organization deploying legal AI tools that need defensible, scalable, expert-aligned evaluation.

Future Directions

• Increase the overall accuracy of the method, particularly improving its ability to detect incorrect and missing information.
• Extend the scalability work to a multi-agent framework that can detect the needs of a task and adapt the metric on the fly.
• Address task variability within a single LLM-as-a-Judge framework, since model-based judges are known to vary across tasks and one solution is unlikely to fit all.
• Close the gap on relevance subjectivity, where inter-annotator agreement stayed low (0.53 manual versus 0.54 with LeMAJ) because relevance is defined by the task rather than a broad industry standard.

Target Audience

This paper suits NLP and evaluation researchers working on LLM-as-a-Judge methods and domain-specific evaluation; legal AI engineers and product teams who need a reference-free way to score legal answers; legal operations and knowledge-management professionals interested in reducing review effort; and benchmark builders who want a reusable, expert-aligned annotation schema for legal question answering. Readers focused on purely technical model architecture work will find it less relevant, since its contribution is an evaluation methodology rather than a new model.

Authors’ abstract

Evaluating large language model (LLM) outputs in the legal domain presents unique challenges due to the complex and nuanced nature of legal analysis. Current evaluation approaches either depend on reference data, which is costly to produce, or use standardized assessment methods, both of which have significant limitations for legal applications. Although LLM-as-a-Judge has emerged as a promising evaluation technique, its reliability and effectiveness in legal contexts depend heavily on evaluation processes unique to the legal industry and how trustworthy the evaluation appears to the human legal expert. This is where existing evaluation methods currently fail and exhibit considerable variability. This paper aims to close the gap: a) we break down lengthy responses into 'Legal Data Points' (LDPs), self-contained units of information, and introduce a novel, reference-free evaluation methodology that reflects how lawyers evaluate legal answers; b) we demonstrate that our method outperforms a variety of baselines on both our proprietary dataset and an open-source dataset (LegalBench); c) we show how our method correlates more closely with human expert evaluations and helps improve inter-annotator agreement; and finally d) we open source our Legal Data Points for a subset of LegalBench used in our experiments, allowing the research community to replicate our results and advance research in this vital area of LLM evaluation on legal question-answering.