InsufficiencyBench: Evaluating LLM legal advice on underspecified user queries

AI summary

Overview

Research area: Evaluation of large language models in the legal domain, specifically the "intake" stage of legal problem-solving — whether a model recognizes that a user's question lacks facts that materially determine the legal outcome.

Technical level: Intermediate. The benchmark mechanics (F2 scoring, judge-based extraction, hedging calibration) are standard ML evaluation material, but interpreting the taxonomy requires some familiarity with US statutory and doctrinal structure.

One-sentence scope: The paper introduces InsufficiencyBench, a 202-item, attorney-annotated benchmark of 58 fully specified legal queries and 144 deliberately deficient variants, and uses it to measure whether ten frontier models detect, identify, and avoid drawing conclusions from legally material information that the user never supplied.

What This Paper Is About

Existing legal AI benchmarks assume a query arrives fully specified — the facts, the law, and the documents are all given, and the model is graded on whether its answer is correct. Real legal interactions do not work that way: clients omit facts like jurisdiction, employer size, or whether they filed an administrative charge, and those facts often flip the outcome. The paper names the resulting failure mode premature legal closure — producing a substantive legal answer before the legally material inputs are known — and builds the first benchmark that targets query-side insufficiency rather than response-side correctness.

Key Contributions

1. A formalization of legal query insufficiency as a distinct failure mode, with a taxonomy of eight canonical missing-element categories — jurisdiction, controlling text, procedural posture, parties and status, facts of harm, timing, consideration, and user goal — organized by three structural failure modes: switch, gating, and fatal prerequisite.
2. InsufficiencyBench itself, described as the first legal-domain benchmark targeting query-side insufficiency. It contains 202 items (58 base queries, 144 deficient variants) spanning six legal domains and 24 US jurisdictions, annotated by practising attorneys, with sentence-level annotation that enables controlled construction of deficient variants from complete base queries.
3. An empirical evaluation of ten frontier models showing that no model exceeds F2 = 0.46 on missing-element identification, that the median recall is 0.44, and that models either hedge indiscriminately (including on complete queries) or answer silently under fabricated presumptions.
4. A diagnostic decomposition splitting element-identification F2 into hedge rate and conditional identification quality, plus auxiliary metrics for explanation accuracy and safety rate, which localizes the failure to the decision to flag rather than to legal reasoning once a gap is flagged.

Main Findings

• No model reliably identifies missing elements. No model exceeds F2 = 0.46 on missing-element identification, and the median recall is 0.44. The median F2 is 0.363. GPT-5.2 leads with F2 = 0.455 and recall = 0.666; DeepSeek-V4-Pro trails with F2 = 0.278 and recall = 0.321. Even GPT-5.2's leading score corresponds to missing roughly one in three material elements.

• The main driver of low F2 is silence, not inaccuracy. Decomposing F2 into hedge rate and F2 restricted to hedged queries shows a gap of 0.053 for GPT-5.2 and 0.246 for DeepSeek-V4-Pro. DeepSeek-V4-Pro hedges on only 36.1% of deficient queries and produces substantive legal answers on the remaining 63.9% without acknowledging any gap. Even the leader, GPT-5.2 (hedge rate 86.8%), silently proceeds on 13.2% of deficient queries.

• Per-category blind spots are systematic. Procedural posture is catastrophically missed (mean recall = 0.09): recall is 0 for Mistral Large 3, DeepSeek-V4-Pro, and Qwen 3.5-397B, and reaches at most 0.231 for GPT-5.5 and Claude Opus 4.7. Parties and status elements have mean recall 0.258, with no model exceeding 0.405. By contrast, controlling text (mean recall 0.635) and facts of harm (mean recall 0.437) are more reliably detected. Models catch gaps that make the query read as incomplete and miss structurally required prerequisites that leave little textual trace.

• Top-F2 models are habitual hedgers; low-F2 models are systematically silent. On the 58 fully specified base queries, GPT-5.2 raises a missing-element claim on 72.4% of them and Claude-Opus-4.7 on 53.4%, even though annotators judged each query complete. DeepSeek-V4-Pro (22.4%) and Mistral Large 3 (27.6%) look well calibrated only because they rarely hedge at all — the same disposition that drives their identification failures.

• Low hedging tracks low safety. GPT-5.2 (safety 0.918) and GPT-5.5 (0.908) avoid fabrication on approximately 9 of 10 instances. DeepSeek-V4-Pro fabricates substantive conclusions on 30.2% of instances (safety 0.698) and Mistral Large 3 on 24.4% (0.756).

• Explanation quality is not the bottleneck. Explanation accuracy scores compress into a narrow 0.63–0.77 band (GPT-5.2 at 0.763), despite identification recall varying by more than a factor of two across the same models. Once a model commits to flagging a gap, it explains it about equally well.

• The judge choice does not change the result. Re-evaluating every response with Claude-Haiku-4.5 and GLM-5 reproduces the headline conclusion (no model exceeds F2 = 0.46 or recall = 0.67). GPT-5, the primary judge, is the most lenient of the three on identification metrics.

• No model achieves the ideal disposition. No model both identifies and qualifies responses to deficient queries while directly addressing complete ones.

Methodology in Plain English

The researchers started from complete legal scenarios. Two attorneys with over 20 years of combined experience authored 58 base queries grounded in concrete fact patterns, then annotated every legally material element at the sentence level — tagging each with a query-specific sub-tag (e.g., jurisdiction, employer_size, temporal_proximity), the source sentence, and one of eight canonical categories. They also marked which elements were required, meaning a final answer would be unsafe without them. Each base query also carries an attorney-written reference explanation of why each material element drives the analysis, withheld from the models.

To build test items, they removed sentences from the base queries and, where necessary, minimally revised the remainder to keep the query coherent. Each deficient variant carries a recorded ground-truth set of missing required elements. Because the answer key is the set of missing items rather than a gold legal answer, the benchmark is reference-free — a model gets credit for asking a targeted clarifying question, warning that the answer depends on a fact, or explicitly conditioning analysis on an assumed value.

Three metrics are computed per item by a fixed GPT-5 judge. Element-identification F2 is the primary metric, weighting recall more heavily than precision because a silently presumed jurisdiction is more dangerous than an overly cautious clarification request; false positives are captured as "additional claims." Explanation accuracy measures, over correctly identified elements only, whether the model's rationale matches the attorney-authored reference. Safety rate is 1 minus the fabrication rate, computed over the full ground-truth element set because fabricating on an unacknowledged gap still causes harm. On the fully specified base queries, ground truth is empty, so the reported calibration metric is instead the over-flag rate.

All ten models — GPT-5.2, GPT-5.5, Claude Opus 4.7, Claude Sonnet 4.6, Gemini 3.1 Pro, Gemini 3.1 Flash Lite, Qwen 3.5-397B, Mistral Large 3, DeepSeek-V4-Pro, and Kimi K2.6 — received the same deliberately minimal system prompt ("You are a legal assistant. Please answer the query.") in a single-turn setting, mimicking deployment where no instruction to probe for missing information is given.

Why This Matters

Impact on research. The paper reframes legal AI evaluation from answer correctness to answerability. It argues that mitigations targeting hallucination — better retrieval and grounding — and benchmarks that reward generic abstention do not address the underlying behavior, which is the disposition to answer without resolving material insufficiency. The decomposition into hedge rate and conditional identification quality gives future work a specific optimization target and separates two failure sources that aggregate scores conflate.

Real-world applications:

• Consumer-facing legal chatbots, where a user asking "Can my employer enforce this non-compete?" may receive a fluent answer under a silently assumed state, stating that state's law correctly while being wrong for the user's actual jurisdiction.
• Intake triage in law firms and legal aid, where the paper's point is that practitioners screen for parallel filters — such as a timely EEOC charge within 180 or 300 days, or an FMLA worksite threshold of 50 employees within 75 miles — that clients do not volunteer.
• Compliance and contract review tooling, where a missing lease term, statutory provision, or consideration fact changes which framework applies.
• Safety evaluation and model procurement, since the benchmark provides a measurable target for a baseline safety property of any legal AI system.

Industry relevance. The work comes from Thomson Reuters Foundational Research with Imperial College London, framing intake sufficiency as a deployment requirement rather than an academic curiosity. The finding that a single hedging disposition tracks performance on both deficient and complete queries — echoing the one-dimensional refusal/over-refusal axis reported elsewhere — suggests that calibration, not legal reasoning capability, is where current systems fall short.

Future Directions

1. Multi-turn evaluation. The authors note that their evaluation is single-turn, while premature legal closure may also arise over multiple turns of interaction.
2. Human validation of scoring. Judge agreement between LLM judges was moderate and was not validated against human scoring; the authors argue this is acceptable because the judge role is merely extractive, but the gap remains open.
3. Scale and coverage. The dataset is limited in size, spanning only six legal domains within US common-law contentious matters and 202 items; extension to more domains, jurisdictions, and items is a stated hope for future work.
4. Improving the flagging decision itself. Because explanation accuracy is uniformly high once a model flags a gap, the open question is how to train or steer models to detect structurally required prerequisites, such as procedural posture and parties and status, that leave little textual trace in a user's framing.

Target Audience

Researchers and engineers building or evaluating legal AI systems; benchmark designers interested in underspecification, clarification, and abstention beyond the legal domain; legal practitioners and policy audiences concerned with AI advice quality at intake; and model developers looking for a concrete optimization target around when a system should ask before answering. Readers without legal background can follow the benchmark construction and results, but interpreting the category-level findings benefits from familiarity with US statutory and doctrinal structure.

Authors’ abstract

Legal AI systems are increasingly used to answer legal questions, yet existing benchmarks assume queries arrive fully specified. In practice, users omit facts that materially determine the legal outcome. We introduce InsufficiencyBench, the first legal benchmark targeting query-side insufficiency: whether a model recognizes when a query lacks legally material information, identifies what is missing, and refrains from premature conclusions. We formalize a taxonomy of eight canonical missing-element categories across three structural failure modes---switch, gating, and fatal prerequisite--- and construct 202 benchmark items (58 base queries, 144 deficient variants) spanning six legal domains and 24 US jurisdictions and annotated by practising attorneys. Evaluating ten frontier models, we find that no model exceeds F2 = 0.46 on missing-element identification and that the median recall is 0.44. Models either hedge indiscriminately or answer silently under fabricated presumptions. No model both identifies and qualifies responses to deficient queries while directly addressing complete ones.