GANDR: Claim Auditing for Verifiable Legal Answer Generation

AI summary

Overview

Research area: Natural Language Processing, specifically retrieval-augmented generation (RAG) for high-stakes legal question answering, with a focus on citation faithfulness and per-claim verification.

Technical level: Intermediate. The paper is readable without deep legal-domain expertise, but assumes familiarity with RAG pipelines, LLM-as-judge evaluation, multi-agent orchestration, and metrics like precision/recall and McNemar tests.

Scope: The paper introduces GANDR, a two-agent legal answer generator that emits a per-claim audit trace alongside every answer, and pairs it with a strict grounding metric that fails any answer containing a citation that does not resolve to a retrieved passage.

What This Paper Is About

Legal AI answers are only useful if a lawyer can verify each individual claim against the source the system cites, yet today's evaluation pipelines score an answer as a whole, so a correct conclusion sitting on fabricated or loosely matched citations can still score well. The authors argue that fixing this requires building verification into the generation architecture rather than applying a stricter post-hoc score, and they construct a system and an evaluation protocol that do exactly that. GANDR separates answer drafting from claim auditing and gates its commit decision on a deterministic structural check rather than on the auditor's own judgment.

Key Contributions

1. A two-agent system that emits per-claim verification as a generation artifact. A Drafter writes in a structured CREAC format (Conclusion, Rule, Explanation, Application, Conclusion) with inline {cite: rule_id} markers, while a Critic running in a disjoint LLM context audits every assertion against its cited source and writes verdicts to an audit trace. No baseline emits this support signal in its standard configuration.

2. A deterministic, uniform grounding evaluation protocol. Instead of aggregate faithfulness scores, the authors define a strict correctness criterion requiring the right disposition and that every emitted citation resolves to a passage the retriever actually returned. The metric is computed once, identically for all systems, and extended to Stable & Correct @ k = 5 across repeated runs.

3. Evidence that the architectural choices, not scale or retrieval luck, drive the lead. On a 185-item benchmark where six systems share one backbone, one BM25 retrieval surface, and one citation instruction, GANDR leads by 11.3 strict points over the strongest baseline and stays positive on three additional backbones.

4. An ablation showing the commit rule is load-bearing. Reverting the protocol-anchored commit gate to the original 0.95 atomic-verification threshold drops strict accuracy by 22.7 points.

Main Findings

• GANDR leads on every primary metric. 70.8% strict accuracy versus 59.5% for the strongest baseline (B2, a single LLM with the same CREAC schema plus a self-check rubric), a gap of 11.3 points at p < 0.01 via paired McNemar. Lenient accuracy is 71.9%; the strict–lenient gap is only 1.1 points, meaning almost every correct answer is also fully grounded.

• Structure absorbs citation drift; extra agents without auditing amplify it. Zero-shot and advanced-RAG baselines score competitively on lenient accuracy (57.3% and 60.5%) but collapse on strict (38.4% and 43.2%). The two multi-agent baselines — a CrewAI researcher/writer crew and a LangGraph self-correction graph — drop to 21.6% strict, with each additional LLM call adding another chance to emit an unresolvable citation.

• The lead comes from the Drafter configuration and the commit anchor, not from rewriting. 98.6% of GANDR runs commit on Round 1, and no answer is ever rewritten into a passing one. The rewrite loop's only realized function is to fail closed: on the 13 runs that exhaust the budget, the system returns passed=False with the full trace rather than silently committing.

• The commit rule accounts for a large share of the gain. Reverting the gate to the 0.95 atomic-verification threshold drops strict accuracy from 70.8% to 48.1% (minus 22.7 points), because rewrites fire on nearly every item and degrade quality. That variant returns passed=False on 92.7% of runs.

• The lead transfers across backbones but narrows with model strength. Strict gains over B2 are +6.5 points on GLM-4.7-Flash, +3.7 on gpt-5.4-mini, and +3.2 on gpt-5.4. Stronger backbones comply with the citation contract unprompted more often, though the per-claim trace and fail-closed exit remain architectural properties no backbone supplies alone. One stability reversal on GLM-4.7-Flash (GANDR trails B2 on S&C@5, 0.297 vs 0.411) is flagged as unexplained.

• The audit trace works as a binary flag but not as a fine-grained label. Against two law-trained annotators on 148 blinded assertions, the tool detects claims needing attention at precision 0.80, recall 0.88, F1 0.84. Its four-way verdict labels agree only weakly with humans (κ = 0.275), with MISCITED the least reliable class — the labels are advisory, not precise.

• The correctness judge is well-calibrated. Claude Opus 4.7, used from a separate model lineage to mitigate self-preference bias, tracks the human annotators at Pearson r = 0.846 and quadratic-weighted κ = 0.845.

• Metered cost is modest relative to the call count. Because Round-1 anchoring commits most items in one round, the real cost is 3.0–3.3× the B2 baseline rather than the roughly 10× that the raw number of LLM calls would imply.

Methodology in Plain English

The authors take a 185-item benchmark stratified across eight legal domains, drawn from LegalBench and LegalBench-RAG, and route every system through the same BM25 index of 13,090 passages, handing each one the same top-8 retrieved passages. This is deliberate: by fixing the backbone, the retrieval surface, and the exact citation instruction across all six systems, any measured difference has to come from architecture.

The systems form a ladder. B1 is plain zero-shot. B2 is a single LLM using the same CREAC prompt as GANDR plus a six-dimension self-check rubric — the head-to-head comparator. B3 improves retrieval with dense retrieval and cross-encoder reranking while leaving generation alone. B4 and B5 are off-the-shelf multi-agent frameworks (CrewAI and LangGraph) running their default recipes.

GANDR itself is a loop. A Drafter writes a CREAC memo. Three tools then run on the draft: a regex structural check (protocol_check), an atomic verifier that splits the explanation and application blocks into (claim, rule_id, span) triples and labels each SUPPORTED/PARTIAL/UNSUPPORTED/MISCITED, and a presentation-quality rubric. A Critic then sees only what a human verifier would see — the question, the passages, and the final draft, never the Drafter's reasoning — and emits a JSON verdict. Crucially, the orchestrator ignores the Critic's verdict when deciding whether to commit; it reads the structural check directly.

Scoring is where the paper diverges most from convention. A separate LLM judge (Claude Opus 4.7) rates correctness 1–5. The strict metric requires both a passing judge score and that every extracted citation resolves to a header in the retrieved top-8, so a single fabricated citation fails the item. For stability, the judge performs all ten pairwise comparisons across five seeds and checks whether holdings are equivalent, strict-correct, and strictly grounded on every run.

The authors then ablate the one component they can cleanly isolate — the commit anchor — and test backbone transfer by swapping only the generator. They also validate the audit tool against two law-trained annotators on a stratified, blinded sample of 148 assertions.

Why This Matters

Impact on research. The paper reframes grounding evaluation from a retrospective score into an architectural property. Its most transferable idea is that the commit rule should be anchored to a deterministic structural check rather than the auditor's own verdict — this breaks the circularity where a model is asked to ratify the reasoning it produced. The finding that default multi-agent configurations worsen strict grounding while improving surface fluency is a sharp counterpoint to the assumption that more agents mean more reliability.

Real-world applications:

• Legal research assistants. A per-claim audit trace lets an attorney scan which assertions are supported rather than re-reading the whole memo against the source, which is exactly the verification workflow the paper identifies as the real post-answer task.

• Regulatory and compliance drafting. The fail-closed exit — returning an explicit failure with a full trace rather than a confident-sounding answer — maps directly onto domains where a fabricated citation carries professional or legal liability.

• Clinical and scientific literature synthesis. The CREAC schema is legal-specific, but the pattern of a structured claim-by-claim contract plus a separate-context auditor generalizes to any domain where each assertion must trace to a retrieved source.

• Audit tooling for existing RAG systems. The atomic cite_verify component can be deployed against a pipeline's outputs without retraining, since it only needs the draft, the question, and the retrieved passages.

Industry relevance. The measured cost of only 3.0–3.3× a single-LLM baseline is what makes the approach practically deployable, since naive multi-agent pipelines are often many times more expensive for no grounding benefit. The backbone-transfer results also give product teams an honest signal: on frontier models the architectural gain narrows to a few points, so the case for adoption rests on the audit trace and the fail-closed guarantee rather than on raw accuracy alone.

Future Directions

1. Fixing the verdict taxonomy. The four-way labels are weakly calibrated (κ = 0.275), and MISCITED is the weakest class. The authors name a reproducibly applicable verdict taxonomy as the clearest next step.

2. Isolating the Critic and the decomposition. Because the anchor commits without consulting the Critic on ~98% of items, disabling either leaves the committed answer unchanged, so neither can currently be credited with an accuracy effect. A design in which they can alter the committed text is needed.

3. Breaking weight-level self-preference. Drafter and Critic share backbone weights, so the information barrier blocks the Drafter's chain of thought but not weight-level bias. A different-lineage Critic, and injecting the Drafter's reasoning to test the barrier's value, are both left open.

4. Extending beyond supplied authority. The benchmark indexes each item's own context and is classification- and NLI-weighted, so GANDR is validated as a citation-discipline layer over provided material, not as an open-corpus legal research tool. Multi-source memos and harder retrieval remain untested.

5. Strengthening human validation. Annotators are project collaborators rather than independent third parties, and the same pair supplies both the audit validation and the judge calibration, so correlated expectation bias cannot be ruled out. An independent-annotator subset is proposed.

Target Audience

This paper benefits most researchers and engineers working on grounded generation and faithfulness evaluation, particularly those in retrieval-augmented NLP who need metrics that cannot be satisfied by a correct-sounding answer with bad citations. It is also relevant to applied teams building legal, compliance, or medical AI assistants who need an auditable failure mode rather than a confidence score, and to multi-agent systems researchers interested in evidence that default orchestration patterns can degrade grounding rather than improve it. Readers should be comfortable with RAG terminology and standard evaluation methodology; no legal training is required.

Authors’ abstract

In high-stakes domains such as legal practice, a language-model answer is only useful to the extent that a reader can verify each claim against the source the system cites. Current grounded-generation pipelines score the answer as a whole, so a correct conclusion can rest on fabricated or loosely matched citations and still score well. Closing this gap requires both a system built for per-claim verification and an evaluation that measures it. We introduce GANDR (Grounded ANswer DRafter), a two-agent system in which a Drafter writes an answer in a structured legal-reasoning format and a separate Critic, with the same view as a human verifier, audits each claim against its cited source and emits a per-claim audit trace on every round. We pair it with a strict correctness criterion requiring every citation to resolve to a passage the retriever returned. On a 185-item legal benchmark where all six systems share one backbone, one retrieval surface, and one citation instruction, GANDR ranks first on every primary metric, reaching 70.8% strict accuracy and leading the strongest baseline by 11.3 points (p&lt;0.01). Reverting the protocol-anchored commit rule lowers strict accuracy by 22.7 points, and the strict lead stays positive on three further backbones, at +3.2 to +6.5 points. This lead traces to the Drafter configuration and the protocol-anchored commit, not to rewriting. Against two law-trained annotators the audit flags under-supported claims at F1 0.84 as a binary detector, while its four-way verdict labels agree only weakly and are advisory. Code is available upon request.