Beyond Majority Vote: Multi-Perspective Adjudication for Medical Hallucination Detection

AI summary

Overview

Research area: Natural Language Processing, specifically hallucination/factual-error annotation and benchmark construction for chatbot responses in medical domains.

Technical level: Intermediate. The methods (annotation workflows, agreement metrics such as Krippendorff's alpha and cross-replication reliability) require some familiarity with NLP evaluation practice, but the argument is stated in largely non-technical terms.

Scope: An annotation study and benchmark analysis of 541 chatbot responses to medically relevant questions (cystic fibrosis and pediatric infectious diseases), comparing first-pass annotation, an LLM-as-a-Judge, and two styles of adjudication.

What This Paper Is About

Most factual-error datasets are built with a single annotator reviewing a response once, which assumes that identifying a hallucination is a straightforward single-pass labeling task. In long-form chatbot answers, however, errors can be small phrases embedded in otherwise correct text, so a single pass may systematically miss them. The authors build MDHJudgments, a medical chatbot response dataset annotated from multiple perspectives, and use it to measure whether first-pass annotators, an LLM-as-a-Judge (LaJ), and human adjudicators find the same errors.

Key Contributions

1. The MDHJudgments dataset: 541 chatbot responses to medically relevant questions across cystic fibrosis and pediatric infectious diseases, containing 4,377 sections, annotated by first-pass annotators, an LLM-as-a-Judge (GPT-5), and two styles of adjudicator (medical expert and fact-checking), with disagreements preserved rather than collapsed into a single label.
2. A multi-perspective pipeline that treats error labeling as discovery followed by adjudication: disagreement between first-pass annotators is supplemented with disagreement between annotators and the LaJ, producing an adjudication pool of 834 sections.
3. Agreement analyses within and across labeling approaches: reporting Krippendorff's alpha, percent agreement, and cross-replication reliability (xRR) to show how much first-pass annotators, adjudicators, and the LaJ actually converge.
4. A benchmark-impact analysis and an external replication: showing how reference-set construction (FP:AG, FP+ME, FP+FC) changes measured detector performance, and applying the same LaJ-plus-adjudication approach to the MedExpert dataset, where 78% of LaJ-flagged sections implicitly labeled correct were adjudicated as containing an error.

Main Findings

• First-pass annotators have low recall against adjudicated labels. Individual first-pass annotators achieve precision of 87 and recall of 21 (F-1 of 33) against the FP+ME reference, and precision of 75 and recall of 20 (F-1 of 31) against the FP+FC reference. The authors conclude first-pass annotation alone yields an incomplete reference.

• First-pass annotators miss errors that adjudicators later validate, and the errors are sometimes major. 35% of initial medical-expert adjudicator judgments indicate a major error after the first step of expert adjudication.

• The LLM-as-a-Judge improves candidate coverage but is not sufficient alone. The LaJ misses factual errors that annotators catch; 8 sections in the adjudication pool were found by a first-pass annotator but not by the LaJ. With the reference set, the LaJ reaches precision of 90 and recall of 71 against FP+ME, and precision of 90 and recall of 80 against FP+FC.

• Without adjudication, the LaJ looks unusable. Against the FP:AG reference (only sections where all first-pass annotators agree), the LaJ has precision of 2 and recall of 83. Only 6 sections are unanimously labeled as containing an error, and the LaJ misses only 1 of those 6.

• Adjudicators disagree with each other too. Between medical-expert and fact-checking adjudicators, percent agreement is 73.0% and xRR is 0.18. Agreement within medical-expert adjudication shows percent agreement of 81% and alpha of 0.05; within fact-checking adjudication, 67% and alpha of 0.15.

• First-pass annotators disagree with adjudicators in a consistent direction. Against medical experts, 70.3% of first-pass/medical-expert pairs are disagreements where the annotator says "true" (error) and the expert says "false"—i.e., the reverse pattern. The xRR between first-pass annotators and both adjudication styles is 0.01.

• Agreement across the whole process is low even when raw percent agreement looks moderate. Within first-pass annotation, micro alpha is 0.15 with 85% agreement; macro alpha is 0.15 with 88% agreement. The authors attribute this partly to alpha's treatment of common versus rare classes and to high class imbalance.

• Judgment is required even at the expert level. In 12% of judgments, medical-expert adjudicators indicated they needed more research to make a decision, suggesting has_error is not always a simple recognition decision.

• Reviewer comments influence adjudication. After reviewing comments from first-pass annotators and the LaJ, expert adjudicators assigned has_error judgments both where the expert had acknowledged needing more information and where the expert initially did not see an error. Severity labels for this subset were roughly balanced between major and minor. Most has_error sections had comments categorized as "other medical information," but citation errors and missing information were well represented.

• Response-level accuracy and section-level errors are only partly related. The correlation between average response-level accuracy (1–3 scale, 3 most accurate) and the percentage of sections labeled has_error is -0.39. Among responses with the highest possible average response-level accuracy, most contain at least one section labeled has_error, and several have more than 50% of their sections labeled has_error. Among 1–2 section responses where all sections are labeled has_error, the most common response-level rating is "fully accurate."

• The pattern replicates on an existing benchmark. On a subset of 124 MedExpert sections where the LaJ disagreed with the MedExpert annotation label, 78% of sections found by the LaJ but implicitly labeled as correct in MedExpert were adjudicated as containing an error.

• Model choice matters. Novel LaJ outputs that did not contribute to the adjudication pool, including runs using Gemini instead of GPT-5, show both precision and recall dropping relative to the original LaJ, though their recall still far exceeds that of an individual first-pass annotator.

Methodology in Plain English

The authors assembled a set of medically relevant questions about cystic fibrosis and pediatric infectious diseases and generated chatbot responses to them. Responses came from Gemma3-12b, Qwen3-32b (with reasoning), and GPT-4.1, across different prompt augmentations (none, "be brief," "provide medical evidence," and an intentional hallucination-injection prompt). The result is 541 question-response pairs: 169 with no augmentation, 186 "be brief," 46 "medical evidence," and 140 with injected hallucination.

Responses were split into sections by rule using newlines; responses contained between 1 and 27 sections, with a mean of 8.1, giving 4,377 sections. First-pass annotators—a mix of medical professionals, medical trainees, and AI researchers, with the latter two groups asked to find at least one source—first gave a whole-response gestalt accuracy judgment on a three-point scale, then worked section by section, deciding whether a section contained a claim, then whether it contained an error. For analysis, the fine-grained accuracy and attribute judgments were collapsed into a single binary has_error decision.

Rather than trying to annotate everything, the team built an adjudication pool out of cases where some signal suggested an error: 402 sections where first-pass annotators disagreed, 389 sections where an annotator disagreed with the LaJ, 35 injected-hallucination sections missed by everyone, and 8 found by a first-pass annotator but not by the LaJ—834 sections total, 95% of which came from disagreement. Adjudicators were recruited through Prolific using its certification process, and two kinds were used: medical experts, who reviewed sections, comments, and explicitly revised their judgments, and fact checkers, who located published evidence to ground their decisions.

The authors then measured agreement within each labeling approach (Krippendorff's alpha, percent agreement) and across approaches (percent agreement and cross-replication reliability, which extends Cohen's kappa to comparisons between different groups). Finally, they built three reference label sets—FP:AG (only unanimous first-pass sections), FP+ME (adding medical-expert adjudication), and FP+FC (adding fact-checking adjudication)—and scored first-pass annotators and the LaJ against each, reporting precision, recall, and F-1 with 95% confidence intervals computed from 10,000 bootstrap samples.

Why This Matters

Impact on research. The paper argues that singly annotated, single-pass hallucination benchmarks may achieve scale at the cost of undercounting factual errors, and that adjudication over multiple candidate sources improves coverage but does not eliminate the need for judgment and expertise. It also shows that benchmark scores are sensitive to how the reference set is constructed—the same LaJ goes from precision 2 and F-1 4 against FP:AG to precision 90 and F-1 80 against FP+ME. The authors frame their approach as analogous to TREC-style pooled assessment, where detector outputs determine what gets judged, and caution that scores should be read as relative to the constructed reference rather than as unbiased estimates over an exhaustively judged population.

Real-world applications:

• Safety evaluation of patient-facing medical chatbots, where missed factual errors can affect clinical decisions.
• Designing annotation protocols and quality-control processes for clinical NLP datasets.
• Interpreting existing hallucination benchmarks when deciding whether a detection system is deployment-ready as a guardrail.
• Auditing prior datasets for under-counted errors, as demonstrated on MedExpert.

Industry relevance. Any organization building or procuring factual-error detectors for medical or other high-stakes text needs to know that reported benchmark performance depends heavily on whether adjudication was used, which adjudicator population was used, and whether the evaluation used pooled candidate discovery. The finding that holistic response-level ratings can obscure section-level errors also matters for product teams that rely on human reviewers giving an overall thumbs-up or thumbs-down.

Future Directions

• Separating mistakes from true disagreement. The authors state their paradigm does not distinguish disagreement caused by simple errors (wrong button, fatigue) from genuine disagreement about factual accuracy, and suggest the released data could support this with an additional level of adjudication.
• Measuring adjudicator agreement on random samples. Because adjudication relies on disagreement, the current dataset only reveals adjudicator agreement where conflicting signals already existed; the authors note they do not know agreement on a random sample.
• Varying the LLM-as-a-Judge. The study uses a single prompt and a single underlying model; the authors expect different results if these change, and Appendix C.3 takes a first step by presenting results for Gemini.
• Reconciling broader versus narrower definitions of factual accuracy. Adjudicators surfaced citation errors and missing information that fall outside the original first-pass instructions, raising the question of whether tighter guidelines would improve consistency or produce a definition of accuracy that is too narrow.

Target Audience

NLP researchers and practitioners who build or consume hallucination and factuality benchmarks, especially in medical and other high-stakes domains; dataset creators designing annotation and adjudication protocols; and teams evaluating factual-error detectors for deployment as guardrails. Readers interested in annotation agreement methodology, LLM-as-a-Judge evaluation, or the limits of single-pass labeling will also find the analysis directly relevant.

Authors’ abstract

Understanding the frequency of factual errors in chatbot-generated text and evaluating systems that detect these errors is critical for determining chatbot safety. Yet factual-error detection is often treated as a single-pass, single-annotator labeling problem. In long-form chatbot responses, factual errors can be subtle and embedded within mostly correct text. We develop a multi-perspective annotation study of medically relevant chatbot responses, combining first-pass annotation, LLM-as-a-Judge (LaJ) candidate discovery, and two forms of adjudication: medical-expert and evidence-based fact-checking. First-pass annotators frequently miss factual errors later validated by adjudicators. LaJ improves candidate discovery, but is insufficient on its own: It misses factual errors that annotators catch. We also find disagreement among adjudicators, suggesting that adjudication over multiple candidate sources can improve benchmark completeness, but does not eliminate the need to apply judgment and expertise. Applied to an existing benchmark, this technique reveals a similar pattern of missing annotations. Together, these results suggest that in the settings examined here, single-pass hallucination benchmarks may achieve scale at the cost of undercounting factual errors. Multi-pass adjudication can improve coverage, but inferences drawn from the benchmarks are still sensitive to the judgment, expertise, and evidence used to determine error presence.