From Detection to Diagnosis: Advancing Hallucination Analysis with Automated Data Synthesis

AI summary

Overview

• Research area: Natural Language Processing; specifically hallucination analysis and mitigation in Large Language Models (LLMs), with a focus on faithfulness hallucinations (outputs that contradict or lack support from a provided source document).
• Technical level: Intermediate. The paper assumes familiarity with LLM post-training concepts such as Chain-of-Thought reasoning, GRPO (Group Relative Policy Optimization), and reward shaping, plus standard hallucination detection benchmarks.
• Scope: The paper proposes a new "Hallucination Diagnosis" task, builds an automated pipeline (HDG) that synthesizes diagnostic training data from raw corpora, and trains a 4-billion-parameter diagnosis model (HDM-4B-RL) using reinforcement learning.

What This Paper Is About

Most existing work on LLM hallucination treats the problem as binary detection, producing a single label such as "Entailment" or "Contradiction" without saying where the error is, why it is wrong, or how to fix it. The authors argue this coarse feedback is not actionable for automated correction or human revision, so they define a richer "Hallucination Diagnosis Task" that requires detection, error localization, causal explanation, and content correction. Their goal is to show that such a diagnostic model can be trained automatically, without hand-labeled data, and still remain small enough to be practical.

Key Contributions

1. A new task definition. The paper formally proposes the Hallucination Diagnosis Task, defined by four core capabilities across the model's output: detection (is the content consistent with the source), localization (which segments are hallucinatory), explainability (why the content is inconsistent), and mitigation (a proposed correction aligned with the source).

2. An automated data synthesis pipeline. The Hallucination Diagnosis Generator (HDG) builds diagnostic training data directly from raw pre-training corpora rather than adapting existing NLI or QA datasets. It uses four stages: task-oriented seed sample generation, multi-dimensional sample augmentation, quality verification, and metadata enrichment.

3. A trained diagnosis model. HDM-4B-RL, a 4-billion-parameter model built on Qwen3-4B and trained with GRPO under a rule-based reward function combining structural, accuracy, and localization signals. The authors chose RL over supervised fine-tuning out of concern that SFT could disrupt the base model's hybrid reasoning ability.

4. A synthesized dataset with diagnostic annotations. The pipeline produced 18,453 samples (5,999 hallucination / 12,454 non-hallucination) spanning summarization, logical reasoning, and mathematics, each carrying sentence-level hallucination annotations, task difficulty labels, and reasoning traces. The authors also constructed a dedicated 285-sample diagnosis benchmark from SummEval.

Main Findings

• Detection performance beats specialized detectors. HDM-4B-RL achieves an average macro F1 of 79.65 across all detection benchmarks, compared to 77.25 for Lynx-8B, the previous state-of-the-art specialized detector. The paper reports this as a 2.4% higher average F1 despite HDM-4B-RL being half the size (4B vs. 8B).

• Strongest on FinanceBench and DROP. The model excels on FinanceBench (F1 84.54) and DROP (F1 74.39) among specialized detection models.

• Competitive with general-purpose LLMs but not the top. HDM-4B-RL (79.65 F1) surpasses GPT-4.1 (75.37 F1) and approaches reasoning-enabled Qwen3-32B (81.70 F1), but remains below o4-mini (82.62 F1). The 32B model requires approximately 2.2× the runtime of HDM-4B-RL.

• Reasoning mode gives large gains. Enabling reasoning mode raised Qwen3-4B's average F1 from 64.92 to 76.74, and raised HDM-4B-RL's from 70.13 (non-reasoning) to 79.65 (reasoning). Reasoning mode requires roughly 1.8× the runtime of non-reasoning mode.

• RL fine-tuning increased recall substantially. In reasoning mode, HDM-4B-RL improved F1 by +2.91, driven by Recall +4.62 and Precision +2.50. In non-reasoning mode, the F1 gain was larger (+5.21), driven almost entirely by Recall +5.97 while Precision stayed flat (-0.26). The authors interpret this as improved fundamental pattern recognition, not just reliance on explicit reasoning.

• Diagnosis performance is competitive but not uniformly best. On the 285-sample diagnosis benchmark, HDM-4B-RL scored Detection ACC 92.28, Localization Hit Rate 58.65, Span Validity 48.49, and Mitigation AlignScore 69.16 (versus an "Original Result" baseline of 59.77). The multi-step pipeline with Qwen3-32B scored higher on localization (76.97 HR) and mitigation (79.55 AS), but at a latency cost of 4.9× versus HDM's 1×, while 32B end-to-end costs 1.5×.

• Per-benchmark detection scores. HDM-4B-RL (reasoning) scored F1 83.96 on HaluEval, 68.52 on RAGTruth, 84.54 on FinanceBench, 74.39 on DROP, 92.50 on CovidQA, 77.79 on PubMedQA, and 75.88 on SummEval.

Methodology in Plain English

The authors start from the premise that detecting a hallucination is only the first step of what should be a full diagnosis, analogous to how a doctor must not only spot a problem but locate it, explain it, and propose a treatment. To train a model to do this, they need data that contains all of those signals — data that largely does not exist in public benchmarks.

So they build one. Working from a Wikipedia dump (20231101.en), they randomly sample 4,500 documents and have an LLM write tasks over them — summaries, logical reasoning problems, and math problems — along with correct answers. For reasoning and math, the model is prompted to produce a Chain-of-Thought structure, and samples are kept in two formats: direct answer and full reasoning chain.

Then they deliberately corrupt or perturb these correct samples. They add distracting information to the context or delete information the answer depends on. They replace key entities in summaries (for example turning a "red rose" into a "pink rose") or perturb a single step inside a correct reasoning chain. They also swap precise figures for vague ones (for example "3,900 people" becoming "nearly 4,000 people"), though only for summarization, since that kind of ambiguity would be unacceptable in math or logic.

Because automatically corrupted data can be noisy, a verification stage follows: an LLM scores each query-answer pair for clarity, fluency, and coherence (deliberately ignoring correctness), while an ensemble of classifiers and LLM-as-a-judge models — Qwen3-32B, GPT-4.1, Lynx, and MiniCheck-7B — vote on whether the sample actually contains a hallucination. Surviving samples are then enriched with metadata: which specific sentences are hallucinated, how difficult the task is (based on the length of the reasoning chain needed), and an explanatory reasoning trace.

For training, the authors avoid supervised fine-tuning because they worry it would damage the base model's built-in reasoning. Instead they apply GRPO, which compares the relative quality of a group of generated responses rather than scoring each one absolutely. The reward function has three parts: a structured output reward that requires the model to emit valid JSON with four fields (conclusion, diagnosis, hallucinations, corrected_answer), a detection accuracy reward that rewards matching the ground-truth pass/fail label, and a localization reward that scores how well the model's predicted hallucinated sentences overlap the ground-truth sentences, penalizing imprecise boundaries by length ratio. These are combined with weights of 1.0, 0.5, and 0.5.

Training used Qwen3-4B as the base model, the ms-swift and Hugging Face Transformers libraries, and 2× NVIDIA A100 80GB GPUs, with the AdamW optimizer at a learning rate of 1×10⁻⁶, weight decay 0.01, batch size 64, and 2,306 total steps.

Why This Matters

The paper reframes hallucination handling from a classification problem into a diagnostic one, which matters because a bare label rarely helps anyone fix the underlying issue. It also demonstrates that high-quality diagnostic training data can be manufactured automatically rather than collected by hand, and that a 4B model can reach performance levels previously associated with much larger models on this specific task.

Real-world applications:

• Long-document summarization: flagging exactly which sentences in a generated summary are unfaithful to the source, and rewriting them.
• Knowledge-based QA in specialized domains: the paper evaluates on finance (FinanceBench), medicine (PubMedQA, CovidQA), and general knowledge (HaluEval, DROP), all areas where an unsupported claim can mislead a user.
• Retrieval-augmented generation (RAG) systems: the detection benchmarks include RAGTruth, which is directly aimed at hallucination in retrieval-grounded generation.
• Human-in-the-loop editing: localization plus natural-language explanation gives a writer or reviewer a concrete starting point for revision rather than a pass/fail verdict.

Industry relevance comes from the efficiency angle: HDM-4B-RL is a 4B model that the authors report outperforms the 8B Lynx-8B detector on average F1 and runs at roughly a third of the latency cost of a 32B end-to-end model (1× vs. 1.5×, and far below the 4.9× cost of a 32B pipeline). For teams deploying hallucination checks at scale, that cost difference is a practical consideration.

Future Directions

• Expanding the source corpus. The HDG pipeline currently relies solely on Wikipedia. The authors propose extending it to specialized domains such as scientific literature and legal documents, and scaling the dataset size.
• Testing at larger model scales. Experiments were confined to a 4B-parameter model due to computational constraints. Whether the approach holds or improves at larger scale is explicitly listed as future work.
• Closing the localization and mitigation gap. On the diagnosis benchmark, HDM-4B-RL's single-prompt localization Hit Rate (58.65) trails the multi-step pipeline baseline (76.97 with Qwen3-32B), raising the question of how to get pipeline-level fine-grained accuracy inside a single efficient pass.
• Measuring explanation quality. The diagnostic report includes a "diagnosis" explanation field, but the evaluation metrics reported cover detection, localization, and mitigation only. How to evaluate causal explanation quality remains an open question that the paper does not resolve.

Target Audience

This paper is most useful to researchers and engineers working on LLM reliability, factuality, and evaluation, particularly those building detection or grounding systems and looking for richer annotation than binary labels. It will also interest practitioners deploying RAG or summarization systems who need actionable, sentence-level error reports rather than a single score. Readers working on reinforcement learning for LLM post-training will find the reward design (structured output, accuracy, and localization components with weights 1.0/0.5/0.5) directly applicable. Those without background in hallucination benchmarks such as HaluEval, RAGTruth, or SummEval will need some orientation, since the paper assumes familiarity with them and does not define them in depth.

One note on internal consistency: the appendix refers to the training template and model as "HDR-4B-RL" in places where the main text uses "HDM-4B-RL." The evaluation that produced the reported numbers used macro F1 for detection, and the full per-benchmark precision, recall, F1, and accuracy breakdowns appear in the appendix tables rather than the main body.

Authors’ abstract

Hallucinations in Large Language Models (LLMs), defined as the generation of content inconsistent with facts or context, represent a core obstacle to their reliable deployment in critical domains. Current research primarily focuses on binary "detection" approaches that, while capable of identifying hallucinations, fail to provide interpretable and actionable feedback for model improvement, thus limiting practical utility. To address this limitation, a new research paradigm is proposed, shifting from "detection" to "diagnosis". The Hallucination Diagnosis Task is introduced, a task which requires models to not only detect hallucinations, but also perform error localization, causal explanation, and content correction. We develop the Hallucination Diagnosis Generator (HDG), an automated pipeline that systematically generates high-quality training samples with rich diagnostic metadata from raw corpora through multi-dimensional augmentation strategies including controlled fact fabrication and reasoning chain perturbation. Using HDG-generated data, we train HDM-4B-RL, a 4-billion-parameter hallucination diagnosis model, employing Group Relative Policy Optimization (GRPO) with a comprehensive reward function incorporating structural, accuracy, and localization signals. Experimental results demonstrate that our model surpasses previous state-of-the-art detection models on the HaluEval benchmark while achieving comparable performance to advanced general-purpose models. In comprehensive diagnosis tasks, HDM-4B-RL matches the capabilities of larger general models while maintaining a smaller size. This work validates the feasibility and value of hallucination diagnosis, providing an effective methodology for building more trustworthy and reliable generative AI systems.