HalluTruthQA: A Fine-Grained Benchmark for Hallucination Detection, Localization, and Explanation in Arabic Question Answering

AI summary

Overview

• Research area: Natural Language Processing / Arabic question answering; large language model factual reliability and hallucination evaluation.
• Technical level: Intermediate. The paper is a benchmark-and-evaluation study, so it is readable without deep modeling background, but it uses metrics such as Macro-F1, partial-credit span F1 (F1-Sp), and LO-Score.
• Scope: The paper introduces HalluTruthQA, a 2,400-example expert-curated Arabic QA benchmark that jointly supports hallucination detection, character-level span localization, explanation evaluation, and multiple-choice factual verification, and reports zero-shot results for four open-source LLMs plus a FANAR-9B self-detection reference setting.

What This Paper Is About

Most existing hallucination benchmarks for Arabic QA provide only response-level labels saying whether an answer is correct or hallucinated. That tells you an answer is wrong but not which part is wrong, why it is wrong, or whether the model can pick out the correct fact. This paper builds HalluTruthQA, a benchmark of 2,400 expert-curated Arabic QA examples across Islamic knowledge, history, science, and geography, where each example includes a verified reference answer, a binary hallucination label, six candidate answers, and, for hallucinated responses, character-level erroneous spans with human-written explanations.

Key Contributions

1. HalluTruthQA benchmark: 2,400 expert-curated Arabic QA examples across four knowledge-intensive domains: Islamic knowledge, history, science, and geography.
2. Multi-layer annotation scheme: response-level hallucination labels, character-level erroneous spans, human-written explanations, a two-level (macro and micro) hallucination taxonomy, and six candidate answers per question for factual verification.
3. Zero-shot evaluation of four open-source LLMs (ALLaM-7B, Falcon-H1R-7B, Qwen3-32B, SILMA) across four tasks: hallucination detection, span-level localization, factual verification, and explanation evaluation, plus FANAR-9B reported separately as a self-detection reference setting.
4. Public release of code, dataset, prompts, and evaluation scripts at https://gitlab.com/nlpresearcher/HalluTruthQA.

Main Findings

• Detection is a distinct skill from localization: Among independent models, ALLaM-7B achieves the highest global Macro-F1 for hallucination detection at 0.880, followed by Falcon-H1R-7B (0.859), Qwen3-32B (0.849), and SILMA (0.838). FANAR-9B reaches 0.912 in the self-detection reference setting.
• Span localization is much harder: Qwen3-32B has the best global F1-Sp at 0.516 (precision 0.656, recall 0.426). FANAR-9B follows at 0.508 with the highest precision (0.727) but lower recall (0.390); Falcon-H1R-7B scores 0.486, ALLaM-7B 0.449, and SILMA 0.296 (recall 0.188).
• Better detection does not mean better verification: For multiple-choice factual verification, Falcon-H1R-7B achieves the best global LO-Score among independent models at 0.852, followed by Qwen3-32B (0.844) and ALLaM-7B (0.842). Falcon-H1R-7B selects the wrong option in only 3.6% of true-positive cases.
• Explanation quality favors a different model: ALLaM-7B outperforms all evaluated systems, including FANAR-9B, on explanation quality, with Error Identification 0.673, Factual Correction 0.614, and the best Final Score of 0.644.
• No single model wins everywhere: Best scores across tasks are 0.880 Macro-F1 (detection), 0.516 F1-Sp (localization), 0.852 LO-Score (verification), and 0.644 final score (explanation).
• SILMA is conservative: Its hallucination F1 is lower than its non-hallucination F1, indicating it misses more hallucinated responses and has the weakest localization (F1-Sp 0.296).
• Hallucination types skew toward factual contradiction: Across all hallucinated spans, Factual Contradiction is dominant at 59.3%, Context Inconsistency 27.0%, Factual Fabrication 11.3%, Nonsensical/Irrelevant Response 1.4%, and Logical Inconsistency 1.0%.
• Domains behave differently: Factual Contradiction dominates History (80.4%), Science (93.3%), and Geography (58.6%), but in Islamic Knowledge, Context Inconsistency accounts for 61.6%. Geography shows the highest Factual Fabrication rate at 28.2%.
• Domain-specific winners vary: In Islamic Knowledge, Qwen3-32B has the best F1-Sp (0.489). In Geography, Qwen3-32B leads on Macro-F1, F1-Sp (0.498), LO-Score, and explanation quality. In Science, SILMA has the strongest detection among independent models (Macro-F1 0.954) and ALLaM-7B the best span localization (F1-Sp 0.475) and best explanation Final Score (0.742). In History, FANAR-9B has the best F1-Sp (0.664) and ALLaM-7B the best explanation Final Score (0.702).
• Error families: Analysis of 790 unique instances with at least one wrong model prediction (1,556 model-level errors total) yields four families: Temporal, Numeric, and Unit Normalization (39.0%); Source Attribution and Evidence Verification (22.2%); Correct Short Answer with Hallucinated Support (17.1%); and Fully Incorrect Answer (11.3%). Together these cover 89.6% of model-level errors.
• Shared failures are mostly false positives: 39 examples triggered identical failure across all five evaluated systems, of which 79.5% are false positives and 20.5% are missed true hallucinations, concentrated in History (18), Geography (10), Islamic (8), and Science (3).
• Human annotation is reliable: Inter-annotator agreement before final adjudication was Cohen's kappa = 0.93 for binary labels, 0.87 for macro-types, 0.81 for micro-types, 97.9% agreement on the multiple-choice answer key, and span-level agreement of 0.89 character-F1 and 0.84 IoU.

Methodology in Plain English

The team collected Arabic questions in four knowledge-intensive domains and generated one answer per question using a single Arabic-centric model, Fanar-1-9B-Instruct (FANAR-9B), so that all answers come from the same generator. This controls variation in style and error patterns. Generation used a maximum of 1024 new tokens, temperature 0.0, top-p = 1.0, batch size 4, and bfloat16 precision on a 48 GB GPU.

Four experts, one per domain, then annotated each answer: they labeled it hallucinated or not, marked the exact character-level spans that were wrong or unsupported, wrote explanations, and assigned macro and micro hallucination types. They also wrote six candidate answers per question: one verified correct answer and five plausible distractors written in a style close to the generated output so the correct option cannot be identified by stylistic cues.

A second pass had two research assistants review all 2,400 examples. This flagged 84 examples (3.5%); 34 were confirmed without changes, 11 questions were revised for ambiguity, and 39 examples were revised mainly for span boundaries. The final dataset has 2,400 examples: 1,013 hallucinated (42.2%) and 1,387 non-hallucinated (57.8%), with 1,120 annotated hallucinated spans in total.

Evaluation uses a zero-shot, closed-book setup with no external knowledge bases or web search, temperature 0.0, top-p = 1.0, and a maximum of 1024 new tokens. A two-stage prompt first asks the model for binary detection; only for cases predicted as hallucination does a second prompt request span localization, explanation, and factual verification. Metrics are Macro-F1 for detection, partial-credit span F1 (F1-Sp) for localization, LO-Score and Correct for candidate selection, and a 0-2 judge score (normalized to [0,1]) for explanations, using GPT-5.5 as the LLM judge. Two annotators reviewed a random sample of 25 explanations per domain. Because localization, verification, and explanation only run after a hallucination prediction, the authors state these scores should be read as pipeline metrics.

Why This Matters

The benchmark pushes Arabic hallucination evaluation past binary correctness toward locating, explaining, and correcting factual errors, and it shows empirically that detection, localization, verification, and explanation are separate abilities. It also supplies fine-grained data for a setting, Islamic knowledge, where the danger lies less in a wrong short answer than in fabricated scriptural references or misattributed evidence.

Real-world applications:

• Religious and legal question answering: Answers that look correct but cite a fabricated verse reference or wrong attribution can be caught only with source-level verification, which the benchmark tests.
• Educational and knowledge tools in Arabic: Histories, science explanations, and geography answers with wrong dates, numbers, or locations can be flagged and corrected span by span.
• Retrieval-augmented and search assistants: Verified reference answers and candidate options provide a way to measure whether a system identifies the correct fact among plausible alternatives.
• Model and evaluation development for Arabic: Free, reproducible prompts and scripts let teams benchmark Arabic model reliability and study where models fail.

Industry relevance: The paper deliberately evaluates only open models and excludes proprietary API-based systems such as Gemini or Claude, arguing that changing backends, retrieval components, or web search make it hard to measure the model itself. This makes the benchmark directly usable by teams that need reproducible, inspectable, closed-book Arabic QA evaluation, and it exposes the gap between high detection scores and weak span-level coverage that any production Arabic assistant must close.

Future Directions

• Expand the benchmark to additional domains beyond the four covered, including legal, medical, financial, or conversational Arabic QA, which the authors list as out of scope.
• Generate answers with a range of Arabic and multilingual LLMs rather than the single FANAR-9B generator, to broaden the diversity of hallucination patterns.
• Move from evaluation to training, with settings aimed at improving factual reliability in Arabic language models.
• Improve explanation assessment and span annotation, since the LLM-as-a-judge protocol may introduce bias and character-level boundaries remain hard to pin down for implicit reasoning errors and unsupported claims.

Target Audience

Researchers and practitioners working on Arabic NLP, hallucination and factuality evaluation, and knowledge-intensive QA; builders of Arabic LLM assistants who need fine-grained evaluation beyond accuracy; and annotation or data-curation teams interested in a multi-layer labeling scheme with reported agreement figures (Cohen's kappa 0.93 for binary labels, 0.87 for macro-types, 0.81 for micro-types).

Authors’ abstract

Large language models (LLMs) can generate fluent Arabic answers, yet factual errors remain difficult to detect, localize, explain, and verify. Existing hallucination benchmarks often provide response-level labels, with limited support for identifying the exact erroneous content, explaining why it is incorrect, or selecting the correct factual answer. We introduce \textsc{HalluTruthQA}, a fine-grained benchmark for hallucination evaluation in Arabic question answering. The benchmark contains 2,400 expert-curated examples across four knowledge-intensive domains: Islamic knowledge, history, science, and geography. Each example pairs an Arabic question and a model-generated answer with a verified reference answer, a binary hallucination label, six candidate answers for factual verification, and, for hallucinated answers, character-level erroneous spans, human-written explanations, and macro and micro hallucination types. We evaluate four open-source LLMs, \textsc{Allam}, \textsc{Falcon-H1}, \textsc{Qwen32}, and \textsc{Silma}, in a zero-shot setting across hallucination detection, span-level localization, factual verification, and explanation evaluation. Results show that these tasks capture different abilities: no single model achieves the strongest performance across all tasks, with best scores of 0.880 Macro-F1 for detection, 0.516 F1-Sp for localization, 0.852 LO-Score for factual verification, and 0.644 final score for explanation evaluation. Our taxonomy shows that hallucination evaluation should move beyond detection toward localizing, verifying, and explaining factual errors. The code, dataset, prompts, and evaluation scripts are available at https://gitlab.com/nlpresearcher/HalluTruthQA.