ToolHACE: A Dataset for Detecting Hallucinations in Tool-Based LLM Responses

AI summary

Overview

Research area: Natural language processing, specifically hallucination detection and evaluation of tool-augmented large language models (LLMs) that call external APIs, search engines, calculators or databases before writing a final answer.

Technical level: Intermediate. Readers should be comfortable with hallucination detection, span-level annotation, fine-tuning of encoder and decoder models, and standard F1 evaluation, but no specialized mathematics is required.

Scope (one sentence): The paper introduces ToolHACE, a span-level dataset of 48,741 tool-use dialogues labeled with five hallucination categories, and benchmarks prompting-based, transfer-based and in-domain detectors on it.

What This Paper Is About

Existing tool-use benchmarks check whether a model picks the right tool and passes valid arguments, then treat the final natural-language answer as out of scope. That leaves a gap: even after the correct tool returns accurate data, a model can still contradict it, add unsupported claims, omit required facts, or pretend it performed an action no available tool can do. The paper calls these post-tool-call hallucinations and builds ToolHACE, a span-level benchmark for detecting and localizing them, together with an evaluation of how well current detectors handle the task.

Key Contributions

1. A new span-level dataset, ToolHACE, for detecting hallucinations in the post-tool-call stage of tool-augmented LLMs, built on top of the ToolACE corpus and labeled with five categories (Correct, Answer Mismatch, Overgeneration, Undergeneration, Missing Tool).
2. A benchmark of state-of-the-art methods on that dataset, spanning zero-shot and few-shot prompting, transfer supervision from external hallucination corpora, and in-domain fine-tuning across encoder-only and generative backbones.
3. An evaluation of how well knowledge transfers from RAG-style hallucination detection to the tool-augmented setting, including PsiloQA, RAGTruth and HAD, plus synthetic tool-calling corruptions and the RAGTruth-trained lettucedetect checkpoints.
4. An ablation comparing the hallucination-type distribution of the synthetic data against naturally occurring errors from deployed small LLMs, showing the two are closely aligned.

Main Findings

• In-domain supervision gives the strongest localization. Training on the ToolHACE training split produces the best span-level results: the text reports Qwen3.5-2B-LoRA reaching the highest span-level F1 of 79.28, and the results table lists the in-domain Qwen3.5-2B row at a span-level average of 78.95 and Qwen3.5-0.8B at 76.12. In-domain ModernBERT models are also strong but by a smaller margin over zero-shot prompting (ModernBERT-base 65.51, ModernBERT-large 64.72 span-level average).

• Post-tool-call hallucinations are widespread. The paper reports that modern tool-augmented LLMs hallucinate even when the correct tool is called and returns accurate information, motivating the benchmark.

• Zero-shot prompting is competitive; few-shot prompting is harmful. All three zero-shot LLMs beat the supervised transfer systems on span localization, with span-level averages of 65.79 (GPT-4o), 65.51 (GPT-5.4-mini) and 67.84 (Kimi 2.6). Few-shot prompting sharply reduces span-level performance for every evaluated LLM, reportedly because models imitate the demonstrated output patterns rather than ground carefully in the tool trace, sometimes repeating the tool call or its response and increasing context length.

• Tool-agnostic supervision transfers unevenly. PsiloQA and HAD transfer poorly to ModernBERT models (PsiloQA span-level average 4.91 for ModernBERT-large, 5.22 for ModernBERT-base; HAD 13.37 for ModernBERT-large, 7.21 for ModernBERT-base) but improve considerably with Qwen-LoRA backbones (PsiloQA 36.95 for Qwen3.5-0.8B and 44.04 for Qwen3.5-2B). LookbackLens provides very limited span localization (35.03 average). RAGTruth transfers more consistently across backbones (44.22 for ModernBERT-large, 44.28 for Qwen3.5-2B), while Glaive is strongest for Qwen3.5-2B (47.26) and weaker for ModernBERT-large (29.36).

• Response-level and span-level results diverge. In-domain models cluster tightly at response level (93.36 to 95.40 average), suggesting backbone choice matters less for detecting that an error exists than for locating it. Few-shot prompting leaves response-level scores broadly similar to zero-shot despite the sharp span-level drop, so response-level scores alone can hide localization failures. RAGTruth is the most consistent transfer setting at response level.

• Undergeneration is a distinct challenge. Because it corresponds to missing information rather than an erroneous span, it has no explicit span to localize. The paper reports it only for decoder-based models and argues it is better treated as a response-level detection problem. In-domain Qwen3.5-2B reaches 93.26 response-level F1 on undergeneration, but transfer settings are far lower (RAGTruth with Qwen3.5-0.8B at 6.76, with Qwen3.5-2B at 12.67). Zero-shot undergeneration ranges from 37.79 (GPT-5.4-mini) to 58.13 (Kimi 2.6).

• Synthetic and real error distributions align. Among positives, the ToolHACE test split contains 35.0% overgeneration, 32.2% missing_tool, 22.2% undergeneration and 10.6% answer_mismatch (4,693 hallucinated samples), versus real-world outputs at 29.8%, 23.3%, 32.7% and 14.2% respectively (2,155 hallucinated samples). Synthetic data has more overgeneration and missing_tool; real outputs have more undergeneration and answer_mismatch.

• Leakage and length shortcuts were identified and removed. A lexical artifact appeared in 99.0% of missing_tool instances (1,497 in the test split, 6,062 in train), all ending with "Would you like…"; these were rewritten with equivalent continuations and span offsets recomputed. Overgeneration answers were systematically longer and undergeneration answers systematically shorter than clean ones, which was addressed with counterfactual clean twins generated by Qwen3.6-35B-A3B. Train/test isolation also removed all examples whose dialogue_id appeared in the test split.

• Human validation shows high label agreement. Ten annotators each reviewed 50 randomly sampled examples (500 validated instances). Accuracy was 97% for both clean and undergeneration, 96% for answer_mismatch, 93% for overgeneration and 90% for missing_tool; most disagreements were borderline class-assignment cases rather than span-localization errors.

• Multi-judge filtering retained most test data. Three independent judges (GPT-5.2, GPT-OSS-120B, GLM 5.2) verified the test split; examples surviving with at least 2/3 agreement number 11,606, or 90.6% of the raw test set. The two GPT-family judges rejected 0.3% and 3.2% of examples, while GLM 5.2 rejected 4%.

Methodology in Plain English

The team started from ToolACE, a large tool-calling corpus of 11,300 dialogues, and kept only samples with a complete interaction chain (user query, assistant tool call, tool response, assistant answer), separating them into single-hop traces (one tool invocation) and multi-hop traces (at least two complete tool-use cycles). Missing tool responses and grounded completions in some single-hop samples were filled in synthetically, then structurally filtered.

Hallucinations were created three ways. Route A used GPT-OSS-120B with class-specific 5-shot prompts to edit tool responses or answers, providing stylistic diversity. Route B edited the JSON tool response directly: corrupting individual leaf values independently (Type 1a) or a whole subtree while leaving siblings intact (Type 1b) to create Answer Mismatch, deleting fields so the model fabricates them (Type 2, Overgeneration), and appending a sentence referencing a tool absent from the system prompt (Type 3, Missing Tool). Route C sampled 2,500 ToolACE single-hop dialogues at random (seed 42) and let four small open-weight models (Llama-3.2-3B-Instruct, Qwen3.5-2B/4B-Instruct, Gemma-4-26B-A4B) generate natural answers at temperature 0.7 and top-p 0.95 with up to 1,024 output tokens, yielding 9,163 retained responses, annotated by two LLM judges (DeepSeek-v4-Flash and GLM-4.7) with GPT-4o arbitrating.

Because the design principle was to apply one well-defined edit per sample, the error span follows mechanically from the edit rather than from a separate annotation pass. The test split was then verified by three independent judges with a 2/3 agreement rule, quality-controlled for lexical and length shortcuts, isolated to prevent dialogue-level leakage, normalized for Unicode, and enriched with fields such as available_tools and turn_role.

Evaluation runs at two levels: response level (does the answer contain any hallucinated span) and span level (are the exact erroneous text segments recovered), with span-level F1 as the primary metric because localization is the goal. Span matching uses greedy pairing with an Intersection-over-Union threshold of 0.75. The final dataset has 48,741 samples: 37,135 train and 11,606 test, with clean 16,335/5,252, overgeneration 7,511/2,316, missing_tool 4,931/1,521, answer_mismatch 3,702/1,196 and undergeneration 4,656/1,321.

Why This Matters

Impact on research. The paper shows that tool-calling benchmarks and span-level hallucination benchmarks cover two different axes, and that neither currently covers the post-tool-call answer in a tool-grounded setting. It also provides evidence that supervision from RAG-style hallucination corpora does not automatically transfer to tool traces, which reframes how researchers should choose training data for hallucination detectors.

Real-world applications:

• Coding and data agents that call APIs, run calculations or query databases and then summarize results, where a contradiction of the tool output or an invented value can corrupt downstream decisions.
• Customer-facing assistants that claim to have booked, sent, fetched or scheduled something when no such tool exists (the Missing Tool category, which has no direct counterpart in prior benchmarks).
• Retrieval- and search-augmented assistants where a plausible-sounding elaboration is added beyond what the retrieved evidence supports.
• Monitoring and auditing pipelines for deployed agents, where automatic span-level flagging of ungrounded sentences supports human review.

Industry relevance. Companies shipping tool-using agents need automatic ways to tell a faithful answer from a fabricated one. The ablation showing that synthetic corruption mirrors real failure modes of small deployed models (4,693 synthetic versus 2,155 real hallucinated samples with closely matching type distributions) supports using this kind of data for practical guardrail training and regression testing.

Future Directions

• Multi-label annotation: each dialogue currently carries a single dominant hallucination type, so co-occurring error types within one interaction are not modeled.
• Multilingual extension. The paper focuses on English tool-augmented settings, and its table also includes a Qwen3.5-2B row evaluated on an "RU test" that scores far lower than the English test (response-level average 46.55 versus 95.40), though the paper body does not discuss this row.
• Better treatment of undergeneration, which has no erroneous span and is argued to belong to response-level detection rather than span localization.
• Reducing dependence on the ToolACE source corpus, whose domain coverage, tool selection distribution and interaction patterns may not reflect all real-world tool use, and extending evaluation to planning-heavy or long-horizon agentic workflows beyond tool calling.

Target Audience

Researchers and engineers working on LLM hallucination detection, tool-using agents and agent evaluation will benefit most, especially those building guardrails or detectors for the answer stage of tool-augmented pipelines. It is also relevant to practitioners choosing training data for hallucination detectors, since the transfer results show that dataset and backbone choices interact strongly, and to annotation-focused teams interested in the multi-judge and human-validation protocol described. Beginners will find the taxonomy and benchmark useful but will need background in span-level evaluation to follow the results tables.

Authors’ abstract

Tool-augmented language models ground their responses in external tool outputs, improving reliability beyond parametric knowledge alone. However, even when the correct tool is called and returns accurate information, models may still contradict such outputs, omit critical details, or generate unsupported claims. Existing hallucination benchmarks for tool-augmented dialogue primarily focus on the tool-calling stage, evaluating whether the correct tool and arguments are selected, while overlooking errors that arise during post-tool response generation. To address this gap, we introduce ToolHACE, a span-level benchmark for hallucination detection in tool-augmented LLM responses, covering five fine-grained hallucination categories. Through extensive experiments with modern hallucination detectors, we show that post-tool-call hallucinations are widespread and remain challenging to detect. Our results further demonstrate that effective detection requires training on specialized tool-augmented corpora, while transfer from tool-agnostic hallucination datasets performs poorly in this setting. These findings highlight the need for dedicated benchmarks and models for reliable hallucination detection in tool-augmented language systems.