DialFork: Going Beyond Factuality to Evaluate Procedural Hallucination in Conversational Agents

AI summary

Overview

Research area: Natural Language Processing — evaluation of large language model (LLM) conversational agents, specifically a new failure mode called procedural hallucination.

Technical level: Intermediate to Advanced. The paper assumes familiarity with dialogue systems, embeddings, and ranking metrics such as Kendall's tau, but the core idea is explained in accessible terms.

Scope: The paper defines procedural hallucination, introduces the DialFork benchmark (36,278 dialogues built by controlled "forking" of conformant dialogues), and proposes two dialog-flow-aware metrics (FlowKL and FlowScore) evaluated across three tasks.

What This Paper Is About

Most work on LLM hallucination asks whether an agent's statements are factually true. This paper identifies a different failure: an agent can say things that are individually fluent and factually plausible while abandoning the step-by-step procedure it was supposed to follow — skipping steps, switching to the wrong workflow, or chasing a different goal.

The authors build a benchmark that isolates this failure by taking dialogues that follow a procedure correctly and deliberately corrupting them at controlled turn positions, then test whether automated methods can detect and localize the deviation from raw text alone.

Key Contributions

1. A definition and taxonomy of procedural hallucination. The authors formalize an agent's intended logic as a tuple W = (S, A, δ, O) — states, actions, a transition function, and an objective — and decompose the failure into four types: Workflow Identity Corruption (WIC), Step Selection Policy Corruption (SSPC), Goal Replacement (GR), and Conflicting System Constraints (CSC).

2. The DialFork benchmark. 36,278 dialogues spanning 24 tasks in 12 domains, generated by a seed-controlled multi-agent pipeline built on STAR's released role-play instructions and workflow graphs, and validated by a human study with 30 annotators.

3. Three evaluation tasks at three granularities. Set-to-set distance (monitoring a deployed agent against a reference set), dialog-to-dialog distance (procedural retrieval), and turn prediction (real-time guardrails that flag where a dialogue first goes off-track).

4. Two dialog-flow-aware metrics, FlowKL and FlowScore, which compare dialogues by how they unfold rather than by what they say, plus baseline results on all three tasks.

Main Findings

• FlowScore outperforms all baselines on dialogue-level comparison. Across seven encoders (LaBSE, Qwen3-Embedding, SBERT, BGE-M3, D2F, DSE, EmbeddingGemma), FlowScore is the strongest method for every one, with statistically significant gains over every other method (one-sided Wilcoxon, Holm-corrected, p < 0.001), lifting all encoders above τ = 0.81.

• Semantic similarity is not procedural similarity. Comparing whole-dialogue embeddings ranks forked dialogues markedly worse (τ = 0.519–0.747) because a fork and its source largely share content. The gap is largest for Qwen3-Embedding (0.519 vs. 0.812) and smallest for D2F, the encoder trained on dialog flow (0.747 vs. 0.834).

• Construction controls do not explain the results. Turn-count difference is close to chance (τ = 0.126); the "shared prefix" control, which reads fork position off the construction itself, reaches τ = 0.682 — yet FlowScore beats it for every encoder without that cue.

• On set-level distance, all three distribution metrics agree. FBD (0.913 average), FlowKL (0.882), and PRD (0.878) all recover the expected fork ordering, well above length-only controls (0.201 for word-count difference). Differences among the three are not statistically significant (p ≥ 0.43), but FlowKL is best on SSPC (0.903), the type targeting step ordering.

• Streaming detectors localize deviations within about two turns. A 3-gram language model with LaBSE localizes the fork within 1.9–2.2 turns, flagging 93–95% of forks at a false-alarm rate of 12–16%. Node-confidence weighting sharpens the Improbable Transition detector to MAE 1.94, the lowest without ground-truth workflow access.

• Zero-shot LLMs are weaker without the workflow. Without workflow access, five prompted open-weight LLMs localize with MAE 4.50–6.48 (vs. 1.94 for streaming detectors) and flag only 72–83% of forks. Given the ground-truth workflow, Llama-3.3-70B localizes best (MAE 1.83) and reaches the best overall Score (77.09, tied with Gemma4-31B), but false alarms rise to 17–82%.

• The problem remains open. No method exceeds a composite Score of 78 out of 100; the best still miss the fork by about two turns and raise false alarms on 12–30% of conformant dialogues.

• Human validation supports the construction. Blind conformance ratings rise with fork position, from 14/100 at k = 1 to 58 at k = 4 and 94 for conformant dialogues (τ_b = 0.64). Annotators confirmed the injected instruction type in 91% of forks versus 8% for foil instructions; onset was placed within one turn of the fork turn in 77% of cases (exactly at it in 67%); and the pre-fork prefix followed the workflow in 95% of cases.

Methodology in Plain English

The benchmark is built by remixing dialogues that already work. The authors start from STAR, a dialogue corpus that uniquely releases both the workflow graph for each task and the exact instructions given to the humans who originally role-played each conversation. They hand those same instructions to two LLM agents — one playing the system, one playing the user — both served from Gemma4-31B. This yields 2,134 conformant dialogues.

Then comes the forking. For each conformant dialogue, they pick four evenly spaced turn positions. To create a corrupted variant, they re-run the same scenario with the same random seed, so the opening turns reproduce identically, and just before the chosen turn they slip an extra instruction into the system agent's context telling it to hallucinate in one of four ways. From that turn onward, the conversation drifts off-process. Because earlier forks share less of the correct prefix, they are more deviant — this gives a ground-truth ordering by construction, which is what the evaluation metrics are scored against.

Each of the 2,134 conformant dialogues yields 16 forks (4 hallucination types × 4 fork positions), giving 36,278 dialogues in total: roughly 750K turns and 10.4M words, generated in about 33 GPU-days, with 8,536 dialogues per hallucination type.

For evaluation, all inputs are raw transcripts with no annotations and no access to the model that produced them. FlowKL builds an explicit, human-readable graph of which conversational "steps" follow which in the reference set, scores each dialogue by how likely its transitions are under that graph, and reports the KL divergence between the reference and target score distributions. FlowScore borrows the BERTScore idea and applies it turn by turn: compute the best-matching turn in the other dialogue for each turn, form a precision and recall, and take their harmonic mean. Notably, because no contextualized turn encoder is publicly available, the current instantiation of FlowScore does not yet capture turn order.

Why This Matters

Impact on research. The paper reframes hallucination evaluation along a second axis. Existing benchmarks ask whether content is true; DialFork asks whether the process is right, and shows empirically that these are separable — a strong semantic encoder like Qwen3-Embedding performs worst among the seven on procedural ranking. Because the generation pipeline works on any workflow-annotated dialogue dataset, the approach is portable beyond STAR.

Real-world applications:

• Pre-deployment monitoring: a company upgrading its support assistant can check whether the new model's conversations unfold like past ones handled by humans or the previous system, and detect drift after deployment.
• Procedural retrieval: searching conversation logs for past cases that unfolded the same way, to see how similar situations were handled.
• Real-time guardrails: streaming detectors that intervene as soon as an agent skips a required step, without needing a formal workflow specification.
• Regulated intake processes: clinical intake, guided troubleshooting, and scheduling, where the order of steps carries consequences independent of whether any single utterance is false.

Industry relevance. The failure mode is directly relevant to any deployed agentic assistant in customer support, scheduling, or troubleshooting. The streaming baselines are lightweight, require no workflow specification, and operate turn by turn — practical properties for production monitoring, even though the paper notes they are simple first baselines.

Future Directions

• Contextualized turn encoders. The authors note that no public contextualized turn encoder exists, so their FlowScore instantiation does not yet exploit turn order; developing one is flagged as a natural next step.

• Better real-time localization. Since no method exceeds a Score of 78 out of 100 and the best miss the fork by roughly two turns, improving streaming detection with fewer false alarms on conformant dialogues is an open problem the benchmark is designed to support.

• Reducing reliance on privileged workflow access. The strongest non-streaming results require the complete dialogue and often the ground-truth workflow, which is typically unavailable in practice; closing that gap without giving up accuracy is unresolved.

• Procedural fidelity as a training objective. The authors envision procedural fidelity as a first-class property of conversational agents — monitored in deployment, searchable in logs, and ultimately optimized during training so agents stay on process by design.

Target Audience

Researchers and practitioners working on LLM evaluation, dialogue systems, and agentic deployment who need to assess whether an assistant follows a prescribed procedure rather than merely producing truthful statements. It is also relevant to engineers building production monitoring or guardrails for conversational agents, and to dataset designers interested in the controlled-forking construction method and its human validation protocol. Readers should be comfortable with embedding-based similarity and rank correlation metrics.

Authors’ abstract

Large language models (LLMs) deployed as conversational agents are increasingly expected to follow procedural logic. While factual hallucination has received extensive attention, a different failure mode has been overlooked: procedural hallucination, in which an agent produces individually fluent and factually plausible utterances while deviating from the intended conversational process. We introduce DialFork, the first benchmark to measure this failure mode on raw dialogue text. DialFork is built by forking conformant dialogues at controlled turn positions and injecting one of four hallucination types, yielding 36,278 dialogues across 24 tasks in 12 domains; 30 human annotators validate the construction. We define three evaluation tasks that characterize procedural deviation at three granularities: how far a set of dialogues deviates from conformant ones (monitoring deployed agents), how far a single dialogue does (procedural retrieval), and at which turn a dialogue first goes off-track (real-time guardrails). For the first two, we propose dialog-flow-aware metrics, FlowKL and FlowScore, that compare dialogues by how they unfold rather than their content. At the dialogue level, FlowScore significantly outperforms all baselines across seven encoders, showing that semantic similarity is no substitute for procedural similarity. We release DialFork under open licenses at https://github.com/idiap/dialfork, hoping the community will measure conversational process alongside content.