To Trust or Not to Trust: Retrieval-Augmented Fact Checking in Speech

AI summary

Overview

Research area: Speech and audio-language modeling, fact verification, retrieval-augmented generation (RAG), and multimodal misinformation detection.

Technical level: Intermediate. Readers benefit from familiarity with retrieval-augmented generation, large audio language models, and ASR, though the paper's core question is stated plainly.

Scope: The paper introduces VeriSpeak, a 3,879-item probe benchmark of spoken claims, and uses it to test whether five Large Audio Language Models (LALMs) can verify spoken claims against retrieved textual evidence.

What This Paper Is About

Fact-checking research has largely assumed the claim arrives as text. But misinformation increasingly arrives as speech — news clips, podcasts, interviews, political speeches, and social videos — so a verifier must recover a claim from audio and then judge it against textual evidence. The paper asks whether the factual verification ability that models show on written claims survives when the identical claim is spoken, and whether retrieving textual evidence or prompting explicit reasoning can repair the failure.

Key Contributions

1. A new benchmark, VeriSpeak. A probe benchmark of 3,879 synthesized spoken claims spanning temporal (year: 1,451), geographical (location: 2,226), and relational (relation: 202) facts, with balanced true and false labels, drawn from 659 subject entities across 60 countries. Each item ships with an audio file, an automatic Wav2Vec 2.0 transcript, and a ground-truth veracity label, and is publicly released on Hugging Face.

2. A controlled six-condition evaluation protocol. The paper defines comparable settings that isolate modality, model interface, retrieval, and reasoning: C0 Text-LLM, C1 Text-LALM, C2 Speech-LALM, C3 Speech-LALM with chain-of-thought (CoT), C4 Transcript-RAG, and C5 Transcript-RAG with CoT, plus an audio-query retrieval variant and a text-RAG upper bound. Two comparison metrics are introduced: Δ (drop relative to the paired text-only LLM) and Lift (gain over the paired speech-only LALM).

3. Identification of a specific failure mode: claim-evidence conflation. A diagnostic study shows that under RAG, standard LALMs often stop verifying the spoken claim and instead verify the retrieved passage, which explains why retrieval alone produces only small gains.

4. Evidence that reasoning must be paired with retrieval. The paper shows CoT alone hurts speech-only verification, but CoT combined with retrieved evidence helps, and a thinking-tuned LALM benefits most, reaching 86.1% accuracy.

Main Findings

• A large, consistent text-speech modality gap. Qwen-7B scores 75.3% on written claims, while Qwen-Audio-Chat and Qwen2-Audio-7B reach only 50.4% and 49.2% on the same claims spoken (Δ of 24.9 and 26.1 points). Audio-Flamingo-3 drops from 71.8% to 58.5% (13.3 points). For Phi-4-multimodal, the prose of Section 5 reports a drop from 68.0% to 49.2% (18.8 points) while Table 2 lists its speech-only average as 53.3% with Δ = 14.7; both figures appear in the paper.

• The gap is driven mainly by the speech interface, not by degradation of text ability. Text-mode differences vary (Δ_text of 9.7 and 6.8 points for Qwen-Audio-Chat and Audio-Flamingo-3, but −1.1 and −4.4 for Qwen2-Audio-7B and Phi-4-multimodal). Speech-input degradation, however, is universal: Δ_speech is 15.2 points for Qwen-Audio-Chat, 27.2 for Qwen2-Audio-7B, 6.5 for Audio-Flamingo-3, and 23.2 for Phi-4-multimodal.

• Retrieval helps only partially. Transcript-RAG with multi-e5 improves every standard LALM, but modestly, and the paper reports Lift ranging from 1.4 to 4.7 points and averaging only 2.3 points across standard LALMs. Residual gaps to the text-only baseline remain large: Δ of 23.5, 24.4, 11.8, and 14.1 points for Qwen-Audio-Chat, Qwen2-Audio-7B, Audio-Flamingo-3, and Phi-4-multimodal.

• Transcript retrieval beats audio-query retrieval. AudioQuery-RAG with CLAP yields an average Lift of −0.3 points versus +2.3 points for transcript-RAG with multi-e5. CLAP's Recall@1 is approximately 0.3%, so the top-ranked passage usually lacks the relevant evidence. Retriever ablation Recall@1 values are 85.6% (multi-e5), 78.2% (e5-large-v2), 49.8% (msmarco-MiniLM-L12-v3), and 0.3% (CLAP); multi-e5 is most reliable overall, though msmarco performs best for Qwen-Audio-Chat.

• RAG shifts the object of verification. In a manual analysis of 50 Qwen2-Audio-7B-Instruct Transcript-RAG examples drawn from the factually incorrect pool, only 31% of extracted claims matched the original spoken claim, 65% matched the retrieved evidence, and 4% were other or ambiguous.

• CoT without evidence backfires. Every standard LALM scores worse with CoT alone: Qwen-Audio-Chat 50.4% to 44.9%, Qwen2-Audio-7B 49.2% to 44.4%, Audio-Flamingo-3 58.5% to 57.0%, and Phi-4-multimodal 49.2% to 16.7% (the latter caused by format failure, with only 29.8% of outputs containing a parseable verdict). Average Δ rises from 20.8 to 31.9 points.

• CoT with evidence works. Adding CoT to Transcript-RAG improves every standard LALM (51.8% to 52.8%, 50.9% to 57.0%, 60.0% to 65.5%, 53.9% to 60.7%), an average gain of 4.9 points, and reduces Δ by 4.9 points on average.

• Thinking-tuned models change the picture. Audio-Flamingo-next-think improves at every step: 66.4% (speech-only) to 71.8% (CoT) to 81.4% (Transcript-RAG) to 86.1% (RAG+CoT), a Lift of 19.7 points and a negative Δ of −14.3, exceeding the Qwen2.5-7B text-only baseline of 71.8%. It still trails the text-only RAG upper bound of 92.4% by 6.3 points (Δ_UB). For comparison, standard Audio-Flamingo-3 goes 58.5% to 57.0% to 60.0% to 65.5%, a Lift of 7.0 and Δ_UB of 26.9.

• Speech recognition errors explain part, but not all, of the drop. Overall WER ranges from 0.13 to 0.23 across LALMs; two human transcribers on 50 sampled clips achieved an average WER of 0.105 (0.15 and 0.06 individually). Roughly 74% of Audio-Flamingo-3's WER edit operations involve subject names or other proper nouns, versus about 3% error on non-name tokens. The transcription-contrast diagnostic (Δ_recognition) ranges from 5.6 to 11.6 points across models.

• Demographics of the benchmark. The 659 subject entities span 60 countries, with the largest representation from the United States (278) and England (79). Gender representation is 59.5% male (392), 40.2% female (265), 0.2% non-binary (1), and 0.2% N/A (1).

Methodology in Plain English

The researchers built VeriSpeak from an existing knowledge base of short celebrity biographies. They enriched each biography with Wikipedia metadata, resolving birth country and gender and splitting umbrella entries such as the United Kingdom into England, Scotland, Wales, and Northern Ireland. They then flagged three fact types using simple detectors: four-digit year patterns for temporal facts, spaCy named-entity recognition for geographical facts, and keyword matching on marriage, parent, and sibling terms for relational facts.

To keep errors attributable to reasoning rather than ambiguity, they asked Llama-3.2-3B-Instruct to rewrite each flagged fact as a single, pronoun-free sentence, discarding outputs shorter than four words or containing unresolved pronouns. Balanced negatives were generated deterministically per category: shifting years by a non-zero offset in [-10, +10] clamped to a plausible range; swapping country, city, or nationality spans with same-type candidates; and either swapping relation words (for example, "married to" to "divorced from") or swapping person and organization entities with same-type knowledge-base entities. Celebrities without a valid negative were discarded to preserve label balance.

All claims were synthesized with the Coqui TTS engine using the tacotron2-DDC vocoder, deliberately with a single speaker so that voice identity cannot serve as a spurious cue. Each item was paired with an automatic Wav2Vec 2.0 transcript, kept deliberately imperfect to reflect deployment conditions.

Five LALMs from three families were evaluated — Qwen-Audio-Chat and Qwen2-Audio-7B-Instruct (Qwen-7B backbone), Audio-Flamingo-3 and Audio-Flamingo-next-think (Qwen2.5-7B-Instruct backbone), and Phi-4-multimodal-instruct (Phi-4-mini-instruct backbone) — each paired with its text-only backbone as a reference. For transcript-based RAG, the wav2vec 2.0 transcript queries a knowledge base using multi-e5 by default, with e5-large-v2 and msmarco-MiniLM-L12-v3 as sensitivity checks; the audio-query variant uses CLAP. Accuracy is reported per category and averaged across Year, Location, and Relation, with Δ and Lift as comparison metrics. CoT outputs lacking a parseable verdict in the required <answer> field are scored as incorrect.

Why This Matters

Impact on research. The paper argues that spoken fact-checking is not simply text fact-checking with an ASR front end. It separates three distinct failure sources — perception errors, model-level degradation introduced by multimodal adaptation, and the inability to keep the spoken claim separate from retrieved evidence — and provides a controlled benchmark with the settings needed to study each. The claim-evidence conflation result is the sharpest contribution: it reframes retrieval failure as a targeting failure rather than a retrieval-quality failure.

Real-world applications:

• Automated verification of claims made in news clips, podcasts, interviews, and political speeches.
• Voice-based content moderation on social and video platforms, where claims arrive as audio rather than captions.
• Misinformation monitoring of viral audio and short-form video, where transcripts are noisy and entity names are frequently misrecognized.
• Assistive or editorial fact-checking tools that surface retrieved evidence alongside a spoken claim for human review.

Industry relevance. Any platform handling spoken media — social video, podcasting, broadcast, or voice assistants — faces the problem this paper tests. The findings suggest that bolting retrieval onto a speech model is insufficient, and that reasoning-tuned models plus careful prompting are what make retrieved evidence usable. The low Recall@1 of the audio-query retriever also indicates that as long as ASR transcripts are good enough, transcript-based retrieval is the more practical engineering path.

Future Directions

• The provided paper content is truncated at the start of the Future Work section, so the authors' own listed next steps are not reported.
• Open question raised by the results: what training or architectural change would let standard LALMs preserve the spoken claim as the verification target instead of verifying retrieved text, given that 65% of extracted claims matched the evidence.
• Reducing the remaining 6.3-point Δ_UB gap between speech-based RAG (86.1%) and fully text-based RAG (92.4%).
• Improving entity recognition during speech understanding, since roughly 74% of Audio-Flamingo-3's transcription edits involve proper nouns, and these tokens are label-critical for verification.

Target Audience

Researchers and practitioners working on speech-language models, retrieval-augmented generation, fact verification, and multimodal misinformation detection. It is also useful for engineers building content-moderation or media-monitoring pipelines who need to know where speech-based verification currently fails, and for benchmark designers interested in how to construct controlled, balanced, category-stratified evaluation sets for cross-modal reasoning.

Authors’ abstract

Online misinformation increasingly appears in spoken formats such as news clips, podcasts, interviews, political speeches, and social media videos, creating a need for fact-checking systems that can verify claims directly from speech. We introduce VeriSpeak, a probe benchmark for studying speech-based fact verification in Large Audio Language Models (LALMs). VeriSpeak contains 3,879 spoken claims spanning temporal, geographical, and relational facts, with balanced true and false labels. The benchmark is designed to examine whether factual verification ability transfers from text to speech, and whether retrieval-augmented LALMs can use textual evidence to correctly support or refute spoken claims. Our experiments reveal a consistent text-speech modality gap: LALMs that verify written claims reliably often fail on the same claims when spoken. Moreover, retrieval alone provides limited gains because models frequently conflate retrieved evidence with the spoken claim. In contrast, retrieval combined with explicit reasoning improves claim-evidence comparison, with a thinking-tuned LALM reaching 86.1% accuracy. VeriSpeak highlights that effective speech misinformation detection requires not only speech understanding, but also grounded reasoning over retrieved evidence. The dataset is publicly available via Hugging Face at https://huggingface.co/datasets/abhiram4572/VeriSpeak.