TriDF: Evaluating Perception, Detection, and Hallucination for Interpretable DeepFake Detection

AI summary

Overview

Research area: Computer Vision / AI-generated media forensics — specifically interpretable DeepFake detection evaluated with multimodal large language models (MLLMs).

Technical level: Advanced. The paper assumes familiarity with MLLM architectures, benchmark design, forgery generation families (GANs, Stable Diffusion, diffusion transformers), and metrics such as CHAIR, Cover, and F-scores.

Scope in one sentence: TriDF is a benchmark of roughly 5K real–synthetic pairs covering 16 DeepFake types generated by over 50 specialized models, built to jointly measure how well MLLMs perceive manipulation artifacts, detect forgeries, and avoid fabricating evidence in their explanations.

What This Paper Is About

DeepFake detectors built on multimodal large language models are increasingly expected not just to output real-or-fake labels, but to explain which visual or acoustic artifacts justify the decision. Existing datasets and benchmarks mostly score binary accuracy or ruator explanations, and none systematically check whether a model's stated reasoning is actually grounded in observable manipulation evidence rather than invented. TriDF addresses this by building a human-annotated benchmark that evaluates perception of fine-grained artifacts, detection accuracy across many forgery families, and hallucination — fabricated reasoning — as three interdependent aspects of the same system.

Key Contributions

1. A three-perspective evaluation benchmark (TriDF) covering 16 DeepFake types across image, video, and audio, with ~5K real-synthetic pairs generated using over 50 specialized models drawn from more than 30 public datasets. Table 1 lists the testing set at 65K with 51 generators.

2. A fine-grained artifact taxonomy splitting manipulation cues into quality artifacts (localized issues such as blurriness, noise, or flicker, some of which are grounded at specific locations like the nasal area, limbs, or background) and semantic artifacts (issues requiring common sense, such as anatomical inconsistencies, object integrity flaws, unrecognizable text, and unnatural prosody). Annotations are collected from humans rather than generated by LLMs to avoid the self-preference bias the paper attributes to GPT-4o-based judging pipelines.

3. A question-format design spanning three types — True-False Questions (<TFQ>), Multiple-Choice Questions (<MCQ>), and Open-Ended Questions (<OEQ>, split into Type-A where the sample is known to be fake and Type-B where the model must classify first) — mapped onto the Perception, Detection, and Hallucination dimensions.

4. A hallucination-aware metric suite. Responses are parsed into artifact lists by an external LLM (Gemini 2.5 Flash-Lite) and scored with Cover (coverage of annotated artifacts), CHAIR (frequency of hallucinated artifacts), Hal (share of responses containing any hallucination), and an Fβ-score with β = 0.5, which weights precision twice as heavily as recall following THRONE.

Main Findings

• Proprietary models lead on perception. On <TFQ> and <MCQ>, GPT-5 and Gemini 2.5-Pro generally outperform open-source MLLMs, and even the strongest systems only moderately exceed the 50.00% random-choice baseline in several settings.

• Question format exposes different weaknesses. Claude Sonnet 4.5 achieves the strongest <MCQ> performance but shows a noticeable drop on <TFQ>, suggesting it exploits answer structure and distractor context rather than making isolated binary judgments.

• Semantic artifacts are the bottleneck. Analysis of artifact-wise <TFQ> accuracy shows quality artifacts such as blockiness, banding, and reflection inconsistency are detected with relatively high accuracy, while semantic artifacts requiring physical or social reasoning (anatomical inconsistencies, abnormal motion, background–subject incoherence) are consistently much harder.

• Localization hints are unreliable. Using Benefit (wrong without a location hint, right with one) and Cost (right without a hint, wrong with one), most models gain little or are actively harmed. MiniCPM-V-2.6 shows costs far exceeding benefits; InternVL3_5-8B, Qwen3-VL-8B-Instruct, and GPT-5 show high instability; only models such as InternVL2_5-8B and Claude Sonnet 4.5 show modest net benefit. Gemini 2.5-Pro and GPT-5 are among those where hints introduce distraction.

• Hallucination is widespread in explanations. On Type-A <OEQ>, GPT-5, Gemini 2.5-Pro, Claude Sonnet 4.5, and LLaVA-OV-72B reach higher Cover, but CHAIR and Hal scores are generally high across models. Claude Sonnet 4.5 records the highest image Cover at 0.6410 while also showing a CHAIR of 0.6241 and a Hal of 0.9953.

• Low hallucination and low coverage can co-occur. InternVL2_5-8B exhibits both low Cover (worse) and low CHAIR (better) because it consistently reports only a small, limited range of artifact types. Claude Sonnet 4.5 shows the opposite pattern, with an average response length roughly twice that of other models.

• Video detection is substantially harder than image detection. For Type-B <OEQ> on video, most models drop in accuracy, and cover roughly halves, indicating a greater difficulty in explaining video DeepFakes.

• Perception does not guarantee detection. Models with similar Type-B detection scores can differ substantially in Cover and hallucination severity, and the paper documents cases where a model correctly identifies artifacts in Type-A yet still misclassifies real-fake pairs in Type-B, or produces high-Cover explanations contaminated by hallucinated artifacts. A specialized detector, FakeVLM, scores only 2.00% average accuracy on the image <TFQ> set.

• The three dimensions are coupled but not redundant, justifying the benchmark's central claim that perception, detection, and hallucination must be evaluated together.

Methodology in Plain English

The authors first collected open-source human-related datasets across image, video, and audio, selecting real samples from test sets or unused training data to simulate real-world conditions. For each of the 16 DeepFake techniques — spanning partially manipulated tasks such as face swapping, facial attribute manipulation, lip-syncing, face reenactment, full-body puppetry, subject-driven editing, and voice conversion, plus fully synthetic tasks such as audio-driven talking head synthesis, identity-preserving generation, text-to-human image/video generation, human image-to-video generation, and voice cloning — they generated fakes with at least three distinct models to enforce one-to-one real-fake pairings. Generator families include GAN-based approaches, Stable Diffusion-based models, diffusion transformer-based models, and proprietary systems. Automatic authenticity and consistency metrics were applied as quality control before annotation.

Next they built a taxonomy of fine-grained artifacts and collected human annotations to establish ground truth, then wrote three question types. Finally, they fed those questions plus the samples to a wide range of MLLMs and scored the responses. Scoring works in stages: an external LLM converts a free-form response into a list of predefined artifacts, that list is compared against the annotated artifacts of the sample, and the overlap drives Cover, CHAIR, Hal, and the Fβ-score. For <MCQ>, a model gets +1/K for each correct option and loses 1/(M−K) for each wrong one, where K is the number of correct options and M the total — so guessing is penalized. If a model's mapped artifact list is empty, CHAIR is set to 1 as a penalty; the same penalty applies if the model calls a fake sample real.

Why This Matters

DeepFakes target people specifically, which amplifies their potential for misinformation campaigns, financial fraud, identity theft, reputational attacks, and harassment. As MLLMs take on detection roles, an explanation that sounds plausible but cites non-existent artifacts is arguably worse than no explanation, because it invites misplaced trust in systems that may influence public perception or legal judgments. TriDF shifts evaluation from a single accuracy number toward a diagnostic view of why a model succeeds or fails.

Real-world applications:

• Content moderation platforms that need to justify takedowns rather than issue opaque automated decisions.
• Digital forensics and legal review, where an analyst must point to specific evidence supporting a manipulation claim.
• Journalism and fact-checking, where reporters must verify whether submitted media of a public figure is authentic.
• Financial and identity verification systems, where synthetic voice or video is used to impersonate account holders.

Industry relevance: The finding that localization hints often hurt performance has direct implications for prompt and tooling design in commercial detection products. The metric suite (Cover, CHAIR, Hal, F0.5) also offers a template for measuring explanation reliability in any MLLM deployment where hallucination carries a cost.

Future Directions

1. Solving the semantic artifact gap. Semantic artifacts requiring physical and social reasoning remain the primary bottleneck, and no current model closes it; methods specifically targeting common-sense consistency, not just pixel-level forensics, are needed.

2. Making spatial guidance trustworthy. Since location hints help only a few models, more work is needed on how to combine spatial cues with visual reasoning without destabilizing the model, potentially through training rather than prompting.

3. Reducing hallucination without suppressing coverage. The inverse relationship between Cover and CHAIR in models like InternVL2_5-8B and Claude Sonnet 4.5 raises the question of how to raise coverage while keeping fabricated artifacts low — the paper points to grounding approaches such as FFTG and AvatarShield as starting points.

4. Deeper three-dimensional analysis. The paper notes that finer-grained analysis — partial correlations and stratified perception-to-detection curves under different hallucination regimes — is provided in the supplementary material, leaving broader cross-modality studies (including the audio results deferred to supplementary) as open ground.

Target Audience

Researchers and practitioners working on DeepFake detection, AI-generated media forensics, and MLLM evaluation will benefit most, along with benchmark designers interested in measuring interpretability and hallucination rather than accuracy alone. Trust-and-safety teams, digital forensics analysts, and product engineers building MLLM-based detection or verification systems will find the metric design and the negative results about localization hints directly actionable. The paper is best suited to readers with prior exposure to multimodal model evaluation; those new to the area will need background on MLLM architectures and explainability metrics to follow the tables.

Authors’ abstract

Advances in generative modeling have made it increasingly easy to fabricate realistic portrayals of individuals, creating serious risks for security, communication, and public trust. Detecting such person-driven manipulations requires systems that not only distinguish altered content from authentic media but also provide clear and reliable reasoning. In this paper, we introduce TriDF, a comprehensive benchmark for interpretable DeepFake detection. TriDF contains high-quality forgeries from advanced synthesis models, covering 16 DeepFake types across image, video, and audio modalities. The benchmark evaluates three key aspects: Perception, which measures the ability of a model to identify fine-grained manipulation artifacts using human-annotated evidence; Detection, which assesses classification performance across diverse forgery families and generators; and Hallucination, which quantifies the reliability of model-generated explanations. Experiments on state-of-the-art multimodal large language models show that accurate perception is essential for reliable detection, but hallucination can severely disrupt decision-making, revealing the interdependence of these three aspects. TriDF provides a unified framework for understanding the interaction between detection accuracy, evidence identification, and explanation reliability, offering a foundation for building trustworthy systems that address real-world synthetic media threats.