DF26: We Cannot Tell Fake From Real Anymore

AI summary

Overview

Research area: Computer vision, specifically AI-generated video / deepfake detection benchmarks, with a focus on modern text-to-video and image-to-video generative models.

Technical level: Intermediate. The benchmark construction and human study are easy to follow, but the detector results require comfort with AUROC/EER metrics and cross-dataset evaluation protocols.

Scope: The paper introduces DF26, a controlled benchmark of single-person public-speaking videos (271 real, 2,420 synthetic, 2,691 total) produced by seven recent video generators, and shows that both humans and state-of-the-art detectors struggle to separate the generated clips from the real ones.

What This Paper Is About

Most deepfake detectors are built and tested on older datasets built around face swapping and facial reenactment, but modern text-to-video and image-to-video systems now generate entire realistic scenes from a text prompt or a single frame. The authors ask whether existing detectors (and people) can still tell real from fake when the fake is a fully synthesized public-speaking video, and they build a controlled benchmark to measure this.

The paper's core message is that the answer, for now, is largely no: on DF26 (arXiv:2609.07369v1, 07 Sep 2026), several state-of-the-art detectors score close to random chance, and human accuracy on the generated clips is 52.6%, barely above chance.

Key Contributions

1. A controlled single-person public-speaking benchmark for generated-video detection. DF26 contains 2,691 videos across three public-speaking scenarios and seven modern generators. Open-source generators are evaluated in both text-to-video (T2V) and image-to-video (I2V) settings where supported, while commercial generators are evaluated only in a T2V setting.

2. A reproducible dataset protocol and metadata design. DF26 provides source identifiers, scenario labels, generator labels, generation-mode labels, released prompts, technical normalization details, and preprocessing information for both dataset construction and detector evaluation.

3. Evidence that visual state-of-the-art deepfake detectors degrade substantially on modern generators, with some scoring close to random chance.

4. A human study comparing DF26 against CelebDF++ (CDFv3) and DeepSpeak v2 (DSv2). Human accuracy on DF26 deepfakes is 52.6%, near chance, compared with 74.5% on CDFv3 and 69.8% on DSv2, while accuracy on real videos is similar across the three datasets.

Main Findings

• Benchmark composition: DF26 holds 271 real clips and 2,420 generated clips. The real clips come from three sources: OpenVid (197), TalkingCelebs (62), and MAVOS (12), distributed across Direct-to-Camera/Casual (100), Official Statement (71), and Studio Interview (100). Synthetic clips come from seven generators: closed-source Wan 2.6, Veo 3.1, Grok Imagine 1.0, and Kling 3.0, plus open-source Wan

Authors’ abstract

We introduce DF26, a novel benchmark for detecting AI-generated videos containing fully synthetic clips produced by recent text-to-video and image-to-video models. The videos capture single-person public-speaking scenarios, spanning direct-to-camera recordings, official statements, and studio interviews - 271 real and 2,420 synthetic videos generated by seven modern video models. The study on DF26 shows that human performance in detecting AI-generated videos, as well as state-of-the-art deepfake detectors, is close to random chance. Our results highlight the limitations of current evaluation protocols and motivate the need for benchmarks that explicitly measure robustness to modern generative model distribution shifts.