Temporal Slice Learning for AI-Generated Video Detection with 400$\times$ Fewer FLOPs

AI summary

Overview

Research area: AI-generated video detection (media forensics / computer vision), specifically efficient detection of synthetic videos produced by modern text-to-video and image-to-video generators.

Technical level: Intermediate. The core idea is intuitive, but the paper assumes familiarity with Transformers, positional encoding, FLOPs-based efficiency reporting, and standard video-recognition benchmarks.

Scope: The paper proposes a representation-level reformulation of AI-generated video detection — compact frequency-aware "temporal slices" instead of full video volumes — instantiated as a lightweight network (TSC-Net), plus a new benchmark (AI-Artist) built from recent high-fidelity generators.

What This Paper Is About

As generative video models become more visually convincing, the traces that reveal a fake are increasingly subtle temporal inconsistencies — flickering textures, unstable structures, or motion that is unnaturally smooth — rather than obvious per-frame artifacts. The paper argues that existing detectors waste most of their computation on redundant scene appearance because they process entire video volumes with heavy backbones, which both costs a lot of FLOPs and can dilute the weak signals that actually matter. The goal is to restructure each video into a compact representation that isolates how selected local regions evolve over time, and then detect fakes from that representation with a very small model.

Key Contributions

1. A new formulation: frequency-aware temporal slice consistency learning. Instead of full-frame video modeling or static patch mining, the paper reparameterizes a video into temporal slices — selected spatial rows and columns tracked across time — so that local temporal evolution is explicitly encoded and redundant appearance is suppressed.

2. TSC-Net (Temporal Slice Consistency Network). A lightweight detector integrating task-driven slice localization and assembly, hierarchical positional encoding (frozen sinusoidal structural encodings plus a learnable frequency embedding), and a compact Transformer that reasons over cross-slice and cross-frequency dependencies.

3. The AI-Artist benchmark. A new evaluation set of 380 AI-generated videos from recent high-fidelity generators absent from GenVideo, split into a web-collected subset (221 videos, e.g., Kling 3.0) and a self-generated subset (159 videos from models such as Runway Gen-4.5, PixVerse V5.6, Hailuo 2.3, Seedance 2.0, and Luma Ray3.14), paired with 380 real videos from the GenVideo test split for class-balanced binary evaluation.

4. An empirical efficiency argument. Reported gains come with 11.6× fewer parameters, over 400× fewer model-forward FLOPs, and 3.6× faster model inference than the video-based baseline ReStraV, alongside much lower false-positive rates on real video.

Main Findings

• Slice representations are more separable than patches. On GenVideo, natural vs. AI-generated videos in the patch domain have AUROC close to chance (0.502 and 0.525 for the two frequency regions analyzed), but after reorganizing the same content into temporal slices, high-frequency slices reach AUROC 0.770 and low-frequency slices 0.650. This is the paper's central empirical motivation.

• Best averaged accuracy on GenVideo. Averaged over the 10 test subsets, the method reports mean ACC 0.9698, AP 0.9973, and AUROC 0.9967. This exceeds ReStraV (0.9562 ACC, 0.9961 AP, 0.9959 AUROC) by 1.36, 0.12, and 0.08 percentage points respectively, and exceeds TALL (0.9602 ACC) by 0.96 percentage points. Against the training-free D3 method (0.9740 AP, 0.9722 AUROC), the gains are 2.33 and 2.45 percentage points.

• Large efficiency gains. The model uses 1.731 M parameters vs. 20.06 M for ReStraV (11.6× fewer), 0.109 vs. 44.20 model-forward GFLOPs (over 400× fewer), and 1.478 vs. 5.307 ms model inference (3.6× faster). Including preprocessing, the total is 0.187 vs. 44.212 GFLOPs (236× fewer) and 4.676 vs. 13.444 ms (2.9× faster end to end, the lowest total cost among compared methods). The gap between FLOPs reduction and wall-clock speedup is attributed to fixed costs such as memory access and kernel launches. Against XCLIP, DeMamba, and D3, the model-forward FLOPs reduction exceeds 1,200×.

• Strong performance on the newer AI-Artist benchmark. The method reports ACC 0.8558 with roughly 0.98 AP and AUROC, surpassing D3 and ReStraV (around 0.97) by about 1 percentage point. TALL has the highest ACC on this benchmark, but the paper notes its AP and AUROC are markedly lower, and does not report its ACC numerically in the text (it is shown only in the figure).

• Very low false positives on real video. On 10,000 real MSR-VTT clips (7,180 source videos, 20 categories) at a 0.5 threshold, TSC-Net yields 7 false positives (0.07% FPR, 99.93% specificity), versus 39 for ReStraV (0.39%) and 422–7,391 for the other detectors. On six overlapping 2,500-clip stress subsets (strong temporal change, high contrast, rich high-frequency content, camera motion, residual motion after compensation, weak motion), it produces 0, 1, 5, 1, 0, and 3 false positives (FPR ≤ 0.20%).

• Robustness to post-processing. Under Gaussian blur, JPEG compression, and Gaussian noise, the paper reports the highest AP and the most stable behavior across degradation levels; JPEG compression leaves its performance almost unchanged, while Gaussian noise is the hardest perturbation for all detectors.

• Slice construction choices matter. Ablations show that replacing slices with a plain selected patch (0.9871 GenVideo mAP) or resizing (0.9919) underperforms the full model (0.9973), confirming that the temporal-slice structure — not just the reduced input size — drives the gains. Using four temporal windows instead of one ("4-Seg.") improves metrics (0.9982 GenVideo mAP, 0.9849 AI-Artist mAP) but raises preprocessing time from 3.198 to 8.908 ms, so the single-window default is kept.

Methodology in Plain English

The pipeline has two stages.

Stage 1 — build the slices. The video is first cropped to remove invalid borders (using the first frame, with the same crop applied to every frame so temporal alignment is preserved). The system then picks a "key frame" as the frame with the largest average pixel change relative to the next frame, and keeps a short window of 8 frames centered on it. From that key frame it locates two small patches: one with the highest gradient energy (high-frequency, i.e., detailed/textured) and one with the lowest (low-frequency, i.e., smooth/coarse). Both are good candidates for where artifacts hide — unstable details in the first case, abnormal smoothness in the second. Within each patch, it measures how much each row and each column changes over time, keeps the top-scoring rows and columns, and stacks those pixel lines across time. The result is a "slice" — a 2D picture of how one line of the video evolves. First- and second-order temporal differences are also computed and stacked, giving the model explicit views of change and acceleration of change. Row-wise and column-wise slices are produced separately.

Stage 2 — classify with a small Transformer. Each slice is split into small non-overlapping patches, each patch is flattened and linearly projected into a token, and a class token is prepended. Every token gets a positional encoding made of two parts: a fixed sinusoidal encoding that records structural facts (row vs. column, which slice, where in the slice), and a learnable embedding that marks whether the token came from the high- or low-frequency group. A 3-layer, 4-head Transformer with embedding dimension 256 processes the sequence, and the class token is passed through layer norm, a projection, and an MLP to a sigmoid output trained with binary cross-entropy.

Training setup. Training follows the GenVideo protocol used by DeMamba: real videos from Youku-mPLUG plus fakes from ten generators (ZeroScope, I2VGen-XL, SVD, VideoCrafter, Pika, DynamiCrafter, SD, SEINE, Latte, OpenSora). Testing is strictly out-of-domain — GenVideo's test split uses a different real source (MSR-VTT) and ten unseen generators (ModelScope, MorphStudio, MoonValley, HotShot, Show_1, Gen2, Crafter, Lavie, Sora, WildScrape). Training uses batch size 32, learning rate 3×10⁻⁴ with cosine annealing, AdamW, and 100K iterations on an NVIDIA RTX PRO 6000.

Why This Matters

Impact on research. The paper pushes back on the prevailing "scale up the backbone" instinct in generated-content detection. Its argument is a representation argument: if artifacts occupy only a small fraction of the video volume, then restructuring the input is a better use of compute than enlarging the model. The patch-vs-slice AUROC analysis (0.502/0.525 → 0.770/0.650) gives a concrete diagnostic that other researchers can reuse, and the efficiency numbers make lightweight detectors a realistic baseline rather than a compromise. The AI-Artist benchmark also addresses a real methodological problem: existing benchmarks are being saturated by older generators and no longer reflect what current models produce.

Real-world applications (implied by the work, not claimed as results):

• Platform-scale content moderation, where per-video cost determines whether detection can run on every upload rather than a sampled fraction.
• Newsroom and fact-checking verification of user-submitted clips before publication.
• Legal, insurance, and forensic review of video evidence, where low false-positive rates on authentic footage are as important as catching fakes (the 0.07% FPR result speaks directly to this).
• Edge or on-device screening and provenance tooling, since the model is small (1.731 M parameters) and cheap in FLOPs.

Industry relevance. Cost per inference and false-positive volume are the two numbers that decide whether a detector is deployable. The paper reports both, and reports them against a strong baseline (ReStraV) rather than only against weak ones, which is the comparison practitioners actually need.

Future Directions

• Better slice localization. Localization currently relies on a fixed gradient-energy criterion on a single key frame. The paper's own appendix includes an "Alternative Slice Localization" section, leaving open whether learned or attention-driven selection would find more diagnostic regions — particularly when artifacts appear away from the highest- or lowest-frequency patch.

• Temporal coverage vs. cost trade-off. The four-window variant improves mAP on both benchmarks but nearly triples preprocessing time (3.198 to 8.908 ms). Closing that preprocessing gap would make multi-window coverage practical, and the paper explicitly notes that the end-to-end speedup lags the FLOPs reduction because of fixed overheads.

• Keeping pace with generator evolution. AI-Artist is a snapshot of generators available at publication. Whether a slice-consistency detector trained on ten generators transfers to the next generation — and whether it can be updated cheaply — is the central open question for deployment.

• Adversarial and laundering robustness. Robustness is tested against blur, JPEG, and noise, but not against deliberate evasion (re-encoding pipelines, compression chains, or attacks crafted against the slice-selection step). Since the detector depends on specific localized temporal cues, targeted removal of those cues is a natural threat model to test.

Target Audience

Researchers and engineers working on synthetic media detection, video forensics, and efficient vision architectures. It is most useful for readers who already know Transformer-based video models and want either (a) a representation-design alternative to scaling backbones, or (b) a practical, low-FLOP detector they can benchmark against. Practitioners building moderation or verification pipelines will find the efficiency and false-positive tables the most directly actionable parts; readers new to the field should start with the motivation figures and Section 2 before the results tables.

Authors’ abstract

Recent advances in generative video models have improved visual realism, making detection increasingly dependent on subtle temporal inconsistencies. However, these signals are weak and spatially localized, while most of the video content is redundant. This creates a fundamental mismatch: existing detectors rely on heavy backbones to model full videos, which leads to high computational cost and limited scalability. In this work, we rethink AI-generated video detection from a representation perspective. Instead of modeling entire videos, we restructure them into compact temporal slice representations that isolate informative temporal evolution. We introduce a lightweight framework using frequency-aware temporal slice consistency learning, where informative spatial rows and columns are aggregated over time to form slice-domain inputs. This reparameterization suppresses redundant appearance while preserving discriminative temporal dynamics, enabling efficient and targeted detection. By constructing slices from both high- and low-frequency regions, our method exposes complementary temporal artifacts, including unstable details and abnormal motion smoothness. We instantiate this idea in a Temporal Slice Consistency Network, which integrates task-driven slice localization, hierarchical positional encoding, and a lightweight Transformer to model cross-slice and cross-frequency dependencies. We further introduce AI-Artist, a new benchmark with artist-curated videos from recent high-fidelity generators, including Seedance and Kling. Experiments on GenVideo and AI-Artist show that our method achieves strong performance while requiring over 400$\times$ fewer model-forward FLOPs and 3.6$\times$ faster model inference than the strong video-based baseline ReStraV (2.9$\times$ faster end to end, including preprocessing), enabling scalable and practical deployment.