AV-FABLE: Audio-Visual Manipulation Forensics in Long-Form Videos

AI summary

Overview

Research area: Computer vision and audio-visual forensics, specifically temporal manipulation localization in long-form video, evaluated with audio-visual large language models (AV-LLMs).

Technical level: Advanced. The paper assumes familiarity with temporal action localization (mAP, IoU, focal and DIoU losses), deepfake detection benchmarks, and the encoder/interface structure of omni-modal LLMs.

Scope in one sentence: The paper introduces AV-FABLE, a 16,650-pair (998.1-hour) dual-timeline benchmark for sparse audio and visual manipulations in open-scene long-form videos, and proposes NFSR, a supervised readout of frozen AV-LLM features that localizes audio and visual edits on separate timelines.

What This Paper Is About

Most audio-visual deepfake benchmarks use short, speaker-centered clips and evaluate a single merged timeline, so they cannot say when a manipulation happened or which stream was altered. Real edits in long-form footage often touch only a few seconds of audio, of video, or of both, sometimes at different boundaries, and a video-level "fake" label says nothing about location. The paper builds a benchmark that preserves both temporal boundaries and modality identity, then asks whether today's audio-visual LLMs can actually expose the manipulation evidence they seem to encode internally.

Key Contributions

1. AV-FABLE, a dual-timeline benchmark and task. 16,650 pristine–manipulated twin pairs (33,300 videos, 998.1 hours) built from 27 editing configurations, with 10,306 audio and 11,847 visual events. Each pair shares a source video and an identical export process, and audio and visual timelines are annotated separately. The benchmark defines four tasks: temporal localization, video-level detection, four-way modality attribution (clean, audio-only, visual-only, both-track), and evidence explanations, plus three out-of-distribution (OOD) settings.

2. Native-track evaluation. The primary localization protocol, AV-mAP (the mean of A-mAP and V-mAP), forbids a visual prediction from matching an audio event even at identical timestamps. The paper shows this exposes failures hidden by joint-timeline matching: NFSR's visual predictions alone reach 64.04 Joint-mAP, above every visual-only baseline, while scoring zero A-mAP.

3. A finding about AV-LLMs. Zero-shot Qwen2.5-Omni and Qwen3-Omni reach only 51–52 detection AUROC and at most 0.30 mAP on either track, yet linear probes on their frozen perception features separate manipulated from authentic segments at 90.8 (audio) and 80.8 (visual) AUROC. Forensic evidence appears to be encoded but not exposed through language outputs.

4. NFSR (Native Forensics via Separated Readouts). A method that reads frozen Qwen3-Omni features — combining an intermediate layer and the interface layer — conditions them on host-video statistics, and localizes audio and visual manipulations with independently parameterized temporal decoders.

Main Findings

• Zero-shot prompting is near chance. Qwen2.5-Omni-7B, Qwen3-Omni-30B-A3B, and video-SALMONN 2+ 7B all land at roughly 50 AUROC (49.98–52.28) and at most 0.30 mAP on either track in-domain. Supervised timestamp fine-tuning of Qwen3 raises A-mAP to 57.30 but V-mAP only to 18.30.

• Frozen features carry strong forensic signal. NFSR on frozen Qwen3-Omni features reaches 96.04 detection AUROC, 83.61 attribution F1, and 91.30 AV-mAP (92.24 A-mAP, 90.36 V-mAP) in-domain.

• NFSR beats a matched-feature localizer. Using the same frozen features and training budget, UMMAFormer reaches 84.76 AV-mAP versus NFSR's 91.30, a 6.50-point gap, with the largest single gain on the visual track (+10.74 V-mAP) and a 6.86-point AUROC improvement.

• OOD gains hold without adaptation. NFSR has the highest AV-mAP in all three OOD settings: 70.15 under source transfer (HDTF plus a source-disjoint TASLE subset), 49.12 on held-out generators, and 68.07 on held-out TI2V (visual-only, so AV-mAP equals V-mAP).

• Realized false alarms stay near budget. With thresholds fixed at a 5% budget on 256 pristine source-validation videos, NFSR's realized FAR is 2.97–4.67% across all evaluation sets. UMMAFormer reaches 12.21% FAR on source transfer with lower coverage; DAV-Det covers more visual events there but at 24.03% and 40.45% FAR.

• Visual coverage under shift is the open weakness. NFSR reaches only 26.42–41.21 V-IoU on shifted data, and visual-only baselines remain competitive on their own track — the AV-FABLE-trained ActionFormer slightly exceeds NFSR's V-mAP on held-out generators and leads AUROC, attribution F1, and V-IoU on held-out TI2V, but its AV-mAP caps at 37.50 in the two audio-containing settings.

• Ablations show each component matters more under shift. Removing intermediate-layer features costs 6.55 AV-mAP in-domain and 8.72 on OOD-All; removing interface features costs 4.07 OOD-All AV-mAP and more than doubles OOD FAR; removing host conditioning costs 4.73 OOD-All AV-mAP.

• Separated decoders avoid cross-modal false firing. On audio-only speech replacements, the separated visual readout fires on 2.9% of both edited videos and pristine twins, while a cross-modal variant's visual head fires on 4.6% of pristine twins but 11.8–15.1% of edited videos.

• Dataset shortcuts are controlled. Metadata-only classifiers reach at most 54.6 AUROC on AV-FABLE, and pristine twins pass through identical cropping, reinsertion, and export steps.

Methodology in Plain English

The benchmark is built by editing events that already exist in real footage rather than by inserting synthetic clips into arbitrary windows. A construction pipeline called SEE-Forge assembles an event catalog for each host video from captions, speech transcripts, acoustic onsets, motion, and object interactions, using ActivityNet, EPIC-KITCHENS-100, and (as held-out source) HDTF. Gemini 3.7 Flash acts as a planner: it picks one catalogued event, specifies the track, interval, and semantic change, and a second model verifies that the target is observable and that a generator can execute it.

Each edit is routed to one of 27 generator-plus-route configurations — frame-conditioned generation, masked or unmasked video-to-video editing, joint audio-visual generation, audio generation, or speech replacement — with SAM 2 masks constraining localized visual edits. Generators see 2–5 s of surrounding context, and outputs are reinserted at recorded coordinates with checks on motion continuity, visual anchors, and loudness at the seams. Every manipulated video has a pristine twin from the same host passing through the same export steps, which suppresses content and processing confounds. Temporal labels come from the executed edits themselves, and for 20,529 events a model drafts candidate evidence records separating what is observed inside the event, context from surrounding media, and the judgment those observations support. Two reviewers independently check all 22,153 events (Cohen's κ = 0.84 for edit validity, 0.82 for evidence support), with a third adjudicating the 7.7% of disagreements.

On the modeling side, NFSR takes the frozen audio and visual encoders of Qwen3-Omni and extracts two feature types: an intermediate layer for fine perceptual detail and the interface layer that feeds the language model. Both are aligned to positions every 0.25 s, attention-pooling visual spatial tokens, and fused into 256-dimensional features via a gated projection. Each modality's features are then modulated by scale-and-shift vectors predicted from that video's own channel-wise mean and standard deviation (FiLM-style), so a local change is interpreted against its host context. Two independent decoders — one audio, one visual — build temporal pyramids with masked dilated convolutions and predict, at each point, a confidence plus left and right boundary distances in stride units, merged by temporal Soft-NMS. Training uses sigmoid focal loss plus 1D DIoU loss per branch for 12 epochs with AdamW at 1e-4, with pristine videos and unmodified tracks as negatives. Video-level scores come from the highest proposal confidence per track, converted to upper-tail probabilities against pristine validation videos; a single threshold per decision, calibrated once at a 5% false-alarm budget, yields detection and the four-way attribution label without changing the predicted intervals.

Why This Matters

Impact on research. The paper reframes audio-visual forensics from clip-level classification to sparse, modality-aware temporal localization, and supplies an evaluation protocol that penalizes a model for finding the right moment on the wrong stream. Its most transferable result is diagnostic rather than architectural: the large gap between zero-shot AV-LLM outputs and linear probes on the same frozen features suggests current omni-modal models have perceptual knowledge that their language interface does not surface. That is a concrete target for anyone working on multimodal LLM probing, readout heads, or forensic explainability.

Real-world applications:

• Platform trust and safety: flagging the specific seconds and stream of a modified upload rather than issuing a whole-video verdict.
• Journalism and fact-checking: verifying broadcast or user-generated footage where an inserted sound or altered action changes the meaning of an otherwise authentic sequence.
• Legal and insurance evidence review: producing timestamped, modality-attributed findings with accompanying evidence statements from long recordings.
• Archive and broadcast integrity: auditing large video collections for localized edits, including cases where audio and visual changes do not overlap (561 such both-track videos exist in the benchmark).

Industry relevance. The benchmark's minute-scale videos (median 88 s, 80% between 33 and 207 s, with a median 9.45% of each video manipulated) match realistic upload lengths better than talking-head clip benchmarks. The strong result from frozen features plus lightweight readouts is also practically attractive: teams can reuse an existing omni-modal encoder and train small decoders rather than fine-tuning a large model. The reported limits matter too — explanation quality remains the weakest dimension, with baselines producing well-formed but often wrong operation or target descriptions.

Future Directions

• Extending to broader real-world editing processes. The current 27 configurations would need to grow toward the editing tools and pipelines actually used in the wild, including more held-out routes like the withheld TI2V setting.
• Refining perceptual boundary annotations. Labels mark where edits were applied, not where a viewer perceives change, so a perceptible change may cover only part of an interval or begin slightly off its boundaries; reviewers check boundaries but do not re-draw them.
• Robustness to media corruption. The authors name this explicitly, and the reported 26.42–41.21 V-IoU under distribution shift shows visual coverage degrades well before audio does.
• Faithful evidence-grounded explanations. Explanation is the hardest task in the benchmark; closing the gap between well-formed text and correct identification of the operation and target is left open.

Target Audience

Researchers and graduate students in video forensics, deepfake detection, and temporal action localization who need a benchmark with modality-separated labels. Also relevant to practitioners building trust-and-safety or verification systems on top of omni-modal LLM encoders, and to multimodal LLM researchers interested in what frozen perception features encode versus what language outputs reveal. Readers should be comfortable with mAP/IoU-based detection metrics and encoder-decoder localization architectures; the dataset construction and annotation protocol sections are accessible without that background.

Authors’ abstract

Localized AI manipulations in long-form videos often affect brief events within otherwise authentic content, making video-level classification insufficient for forensic analysis. Existing audio-visual deepfake benchmarks largely focus on short talking-head clips, while long-form temporal forensics remains predominantly visual. We introduce AV-FABLE, a dual-timeline benchmark for sparse manipulations in open-scene long-form audio-visual videos. AV-FABLE contains 16,650 pristine--manipulated twin pairs (998 hours) created with 27 editing configurations, with separate audio and visual timelines that enable modality-specific temporal localization and expose errors hidden by joint-timeline evaluation. On AV-FABLE, zero-shot prompting of audio-visual large language models (AV-LLMs) yields near-chance detection and poor temporal localization. In contrast, supervised readouts of their frozen perception features recover strong forensic signals, suggesting manipulation evidence is encoded in these representations but not effectively exposed through language outputs. Motivated by this observation, we propose NFSR(Native Forensics via Separated Readouts), which reads frozen AV-LLM features with separate audio and visual temporal decoders. NFSR achieves the best audio-visual localization performance in-domain and across all three out-of-distribution settings, including transfer to held-out videos from an external long-video forensic benchmark. AV-FABLE, code, and model weights will be publicly released.