Venus-DefakerOne2: A Vision-Forensics Foundation Model with Broader Coverage

AI summary

Overview

Research area: Computer Vision — visual media forensics, specifically unified (foundation-model) detection and localization of forged images and videos.

Technical level: Advanced. The paper assumes familiarity with multimodal large language models (MLLMs), LoRA fine-tuning, DINOv3-style vision backbones, segmentation decoders, and benchmark protocol design (AUC, balanced accuracy).

Scope (one sentence): The paper introduces Venus-DefakerOne2, a 2B-parameter vision-forensics foundation model built on Qwen3.5-2B that unifies detection of digital and physical forgeries across both images and videos, trained on roughly 50M samples and evaluated on 53 forgery-detection and 14 forgery-localization benchmarks.

What This Paper Is About

Real-world forgery is not one problem but a matrix: content can be digitally manipulated (splicing, inpainting, AI generation) or physically spoofed (mask presentation attacks, document recapture), and it can arrive as a still image or a video. Existing detectors are usually built for one cell of that matrix — one manipulation type, one media format, one acquisition setting — and are studied in separate communities (digital image forensics, face anti-spoofing, document fraud, video forensics) with separate benchmarks.

The authors' earlier model, Venus-DeFakerOne, unified multi-scenario digital image forgery into a single foundation model, but stopped at that boundary: physical forgeries and video media were out of scope. This paper's goal is to extend the boundary along both the forgery-form and media-type dimensions in one deployable model, and to show it is not beaten by domain specialists on their own domains.

Key Contributions

1. A broader-coverage forensic foundation model. Venus-DefakerOne2 is a single 2B-parameter MLLM (Qwen3.5-2B backbone) that performs binary real/fake detection and pixel-level localization across nine forensic domains in the training corpus: AI-generated images, DeepFake faces, tampered documents, forged natural images, face anti-spoofing (images and videos), document anti-spoofing (images and videos), and AI-generated videos.

2. Tide-Style Tuning. A three-phase training schedule that alternates LoRA-only "ebb" phases with a full-parameter "flood" phase, combined with progressively stronger DINOMAC adversarial noise, to balance computational cost, generalization, and fitting of hard samples while limiting catastrophic forgetting.

3. Modality Synergism. A training-plus-inference scheme for the video "modality dilemma": joint modality training (each video seen both as a native clip and as a frame sequence extracted with an adaptive frame-rate strategy, Ada-FPS, with an independent loss per modality) and entropy-based test adaptation at inference, which picks the prediction from the modality with lower Shannon entropy. No extra training or learned router is needed.

4. Anomaly Magnifier with a calibration loop, plus a new benchmark. An auxiliary DINOMAC/DINOv3-Large encoder and mask decoder produce localization masks, with the binarization threshold driven by the MLLM's global forgery probability. The authors also release the Realistic Rumor Video Benchmark (RRVBench): 1,319 authentic videos plus 11,000 synthetic videos from 11 recent generators, scored by macro-averaged balanced accuracy across the 11 generator subsets.

Main Findings

• State-of-the-art across domains: Venus-DefakerOne2 reports 96.7% on DeepFake faces, 93.3% on AI-generated images, 87.6% on tampered documents, 89.6% on manipulated natural images, 93.1% on physical spoofs, and 96.0% on AI-generated videos.
• Breadth of evaluation: It is evaluated on 53 visual media forgery detection benchmarks and 14 forgery localization benchmarks, ranks first on every one of the eight domains, and is state-of-the-art on 40/53 detection and 11/14 localization benchmarks.
• Large gains over the predecessor: Average performance rises from 80.4 to 90.9 on images and from 57.6 to 95.7 on videos compared with DefakerOne.
• Physical forgery is newly covered: This category was entirely outside DeFakerOne's scope; it climbs from 50.7 to 87.2.
• Frontier video generators: On RRVBench, DefakerOne2 reaches 95.3, while 7B-parameter video-forensics MLLMs degenerate to near-chance guessing (best 69.8).
• No specialist beats it at home: The paper states the model is never outperformed by a domain specialist on that specialist's own domain, and its narrow cross-domain spread indicates no domain pays for the generality.
• Localization design matters: The calibration loop makes the mask threshold τ(p) = 0.75 − p/2, so confident-forged inputs get a lower threshold (0.25) and confident-authentic inputs a higher one (0.75); in log-odds terms this is an additive bias bounded within [−log 3, +log 3] ≈ [−1.099, +1.099], i.e., a soft prior rather than a hard override.
• Data composition (examples): Approximately 16M AI-generated-image samples (including ~8M public and 5.3M Ada-FPS-extracted video frames), 24.4M DeepFake image samples (15.1M public, 9.3M private), 3.3M document-tampering samples (0.95M public, 2.338M private, spanning >4,000 document classes), 3.7M forged natural images, and 1.3M AI-generated video samples.

Methodology in Plain English

Model. The system is a chat-style multimodal LLM. It is asked a fixed question — "Is this image/video real or fake? Answer only Real or Fake." — so the answer collapses to a single token, which makes inference fast and gives a confidence value to reuse downstream. Because an LLM reasons at a global level but is poor at pixel-level work, a separate "Anomaly Magnifier" encoder (a DINOv3-Large model fine-tuned with LoRA) supplies fine features that a small convolutional mask decoder turns into a tampering mask.

Coupling the two levels. The mask is normally thresholded at a fixed value, which over-segments benign content. Here the threshold is derived from the model's own global verdict: an input judged confidently fake gets a permissive threshold so faint tampering is not suppressed; an input judged authentic gets a strict threshold so only blatant anomalies survive. The shift is symmetric and capped, so strong pixel evidence can still overrule the global call.

Training in three tides. Fine-tuning 50M highly varied samples is awkward: full fine-tuning overfits early easy samples and forgets pretrained knowledge, while LoRA alone underfits at this scale. So training alternates. First an "Adaptation Ebb": LoRA only (rank 64, α = 128) on an initial 15M samples under light adversarial noise — a cheap warm-up that preserves world knowledge. Then a "Full-Parameter Flood": LoRA weights are merged and the whole model trains on the full corpus (about 40M samples used in this stage) under medium noise. Finally a "Hard-Sample Ebb": LoRA again on hard and newly collected samples under heavy noise, acting as a regularizer and cheap continual adaptation. The three stages use roughly 15M, 40M, and 40M samples, run across 320 NVIDIA A100 GPUs.

Handling videos. Video evidence is sometimes temporal (flicker, motion inconsistency) and sometimes spatial (per-frame texture), and which one dominates is unknown per clip. During training each video is fed both as a native clip and as frames, with separate losses. Frame extraction uses Ada-FPS: a default of 1 FPS, capped at 10 FPS, with each source's rate raised or lowered so that a few abundant sources do not dominate the frame budget. At test time, the first and last 10% of each video are discarded, the middle 80% is sampled, both modalities predict, and the lower-entropy (more decisive) prediction wins.

Data. The corpus combines public datasets, self-synthesized data, and manually annotated in-the-wild attacks. Two pipelines generate synthetic data: Creative Proposer++, which asks MLLMs (e.g., Qwen3.5-32B, GPT-5.5) to describe a real image or video in detail and then feeds that description to a generator (e.g., GPT-Image-2, SeeDance 2.5) so real and fake are semantically matched; and an agent-based loop that analyzes the model's failure cases, identifies missing forgery patterns, and synthesizes targeted new data for the next round.

Why This Matters

Impact on research. The paper argues that forensics has been fragmented into separate communities with separate benchmarks and assumptions, and demonstrates that one model can serve all of them without a specialist beating it on its home turf. That reframes the field's default unit of design from "one detector per threat" to "one foundation model plus per-sample routing," and it introduces two reusable ideas: a parameter-budget schedule that trades off forgetting against fitting, and a training-free entropy rule for choosing input modality.

Real-world applications:

• Content moderation and platform trust & safety, screening uploads that may be AI-generated or manipulated.
• Identity verification and KYC, where face anti-spoofing (physical presentation attacks) and DeepFakes both need to be caught — the paper notes forged media can bypass biometric verification.
• Document fraud, covering credentials, contracts, invoices, and certificates (the private corpus spans >4,000 document classes), including recapture attacks.
• Newsroom and rumor verification, the explicit target of RRVBench, which focuses on military actions, political events, civil unrest, and public health emergencies.
• Judicial and insurance evidence checking, where pixel-level localization masks matter as much as the binary verdict.

Industry relevance. A single 2B-parameter model with one prompt interface replaces a portfolio of scenario-specific detectors, which the authors frame as collapsing a fragmented assembly into one deployable model — directly reducing the cost of deployment and maintenance as new threats appear. The agent-based data loop is also a practical recipe for keeping up with generator releases.

Future Directions

• Extending beyond the current media and forgery matrix. The paper's stated coverage is {digital, physical} × {image, video}; whether the same recipe transfers to other modalities (e.g., audio or audio-visual content) is not addressed and remains open.
• Closing the localization gap. Detection is state-of-the-art on 40/53 benchmarks while localization is state-of-the-art on 11/14 — localization is comparatively less saturated, and the calibration loop's bounded bias may leave room for stronger pixel-level evidence modeling.
• Continual adaptation at generator speed. The agent-based bad-case-to-synthesis loop is presented as an ongoing cycle; how often it must run, and how to avoid drift as new closed-source generators (such as those in RRVBench) keep appearing, is an open engineering and research question.
• Scaling and efficiency trade-offs. The model is deliberately 2B parameters and trained on 320 A100s; whether larger backbones help, and how far Tide-Style Tuning stretches before the LoRA phases underfit, are natural follow-ups. The reported contrast with 7B video-forensics MLLMs (best 69.8 on RRVBench) suggests architecture and training regime matter more than raw size, which invites further study.

Target Audience

Researchers and engineers working on media forensics, AI-generated content detection, and deepfake defense; practitioners building trust-and-safety, KYC/liveness, or document-verification pipelines who need one model instead of many; and MLLM researchers interested in parameter-efficient multi-stage fine-tuning schedules and inference-time modality routing. Readers seeking an introductory treatment of forensics will find the framing accessible, but the method sections assume comfort with LLM fine-tuning and segmentation architectures.

Authors’ abstract

Real-world forgery is inherently heterogeneous: it manifests not only as digital manipulation but also as physical spoofing, and is carried not only by images but also by videos. Despite substantial progress, a unified anti-forgery foundation model covering such a broad threat spectrum remains absent. In this paper, we present Venus-DefakerOne2, which substantially extends the forensic capability boundary along both the forgery-form and media-type dimensions. Trained on tens of millions of open-source and in-house samples, it incorporates two key designs. Tide-Style-Tuning alternates LoRA and full-parameter fine-tuning to balance computational efficiency, generalization, and hard-sample fitting while mitigating catastrophic forgetting. Entropy-based Test Adaptation determines the optimal prediction modality by analyzing prediction entropy. Venus-DefakerOne2 achieves new state-of-the-art performance across a comprehensive suite of benchmarks, including DeepFake faces (96.7%), AI-generated images (93.3%), tampered documents (87.6%), manipulated natural images (89.6%), physical spoofs (93.1%), and AI-generated videos (96.0%). It significantly outperforms both specialized expert models and its predecessor, DefakerOne, establishing itself as a powerful and versatile foundation model for binary-decision forensic tasks.