Simple Domain Generalization for Strong Pixel-Level Image Tampering Detection in Modern VLMs

AI summary

Overview

Research area: Computer vision and image forensics — specifically pixel-level image tampering detection (locating the exact edited pixels in an image) using modern vision-language models (VLMs) as both the source of manipulation and the detector backbone. The paper sits at the intersection of tampering localization, segmentation-based detection, and domain generalization.

Technical level: Advanced. The paper assumes familiarity with segmentation backbones, LoRA parameter-efficient fine-tuning, VLM training pipelines, and out-of-distribution (OOD) evaluation protocols.

One-sentence scope: The paper introduces PIXAR-DG, a training-recipe-only framework (balanced real/tampered minibatch sampling, late injection of a small new-domain source, and a conservative learning-rate schedule) that improves cross-generator pixel-level tampering localization in VLMs while using only 19.2% of the original PIXAR training data.

What This Paper Is About

Detectors that localize AI-edited image regions tend to overfit to the artifacts of whichever generator they were trained on, so they degrade when confronted with images from a different VLM editing pipeline. This paper defines and studies "domain generalization for pixel-level image tampering detection in modern VLMs" — the problem of learning a localizer that stays robust when the target generator (e.g., GPT-Image-2.0 or Gemini-3.1) was never seen during training. The goal is a simple, implementation-light training recipe, compatible with existing segmentation-based tampering detectors, that produces transferable tampering cues rather than generator-specific artifacts.

Key Contributions

1. A training framework for domain-generalized pixel-level tampering detection. Unlike methods that overfit to domain-specific artifacts from particular generators, PIXAR-DG is designed to learn transferable tampering cues that remain effective under cross-model and out-of-distribution shifts, and is described as simple to implement.

2. Three practical ingredients combined. Balanced real/tampered minibatch training, a late-injection mechanism for small emerging distributions, and a low-learning-rate adaptation schedule (including replacing the original warmup-decay scheduler with a constant scheduler).

3. Stronger results with far less data. On the PIXAR dataset, the method uses only 19.2% of the original dataset scale (73K tampered images versus PIXAR's full 380K Qwen-Image training set) yet improves performance across all new domains.

4. A new problem framing plus released code. The paper positions PIXAR-DG as "a completely new task and problem in the community" and releases code at https://github.com/VILA-Lab/PIXAR-DG.

Main Findings

• Headline OOD gains over PIXAR. The abstract and the contribution list report that the framework outperforms the prior state-of-the-art PIXAR by a large margin of 26.1% and 26.8% relative improvement in average gIoU and cIoU respectively, across the OOD VLMs GPT-Images-2.0, Gemini-3.1, FLUX.2, and Seedream 4.5. Note that Section 4.2 reports a different pair of figures for the same 7B comparison: 21.4% relative improvement in average gIoU and 21.1% in average cIoU, with absolute gains of 19.34% in Pixel Recall and 4.80% in Pixel F1 over PIXAR-7B. The 7B table values underlying the Section 4.2 numbers are avg gIoU 0.159 (PIXAR-7B) to 0.193 (PIXAR-DG-7B) and avg cIoU 0.166 to 0.201.

• Full localization table (7B and 13B). At 7B, PIXAR-DG reaches Pixel Recall 45.11, Pixel F1 33.29, average gIoU 0.193 and average cIoU 0.201, versus PIXAR-7B's 25.77, 28.49, 0.159 and 0.166; SIDA-7B reaches 33.71, 27.42, 0.163, 0.159 and LISA-7B collapses to 1.30, 2.54, 0.013, 0.014. At 13B, PIXAR-DG reaches Pixel Recall 62.19, Pixel F1 37.42, average gIoU 0.222 and average cIoU 0.232, versus PIXAR-13B's 33.49, 30.95, 0.176 and 0.183; SIDA-13B obtains 16.84, 21.25, 0.120 and 0.119, and LISA-13B obtains 3.06, 5.73, 0.028 and 0.031. PIXAR-DG-13B lifts average gIoU from 0.176 to 0.222 and average cIoU from 0.183 to 0.232 over PIXAR-13B, while yielding a 6.47% gain in Pixel F1.

• Largest per-generator gain on FLUX.2. At 7B scale, the largest absolute gain is on FLUX.2, where gIoU rises from 0.170 to 0.222, a 30.6% relative gain. The paper states PIXAR-DG improves localization on all four OOD generators.

• Binary real-vs-tampered classification also improves. PIXAR-DG-7B achieves 79.6% average OOD accuracy versus PIXAR-7B's 69.6% and SIDA-7B's 18.1%; PIXAR-DG-13B reaches 84.0% versus PIXAR-13B's 59.0% and SIDA-13B's 29.8%. The paper reports absolute gains of 10.0% over PIXAR-7B and 25.0% over PIXAR-13B.

• Qwen-Image is the best base source. In a single-source study over six candidate training generators, Qwen-Image yields the highest average accuracy at 60.7%, surpassing the second-best source (Gemini-2.5) by 32.4%, and the strongest worst-case OOD accuracy at 41.2% (on Gemini-2.5). Every other training source has a substantially lower worst-case OOD accuracy (all ≤ 14.5%), with several collapsing to single digits — for example 2.6% for GPT-Image-2.0-trained models on Gemini-2.5 and 2.6% for FLUX.2-trained models on Gemini-3.1.

• Gemini-2.5 is the best companion source. Combining Qwen-Image with each remaining generator individually, Qwen-Image + Gemini-2.5 achieves the highest Avg. (OOD) of 66.3% (Avg. All 68.5%), versus 63.6% for Gemini-3.1, 59.1% for FLUX.2, 54.1% for Seedream 4.5, and 54.4% for GPT-Image-2.0. Gemini-2.5 is also the worst-case OOD target for Qwen-Image-trained detectors, motivating it as the companion domain.

• More same-generator data stops helping after 70K. Varying the Qwen-Image base sample count N: 30K gives average accuracy 65.0 and Pixel F1 27.20; 70K gives 79.6 and 33.40; 150K gives 64.9 and 24.63; the full 380K gives 63.4 and 26.74. The paper hypothesizes that as the Qwen-Image portion grows, the relatively small Gemini-2.5 supervision becomes a smaller fraction of each minibatch, weakening its contribution to OOD generalization.

• Gains are not just from multi-source data. Retrained on identical Qwen-Image + Gemini-2.5 data with identical budgets, PIXAR-DG attains 66.3% average accuracy and 29.89% Pixel F1, versus 52.0% and 28.59% for PIXAR and 49.5% and 28.37% for SIDA.

• Each component contributes. Ablation with MiniBatch Sampling (MS) only: 65.3 average accuracy, 26.53 Pixel F1. MS + Late Injection (LI): 64.0 and 28.65. MS + LI + LR schedule: 66.3 and 29.89. The paper concludes every component can yield improvement and combining all three achieves the strongest OOD generalization.

• Stability and geometry effects. The gradient norm of the <CLS> head fluctuates heavily under random sampling (red curve) and becomes substantially smoother and flatter under balanced minibatch sampling (green curve); the loss-landscape visualization shows PIXAR-DG converging to a noticeably flatter minimum than PIXAR. The paper also states it frequently observes training collapse when the proposed sampling strategy is not used.

Methodology in Plain English

The detector keeps PIXAR's base architecture and produces three outputs from shared hidden features: a pixel-level tamper logit map (turned into a probability mask with a sigmoid), a multi-label semantic logit vector for the categories of tampering, and a natural-language description of the tampered content. Training combines five losses — sigmoid cross-entropy for semantics, pixel-wise BCE on a thresholded mask, a DICE loss for mask quality, a cross-entropy classification loss on the <CLS> token for real-vs-tampered detection, and a causal language-modeling loss for the description. The mask threshold is τ = 0.05, following PIXAR.

Three practical changes carry the method. First, instead of drawing minibatches uniformly from a pooled dataset (where the empirical real/tampered ratio simply mirrors the dataset's imbalance), each minibatch is built with a fixed ratio ρ, with ⌊ρB⌋ real images and the rest tampered; unless stated otherwise ρ = 0.5, so real and tampered images are equal in every batch. Second, "late injection": the detector is trained only on the large base domain (Qwen-Image) until convergence, then the small new-domain set (3K Gemini-2.5 tampered images) is mixed in for the remaining steps, with a time-dependent mixture controlled by α ∈ [0,1]. Third, the base learning rate is cut from 1×10⁻⁴ to 2×10⁻⁵ and the original warmup-decay scheduler (linear warmup from 0 to 1×10⁻⁴ over the first 100 steps, then linear decay) is replaced with a constant schedule that holds 2×10⁻⁵ after warm-up.

Training uses LoRA with rank r = 8, scaling factor α = 16, and dropout 0.05. The CLIP vision tower, multimodal projector, and SAM image encoder stay frozen; the SAM mask decoder and four task-specific heads are fine-tuned. Everything runs for 5 epochs of 500 optimization steps with batch size 8, AdamW with (β₁, β₂) = (0.9, 0.95) and zero weight decay, on 4 RTX A6000 GPUs, with results reported across three independent random seeds. The loss weights follow PIXAR's default configuration. Ablations use a reduced set of 7,136 Qwen-Image and 400 Gemini-2.5 tampered images.

Data setup: the base source is a 70K subset of Qwen-Image tampered images sampled from PIXAR's 380K training set, plus 3K Gemini-2.5 tampered images drawn from a held-out partition of the PIXAR Gemini-2.5 test pool and removed from the final evaluation set, giving 73K tampered images total (19.2% of the original scale). Testing adapts the PIXAR protocol by dropping the GPT-Image-1.5 and Gemini-3 subsets and adding GPT-Image-2.0 and Gemini-3.1; Qwen-Image and Gemini-2.5 act as in-domain generators, while GPT-Image-2.0, Gemini-3.1, FLUX.2, and Seedream 4.5 are held out as OOD. Baselines are PIXAR, SIDA, and LISA in 7B and 13B variants, evaluated with g-IoU, c-IoU, binary classification accuracy, and ID/OOD average accuracy.

Why This Matters

Impact on research. The paper reframes VLM generators and editing pipelines as evolving domains and argues that scaling same-generator data is insufficient for cross-generator robustness — 380K Qwen-Image images performed worse than a curated 70K that was paired with a 3K companion source. It also provides a controlled, single-source and source-composition study of which generators transfer best, and shows that a purely recipe-level change (no architecture change) can move pixel-level OOD metrics substantially.

Real-world applications:

• Fact-checking and journalism: localizing exactly which pixels of an image were edited, not merely flagging the image as manipulated.
• Content moderation on platforms handling user-uploaded media.
• Forensic evidence for detecting misinformation, fraud, and malicious image edits.
• Robustness to newly released closed-source generators, reducing the need to retrain detectors from scratch for each new model.

Industry relevance. Detection vendors and platform trust-and-safety teams face a stream of new editing models; a training recipe that generalizes to unseen generators and does not require the newest generator's full dataset lowers the cost of staying current. Parameter-efficient LoRA fine-tuning with frozen vision and image encoders keeps the compute footprint modest (4 RTX A6000 GPUs, 5 epochs), and the released code at https://github.com/VILA-Lab/PIXAR-DG makes the recipe directly adoptable. The paper also notes more capable detectors could be misused to build stronger adversarial attacks, and warns about over-reliance, false positives on benign images, false negatives giving false trust, and uneven performance if training data are biased toward certain domains, objects, cultures, or editing styles.

Future Directions

• Reduce dependence on labeled new-domain data. Late injection requires access to a small amount of labeled data from emerging distributions, which the authors note may not be available for closed-source or newly released VLMs.
• Handle extreme OOD shifts. Performance depends on the diversity and quality of available source domains; if source data does not cover sufficiently representative manipulation patterns, the learned cues may still fail.
• Improve the detector architecture, not only the recipe. The method improves training rather than architecture, so upper-bound performance remains constrained by the capacity and design of the underlying segmentation-based model.
• Extend source composition and test coverage. The paper references appendix studies of compositions with more sources (Appendix C.4), additional sampling-ratio results (Table 11 of Appendix C.6), in-domain results (Table 9 of Appendix C.3), test-set statistics (Table 8 of Appendix C.2), and additional visualizations (Appendix C.5) — content not included in the provided text, leaving those specific figures unreported here.

Target Audience

Researchers and engineers working on image fore

Authors’ abstract

Modern vision-language models (VLMs) have significantly improved image generation and editing capabilities, making pixel-level image tampering detection increasingly important yet challenging under cross-model and out-of-distribution shifts. This work studies domain generalization for pixel-level image tampering detection in modern VLMs like ChatGPT, Gemini, Qwen-Image, etc., aiming to learn tampering localization models that remain robust across diverse VLM-generated manipulation distributions. We propose a simple yet effective domain-generalized training framework built on two practical strategies. First, we introduce a balanced minibatch sampling scheme that strategically samples tampered and real images in each minibatch, preventing biased optimization toward either manipulated artifacts or clean-image priors and avoiding training collapse, ensuring that each optimization step receives proper sampled gradient signals. Second, we adopt a simple late-injection strategy, where the detector is first trained on large-scale base data until stable convergence, and then exposed to a small amount of newly selected supporting data from emerging VLM distributions, improving adaptability without overfitting to limited new domains. Together, these components provide a simple yet strong recipe for improving pixel-level tampering localization and OOD robustness across modern VLMs. Despite the conceptual simplicity, our framework outperforms the prior state-of-the-art PIXAR by a large margin of 26.1% and 26.8% relative improvement in average gIoU and cIoU, respectively, across OOD VLMs of GPT-Images-2.0, Gemini-3.1, FLUX.2, and Seedream 4.5. Our code is available at https://github.com/VILA-Lab/PIXAR-DG