DinoLizer: Separating VAE and Diffusion Artifacts in Generative Inpainting Localization

AI summary

Overview

Research area: Computer vision / multimedia forensics — specifically image forgery localization (IFL) for images manipulated with generative inpainting models.

Technical level: Advanced. The paper assumes familiarity with Vision Transformers, LoRA fine-tuning, latent diffusion pipelines, and VAE encoders/decoders, plus standard forensic metrics such as IoU and F1.

Scope: The paper proposes DinoLizer, a DINOv2-based pixel-level localizer that separates VAE reconstruction artifacts from diffusion inpainting artifacts using a three-class training scheme, and introduces the FOSSIL benchmark for evaluating inpainting localization.

What This Paper Is About

Modern inpainting models such as Stable Diffusion do not just insert or remove objects — they re-encode the entire image through a VAE, so even untouched regions carry generative traces. Existing detectors lump these re-encoded regions in with "real" pixels, which is a contradictory supervisory signal. DinoLizer's goal is to localize the actually manipulated regions in arbitrary-resolution images by explicitly giving VAE-reconstructed areas their own class, and by evaluating this on a new, harder benchmark called FOSSIL.

Key Contributions

1. A three-class localization framework (Real, VAE, Inpainted). Instead of the conventional binary "inpainted vs. real" setup, the model treats auto-encoded regions outside the inpainting mask as a distinct class, so VAE fingerprints are learned rather than confused with authentic content.

2. FOSSIL (Foundation dataset for Occluded Structure Synthesis and Inpainting Localization). A new benchmark pairing real COCO images with tampered images from contemporary inpainting models (DreamShaper8, SD1.5, SD2, SDXL, SD3, Flux Kontex, and Qwen Edit Image), comprising 12,704 images (roughly 2,000 per model).

3. A parameter-efficient DINOv2 architecture. LoRA is applied to the Query and Value projections of each transformer block (rank r = 64, alpha α = 64), plus a single linear layer for patch-level classification at 14 × 14 patch resolution, reducing trainable parameters from 86M to 2.4M.

4. A sliding-window inference scheme for arbitrary image sizes. Because DINOv2 accepts only fixed-size inputs, the method uses 504 × 504 windows with stride s = 128 and accumulates per-pixel logits, producing dense localization maps rather than a single global decision.

Main Findings

• Best average localization performance. DinoLizer reaches an average IoU of 0.55 and F1 of 0.64 across the evaluated benchmarks, versus 0.34 / 0.41 for TruFor, 0.27 / 0.33 for DINOv3-IML, 0.24 / 0.30 for SAFIRE, 0.13 / 0.16 for SparseViT, 0.12 / 0.17 for ManTra-Net, and 0.09 / 0.14 for CAT-Net v2.

• Reported margin over the second-best model. The abstract states DinoLizer achieves a 20% higher Intersection over Union score on average compared to the second best model; the results section states the approach outperforms the other detectors by at least 20% in F1.

• Strong on CocoGlide and FOSSIL-SP. DinoLizer records IoU / F1 of 0.74 / 0.82 on CocoGlide and 0.66 / 0.72 on FOSSIL-SP, the two strongest results in its table.

• One dataset where a baseline wins. DinoLizer attains the highest F1 on all datasets except TGIF, where TruFor surpasses it (TruFor 0.70 IoU / 0.80 F1 vs. DinoLizer 0.60 / 0.68 on TGIF).

• Robustness to regenerated backgrounds. TruFor performs well on FOSSIL-SP but its accuracy drops significantly on FOSSIL-FR (background also regenerated), while DinoLizer is more robust to both situations.

• JPEG (double) compression stability. DinoLizer maintains a consistent F1 score from QF = 100 down to QF = 40, and also under double JPEG compression (first pass QF₁ = 90, second pass QF₂ ∈ {90, 60, 40}).

• Gaussian noise sensitivity. The model remains the top-performing detector across all tested noise levels (σ² ∈ {3, 7, 15, 19}), but the paper reports notable performance drops and identifies this sensitivity as an area for future investigation.

• One resizing weakness. In up/downsampling, DinoLizer keeps a substantial lead over most baselines except at 50% downsampling, where it slightly underperforms DINOv3-IML.

• LoRA beats added transformer blocks. Training two extra transformer blocks (14.2M trainable parameters) on a frozen backbone yields 0.40 IoU on the B-Free pretrain with 2 classes, whereas LoRA yields 0.51 IoU in the 2-class setting and 0.55 in the 3-class setting — a 6% IoU gain for the 3-class setting and 11% for the 2-class setting, with fewer parameters.

• Three classes beat two. With LoRA and B-Free pretraining: 2 classes gives 0.51 IoU / 0.60 F1, 3 classes gives 0.55 IoU / 0.64 F1. The same direction holds for the frozen two-transformer-block setup (0.40 / 0.50 vs. 0.49 / 0.58).

• Backbone pretraining effect is small for LoRA. Fine-tuning DINOv2 on B-Free gives almost negligible improvement for the primary LoRA configuration (LVD-142M LoRA 3-class: 0.54 / 0.62 vs. B-Free LoRA 3-class: 0.55 / 0.64), but a more noticeable gain when using only two transformer blocks (0.44 / 0.54 vs. 0.49 / 0.58).

Methodology in Plain English

Inpainting models work in three stages: a VAE encoder compresses the image into a latent representation, the inpainting is performed in that latent space, and a VAE decoder converts it back to pixels. Everything inside the mask therefore carries both synthesis artifacts and reconstruction artifacts, while everything outside the mask carries only reconstruction artifacts. Standard training calls those outside regions "real," which teaches the model two contradictory things about the same VAE signature. DinoLizer splits the problem into three labels instead: real, VAE, and inpainted, and trains with a weighted cross-entropy loss whose class weights are w₀ = 1, w₁ = min(N₀/N₁, τ), w₂ = min(N₀/N₂, τ) with τ = 50.

The backbone is a ViT-B/14 with registers, refined on the B-Free dataset. Rather than retraining all 86M parameters, the authors freeze most of the network and train only LoRA adapters on the Query and Value projections plus one linear layer that turns each patch token into a class probability via softmax, giving 2.4M trainable parameters. Training uses AdamW with batch size 64, initial learning rate 10⁻³, weight decay 10⁻³, an lrdrop scheduler that halves the rate when validation loss stalls for four epochs, up to 100 epochs on a single NVIDIA A6000 GPU (48GB), with early stopping once the learning rate falls below 10⁻⁷. Training data come from the B-Free dataset (51k real and 309k synthetic images) plus a subset of TGIF (118k synthetic images); augmentation is applied with Albumentations on 504 × 504 images using Lanczos interpolation and includes flips, 90° rotations, crops and resizing, box blur, Gaussian noise, color jitter, and two-stage JPEG compression.

At inference, the fixed-size constraint of DINOv2 is handled by a sliding window of 504 × 504 with stride 128, summing logits over all windows covering each pixel. Small test images are upscaled to at least 1016 × 1016 pixels with bicubic interpolation to guarantee a 5 × 5 grid of windows, and the resulting masks are resized back to the original dimensions. The authors note that standard DINOv2-B averages probabilities over five global crops for a single binary decision, which fails when the inpainted region is small; the sliding window instead yields a dense per-pixel map.

Dataset construction for FOSSIL has two stages: segmentation and masking with YOLOv11, where the mask whose area ratio r is closest to a sample r ~ U(0,1) is selected to vary mask sizes; and inpainting and synthesis, where objects are removed or replaced with semantically similar counterparts. 50% of the images retain their original pristine backgrounds.

Why This Matters

Impact on research. The paper argues that treating VAE reconstruction as "real" is an actively harmful training signal, and shows a measurable gain from correcting it. It also supplies a benchmark (FOSSIL) built on newer generators than B-Free, giving the field a harder test bed as inpainting quality improves. The finding that LoRA on Query and Value outperforms adding whole transformer blocks is relevant beyond forensics, since it suggests forensic features are recoverable with minimal trainable capacity.

Real-world applications.

• Newsroom and fact-checking verification, where an editor needs to know which part of a photo was altered.
• Legal and insurance investigations involving disputed image evidence, including cases where the background was fully regenerated (the FOSSIL-FR scenario).
• Platform content moderation pipelines that must flag synthetically inserted or removed subjects in user uploads.
• Image provenance and authentication tooling, complementing metadata-based C2PA-style solutions with pixel-level proof.

Industry relevance. The single-GPU trainability and the small number of trainable parameters make the approach attractive to organizations without large compute budgets. Its stability from JPEG QF = 100 to QF = 40 matters because images circulating on social platforms are nearly always recompressed, and double compression is common. The remaining weakness under Gaussian noise is a practical limitation for pipelines where images are transmitted through noisy channels.

Future Directions

• Reducing the sensitivity to additive Gaussian noise, which the authors explicitly flag as important future work given the notable performance drops observed across σ² ∈ {3, 7, 15, 19}.
• Addressing the 50% downsampling case, where DINOv3-IML slightly outperforms DinoLizer, and the TGIF case where TruFor surpasses it.
• Extending FOSSIL and the three-class principle to new generators as they appear, since the dataset's stated motivation is that the inpainting landscape evolves faster than existing benchmarks.
• Investigating why backbone pretraining on B-Free gives only negligible gains for the LoRA configuration while giving a more noticeable gain for the two-transformer-block configuration — a question the ablation leaves open.

Target Audience

Researchers and graduate students in multimedia forensics and computer vision who work on forgery localization or synthetic image detection; engineers building image-authenticity or content-moderation systems who need a parameter-efficient detector that survives JPEG recompression; and dataset/benchmark designers interested in how to construct evaluation sets that stay representative as generative models change. Readers without background in Vision Transformers and diffusion pipelines will find the methodology sections dense, though the three-class intuition illustrated in Figure 1 is accessible on its own.

Authors’ abstract

We introduce DinoLizer, a DINOv2-based localizer of manipulated areas in generative inpainting. The model is trained to focus on semantically altered regions by treating reconstructed areas outside the inpainted mask as a separate class, which yields significant improvements w.r.t. the conven- tional approach. We train the model with LORA on the Query and Value of the transformer blocks and simply add 1 linear layer on top of the backbone to predict manipulations on a 14 x 14 patch resolution. Because DINOv2 only accepts fixed- sized images, we use a sliding window approach to aggregate the predictions on larger images. Empirical results show that DinoLizer outperforms state-of-the-art methods on our proposed dataset and SOTA inpainting datasets. Furthermore, it is very robust to JPEG (double) compression. On average, DinoLizer achieves a 20% higher Intersection over Union score compared to the second best model. The code is publicly available here: https://github.com/anonyme610/dinolizer.