DINO-FIL: Multi-Layer Transformer Fusion for Generative Inpainting Forgery Localization

AI summary

Overview

Research area: Computer vision / image forensics — specifically pixel-level localization of regions created by generative inpainting models. Technical level: Advanced (assumes familiarity with Vision Transformers, dense prediction, and segmentation metrics). Scope: The paper introduces DINO-FIL, a framework that fuses intermediate hidden states from a frozen-or-partially-fine-tuned DINOv3 ViT-H+/16 encoder with a lightweight top-down decoder, and evaluates it cross-dataset against DinoLizer and TruFor on five inpainting benchmarks.

What This Paper Is About

When someone uses a generative inpainting tool to remove or replace part of a photo, the edit can be visually seamless because the new content is synthesized together with its surroundings, leaving few of the low-level discontinuities that older forgery detectors look for. This paper's goal is to produce a pixel-level mask that marks exactly which regions of an image were generated. The authors argue that existing DINO-based detectors use only the final transformer layer, and that fusing features from several depths of the network better captures the subtle texture and boundary artifacts left by inpainting.

Key Contributions

1. DINO-FIL architecture: A dense inpainting localization model that taps hidden states from layers 8, 16, 24, and 32 of a 32-layer pretrained DINOv3 ViT-H+/16 encoder and fuses them, instead of relying only on the final layer.
2. Depth-as-hierarchy fusion: Because all selected layers share the same patch grid (patch size P = 16), the decoder treats transformer depth rather than spatial scale as the fusion hierarchy, merging deepest to shallowest in an FPN-inspired top-down pass with ASPP-inspired multi-scale context blocks.
3. Encoder adaptation study: A systematic ablation over 0, 6, 12, 16, 24, and 32 unfrozen transformer layers, combined with discriminative learning rates (5×10⁻⁶ for unfrozen encoder layers vs. 1×10⁻⁴ for the decoder), showing that partial fine-tuning of later layers suffices and full fine-tuning gives diminishing returns.
4. Cross-dataset benchmark and resolution analysis: Evaluation against DinoLizer and TruFor on five datasets, plus evidence that 1024×1024 inference substantially improves localization while 1024×1024 training does not, alongside a false-positive analysis on 9,081 pristine images.

Main Findings

• Best average result: The variant with the final 24 DINOv3 layers unfrozen, evaluated at 1024×1024, reaches an average IoU/F1 of 0.71/0.77 across the five datasets (training resolution fixed at 512px).
• Beats both baselines on average: DinoLizer scores 0.49/0.61 and TruFor 0.36/0.43 on the same average; DINO-FIL is best on four of the five datasets (Beyond the Brush, CocoGlide, SAGI-D, B-Free).
• Largest margins on CocoGlide and SAGI-D: CocoGlide improves from DinoLizer's 0.57/0.69 to 0.87/0.92; SAGI-D from 0.38/0.49 to 0.66/0.73.
• One loss — TGIF: TruFor is strongest on TGIF with 0.70/0.80, while DINO-FIL obtains 0.57/0.63 and is second best.
• Freezing the encoder hurts: With 0 unfrozen layers, DINO-FIL averages 0.51/0.58 at 512×512 and 0.58/0.67 at 1024×1024, below all partially fine-tuned variants.
• Diminishing returns from full fine-tuning: At 1024×1024, unfreezing up to 24 layers steadily improves results; fully unfreezing 32 layers leaves the average essentially unchanged and slightly lowers average F1 before rounding, from 0.774 to 0.771.
• High-resolution inference is the biggest lever: Comparing best configurations per resolution, average IoU/F1 rises from 0.61/0.67 at 512×512 to 0.71/0.77 at 1024×1024. Beyond the Brush goes from 0.36/0.44 to 0.59/0.68, TGIF from 0.38/0.45 to 0.58/0.64, SAGI-D from 0.59/0.67 to 0.66/0.73.
• Two datasets do not benefit: CocoGlide (fixed 256×256 images) gains only 0.87/0.92 → 0.88/0.93, which the authors attribute cautiously to interpolation or random variation; B-Free (fixed 512×512, and the training set) is slightly better at 512×512 than at 1024×1024.
• High-resolution training does not help: In a separate SAGI-D-trained experiment, a 512-trained model evaluated at 1024 reaches 0.71/0.78 average, while a 1024-trained model evaluated at 1024 reaches 0.68/0.74. Only SAGI-D IoU ticks up, from 0.83 to 0.84, with F1 unchanged at 0.88.
• Conservative on pristine images: Across 9,081 non-manipulated COCO, Open Images, and RAISE images, the model records a pixel-level false-positive rate of 0.0032 and an image-level true-negative rate of 0.9719 (COCO 0.9767, Open Images 0.9684, RAISE 0.9810).

Methodology in Plain English

Images are resized to a square (512×512 for training), normalized, and passed through a pretrained DINOv3 ViT-H+/16 encoder. Rather than reading only the encoder's last output, the authors pull the internal representations at four evenly spaced depths — layers 8, 16, 24, and 32 — so the model sees early, detail-oriented features alongside later, more semantic ones. Each of these is reshaped into a spatial map by throwing away the one CLS token and four register tokens, leaving a grid of patch tokens, and then squeezed from the backbone's 1280 channels down to 1024 by a small projection block (1×1 convolution, group normalization, GELU).

Each projected map is then enriched with three parallel depthwise-separable 3×3 convolutions at dilation rates 1, 2, and 4, summed and fused by a 1×1 convolution — a lightweight version of atrous spatial pyramid pooling meant to catch artifacts at multiple spatial scales. Optional spatial dropout (rate 0.05) regularizes these maps. A top-down decoder then merges them from deepest (layer 32) to shallowest (layer 8) using learned 3×3 convolutions, accumulating each merged result into the next stage; because every map is already on the same patch grid, no resizing is needed between stages. A 1×1 head produces patch-level logits, which are bilinearly upsampled to the input size and passed through a sigmoid to give an inpainting probability mask.

Training uses binary cross-entropy plus Dice loss, batch size 12, a 0.5 threshold at test time, and augmentation including horizontal flipping, color jitter, and B-Free-style perturbations (blur, JPEG recompression, down/up-scaling, mild noise). The encoder ablation trains only two epochs on B-Free because of the dataset's size, and the training-resolution experiment likewise trains two epochs on SAGI-D. Evaluation reports per-image IoU and F1, macro-averaged so each manipulated image counts equally.

Why This Matters

This work pushes inpainting forensics away from "look at the last layer of a big model" toward structured use of a model's internal depth, and it isolates a practical finding — that test-time resolution matters more than training-time resolution for this task — that other groups can adopt cheaply without retraining. It also provides a cleaner baseline comparison against DinoLizer and TruFor on a shared five-dataset protocol (28,835 test images in total: 10,000 B-Free, 3,896 Beyond the Brush, 512 CocoGlide, 4,249 TGIF, 10,178 SAGI-D).

Real-world applications:

• Media verification and fact-checking: flagging which pixels of a published image were synthesized so that editors and readers can see the extent of an edit.
• Platform content moderation: screening user uploads for generative edits that alter people, objects, or scenes.
• Digital evidence handling: documenting whether a submitted photograph contains locally regenerated regions.
• Dataset and model auditing: checking whether training corpora contain inpainted content, or evaluating how traceable a given generative editing tool's outputs are.

Industry relevance: the method's decoder is lightweight and reuses a pretrained backbone, and the paper's finding that partial fine-tuning (24 of 32 layers) is enough — plus the observation that pixel-level false positives stay near 0.003 — are directly relevant to teams that need deployable detectors with low false-alarm rates rather than research-only accuracy.

Future Directions

• Adaptive-resolution training and evaluation: group images by native resolution and only process at higher targets when real high-frequency evidence exists, choosing targets as multiples of 16 to match the DINOv3 patch size and avoid implicit cropping.
• Parameter-efficient adaptation: replacing partial or full fine-tuning with LoRA, which the authors note Yu et al. explored for DINO-based forensic backbones and which may improve efficiency and reduce overfitting.
• Understanding the resolution paradox: the paper explicitly leaves open why 1024×1024 training underperforms 512×512 training followed by 1024×1024 inference; it notes that only about 21% of SAGI-D images have a maximum side length of at least 1024px, which may explain the gap.
• Benchmark label consistency: the authors flag that the SAGI-SP / SAGI-FR assignment of individual generators is not consistent across sources (e.g., Inpaint-Anything's placement), which complicates direct comparison and motivates cleaner benchmark definitions.

Target Audience

Researchers and graduate students in image forensics and media authentication; engineers building content-provenance or moderation pipelines who want a strong, low-false-positive localization baseline; and practitioners working with self-supervised ViT backbones who are interested in multi-depth feature fusion or in how test resolution interacts with native image statistics. Readers need comfort with transformer internals, segmentation losses, and IoU/F1 evaluation to get the most from it; the architectural idea itself is accessible at an intermediate level.

Authors’ abstract

We propose DINO-FIL, a transformer-based framework for pixel-level localization of generative image forgeries. The method leverages multi-layer representations from a pretrained DINOv3 ViT-H+/16 encoder combined with a lightweight fusion decoder to capture both semantic structure and fine-grained texture inconsistencies introduced by image editing. By aggregating intermediate transformer features across depths, the model improves sensitivity to subtle, spatially localized artifacts that are often missed by standard single-layer approaches. We evaluate DINO-FIL in a cross-dataset setting, training on B-Free and testing on Beyond the Brush, CocoGlide, TGIF, and SAGI-D. The model achieves strong generalization, outperforming existing baselines on average and on four of five evaluated datasets, while maintaining a high true negative rate on pristine images.