FSOD-VFM: Few-Shot Object Detection with Vision Foundation Models and Graph Diffusion

AI summary

Overview

• Research area: Computer vision — few-shot object detection (FSOD), specifically training-free detection built on vision foundation models.
• Technical level: Intermediate. Readers should be comfortable with object detection metrics (AP, nAP, IoU), proposal-based detection pipelines, and the general capabilities of models such as SAM2, DINOv2, and UPN.
• Scope: The paper proposes a training-free few-shot object detector that combines three vision foundation models and adds a graph-diffusion step to suppress over-fragmented bounding boxes, evaluated on Pascal-5^i, COCO-20^i, and CD-FSOD.

What This Paper Is About

Few-shot object detection aims to detect new object categories given only a handful of labeled examples (for example, K bounding boxes per class). Modern vision foundation models can generate bounding boxes and masks without task-specific training, but when combined naively the resulting proposals are often "over-fragmented" — they latch onto small salient parts of an object instead of the whole object, producing many small false positives. This paper's goal is to fix that fragmentation problem using a graph-based score refinement step, so that a pipeline of foundation models becomes a competitive few-shot detector without any additional training.

Key Contributions

1. FSOD-VFM framework: A few-shot object detector that combines a universal proposal network (UPN) for category-agnostic bounding box generation, SAM2 for mask extraction, and DINOv2 features for matching to support prototypes — operating entirely without additional training.
2. Graph diffusion confidence reweighting: A mechanism that models predicted bounding boxes as nodes in a directed graph and propagates confidence scores through it, elevating whole-object proposals and suppressing local, fragmented parts.
3. Comprehensive benchmark evaluation: Experiments on Pascal-5^i, COCO-20^i, and CD-FSOD showing consistent improvements over prior approaches in the training-free setting, plus ablations on post-processing alternatives, diffusion hyperparameters, and backbone choice.
4. Public code release: Code available at a project page hosted by the Intellindust AI Lab.

Main Findings

• Pascal-5^i gains: FSOD-VFM reaches an average of 77.5 nAP50 across the three novel splits of Pascal-5^i, compared with 71.2 for the prior training-free method No-Time-To-Train and 61.4 for DE-ViT (ViT-B/14). Per-split averages: Split 1 = 77.5, Split 2 = 77.5, Split 3 = 77.5.
• COCO-20^i gains: Under the 10-shot setting, FSOD-VFM achieves 44.0 nAP, 59.4 nAP50, and 47.6 nAP75, versus 36.6 / 54.1 / 38.3 for No-Time-To-Train. Under 30-shot, it reaches 45.8 nAP, 61.9 nAP50, and 49.4 nAP75 versus 36.8 / 54.5 / 38.7.
• CD-FSOD gains: On CD-FSOD, which integrates six datasets from diverse domains, FSOD-VFM averages 25.3 / 31.3 / 31.6 nAP for 1-shot / 5-shot / 10-shot, compared with 18.0 / 22.4 / 21.4 for No-Time-To-Train. The abstract highlights 31.6 AP at 10-shot versus 21.4 AP for prior training-free methods.
• Domain-dependent behavior: FSOD-VFM is strongest on ArTaxOr (51.4 / 62.0 / 61.5) and Clip art1k (29.1 / 43.7 / 46.5), domains close to natural images. It is weaker on NEU-DET (5.9 / 7.4 / 7.2) and UODD (11.8 / 17.3 / 17.5) — datasets involving low-saliency defects in microscopic textures or blurred underwater targets — where fine-tuned methods do better.
• Competitive against fine-tuned methods: The paper reports that FSOD-VFM remains competitive with methods fine-tuned on novel categories, especially in the one-shot setting. Mixture achieves the best overall CD-FSOD average (18.2 / 34.7 / 39.8), but FSOD-VFM retains an advantage over Mixture in the one-shot case.
• Graph diffusion beats other post-processing: On Pascal-5^i (1-shot) and COCO-20^i (10-shot), FSOD-VFM's graph diffusion yields 77.5 and 59.4 nAP50, compared with 66.0 / 50.4 for Soft Merging, 28.1 / 26.6 for Soft NMS, 25.5 / 27.8 for WBF, and 23.4 / 26.1 for NMS. Without refinement (λ = 0) the results drop to 7.4 and 9.9.
• NMS is unnecessary: Adding NMS on top of graph diffusion slightly reduces performance (77.2 / 59.1), indicating the reweighting pipeline does not need it.
• Low hyperparameter sensitivity: Performance is relatively stable across λ, α, and diffusion steps. The authors adopt λ = 0.5, α = 0.3, and 30 diffusion steps; performance improves with more steps and stabilizes after the fifth step, with full convergence around 70 steps.
• Backbone ablation: DINOv2-B achieves the best Pascal-5^i split1 (1-shot) result at 81.2, but DINOv2-L, the chosen backbone, is stronger overall (77.5 on Pascal-5^i, 59.4 on COCO-20^i). Register variants and DINOv3 variants generally score lower in this pipeline.
• Efficiency: Inference on a single image takes approximately 2.4 seconds on a single NVIDIA A40 GPU without additional optimization.

Methodology in Plain English

The pipeline has three stages, all without training.

1. Build class prototypes from the support set. For each of the K labeled examples per class, SAM2 produces a binary object mask from the given bounding box, and DINOv2 produces a dense image feature map. The mask is downsampled to the feature map resolution and used to pool only the features inside the object region. Features belonging to the same class are averaged and L2-normalized to form a single prototype vector per class.

2. Match query proposals. For a query image, UPN generates category-agnostic bounding boxes with confidence scores. Each proposal gets a mask from SAM2 and a feature vector from DINOv2, and is assigned the class whose prototype has the highest cosine similarity with that feature. Each proposal is stored as a quadruple: feature, predicted class, UPN score, and mask.

3. Refine scores with graph diffusion. Within each predicted class, proposals become nodes in a directed graph. An edge from node i to node j is set to zero if i's UPN score is higher than j's; otherwise the edge weight is the fraction of i's mask area that overlaps j's mask. Nodes with high UPN scores keep their energy; lower-scoring nodes pass energy toward more confident, overlapping nodes. This mirrors the PageRank scheme: a prior weight per node is set to its strongest outgoing edge, the transition matrix is row-normalized, and an iterative update mixes the propagated vector with the prior, controlled by a restart probability α. The process stops early when the change between iterations falls below τ = 1e-6. The final score multiplies the proposal's best cosine similarity by (1 − π̂_j)^λ, so proposals that absorb a lot of diffusion mass are downweighted.

In practice, masks are used as the relationship measure instead of bounding-box IoU, which the authors argue captures object boundaries far more accurately than boxes and avoids the hard on/off decisions of NMS.

Implementation specifics: SAM2-L (Hierarchical ViT) for masks, DINOv2-L for features, images resized to 630 × 630 (matching DINOv2's patch size of 14); DINOv3 is also tested at 624 × 624 with patch size 16. UPN is prompted with "coarse," boxes below 0.01 confidence are discarded, at most 500 proposals per image are kept, and the top 100 boxes per image are output.

Why This Matters

Impact on research. The paper pushes the frontier of training-free few-shot detection. It shows that a carefully designed refinement stage — rather than a better backbone or more training — can substantially improve how well foundation models work together. It also offers a principled alternative to NMS and related post-processing, replacing threshold-based suppression with score propagation over mask-level relations.

Real-world applications.

• Autonomous driving: Detecting rare or newly introduced object categories (unusual vehicles, atypical obstacles) when labeled data for those categories is scarce.
• Robotics: On-the-fly recognition of novel objects in unstructured environments where collecting and annotating training data is impractical.
• Medical imaging: Locating rare pathologies or uncommon anatomical structures where only a few expert-annotated examples exist.
• Industrial and remote-sensing inspection: Defect or target detection where the paper reports domain-specific benchmarks (NEU-DET, UODD, DIOR, Deep Fish), while also showing that fine-tuning still wins in these low-saliency domains.

Industry relevance. The approach requires no training loop and no target-domain labels beyond the few support annotations, which lowers the engineering cost of deploying a detector to a new category. The trade-off is throughput: around 2.4 seconds per image on an A40, which the authors explicitly note is not optimized for real-time use.

Future Directions

• Closing the domain gap. Fine-tuned methods still outperform FSOD-VFM on domain-specific datasets such as NEU-DET, UODD, and Deep Fish. Finding ways to make training-free detection robust to microscopic-texture and underwater imagery is an open problem.
• Reducing inference cost. The method relies on multiple foundation models, and inference is roughly 2.4 seconds per image. Distillation, shared backbones, or caching of prototype features are natural next steps.
• Improving the marginal returns of more shots. The paper notes that the gain from 5-shot to 10-shot is marginal compared with 1-shot to 5-shot; understanding and fixing this saturation is an unresolved question.
• Extending the graph formulation. The graph currently operates per class within a single query image. Scaling it to cross-image or cross-class relations, and testing alternative diffusion operators or learned edge weights, are unexplored directions.

Target Audience

• Few-shot detection researchers looking for a training-free baseline and a new post-processing idea.
• Practitioners applying vision foundation models who need to adapt a detector to new categories with minimal labels and no training infrastructure.
• Researchers working on foundation-model composition, interested in how UPN, SAM2, and DINOv2 can be assembled and where that assembly fails.
• Graduate students and engineers with intermediate computer vision background who want a concrete case study in replacing a hand-crafted heuristic (NMS) with a graph-based propagation scheme.

Note: details such as dataset descriptions are deferred to the paper's appendix (Section A.3), and reproducibility information — including hyperparameters, the algorithm listing in Appendix A.1, and public dataset URLs — is provided in the reproducibility statement.

Authors’ abstract

In this paper, we present FSOD-VFM: Few-Shot Object Detectors with Vision Foundation Models, a framework that leverages vision foundation models to tackle the challenge of few-shot object detection. FSOD-VFM integrates three key components: a universal proposal network (UPN) for category-agnostic bounding box generation, SAM2 for accurate mask extraction, and DINOv2 features for efficient adaptation to new object categories. Despite the strong generalization capabilities of foundation models, the bounding boxes generated by UPN often suffer from overfragmentation, covering only partial object regions and leading to numerous small, false-positive proposals rather than accurate, complete object detections. To address this issue, we introduce a novel graph-based confidence reweighting method. In our approach, predicted bounding boxes are modeled as nodes in a directed graph, with graph diffusion operations applied to propagate confidence scores across the network. This reweighting process refines the scores of proposals, assigning higher confidence to whole objects and lower confidence to local, fragmented parts. This strategy improves detection granularity and effectively reduces the occurrence of false-positive bounding box proposals. Through extensive experiments on Pascal-5$^i$, COCO-20$^i$, and CD-FSOD datasets, we demonstrate that our method substantially outperforms existing approaches, achieving superior performance without requiring additional training. Notably, on the challenging CD-FSOD dataset, which spans multiple datasets and domains, our FSOD-VFM achieves 31.6 AP in the 10-shot setting, substantially outperforming previous training-free methods that reach only 21.4 AP. Code is available at: https://intellindust-ai-lab.github.io/projects/FSOD-VFM.