Test-Time Adaptive Object Detection with Foundation Model

AI summary

Overview

Research area: Computer vision — test-time adaptive object detection (TTAOD), vision-language foundation models, parameter-efficient fine-tuning.

Technical level: Intermediate. Readers should be comfortable with object detection pipelines, mean-teacher self-training, and prompt tuning, but the paper explains its components in a self-contained way.

Scope: The paper proposes the first vision-language foundation model-powered test-time adaptive object detector, which adapts during inference without any source data and without a closed-set category assumption, evaluated on cross-corruption (Pascal-C, COCO-C) and cross-dataset (ODinW-13) benchmarks.

What This Paper Is About

Existing test-time adaptive object detection methods adapt a detector online to a new target domain, but they depend on statistical characteristics (such as the mean and variance of feature maps) computed from source-domain data, and they assume the source and target domains share exactly the same category space. This paper removes both constraints: it builds adaptation on the vision-language detector Grounding DINO, tuning only small prompts on test data, so no source data is needed and the detector can adapt to arbitrary cross-domain and cross-category target data.

Key Contributions

1. The first foundation model-powered test-time adaptive object detector, which requires no source data at all and overcomes the traditional closed-set limitation of TTAOD.
2. A Multi-modal Prompt-based Mean-Teacher framework for vision-language detectors, combining text prompt tuning and visual prompt tuning with a Test-time Warm-start strategy, so that only prompts are fine-tuned while the pre-trained detector stays largely frozen.
3. An Instance Dynamic Memory (IDM) module that stores high-quality pseudo-labels from previous test samples, together with two new strategies — Memory Enhancement (refining current predictions using memory prototypes) and Memory Hallucination (mixing stored instances into test images that have no usable pseudo-labels).
4. Extensive experiments on cross-corruption and cross-dataset benchmarks showing consistent improvements over prior state-of-the-art TTAOD methods and successful adaptation to differing category spaces.

Main Findings

• Pascal-C gains: On Pascal-C (constructed from the Pascal VOC test set with 15 corruption types, each corrupted test set containing 4,956 images over 20 classes), the method reaches an average AP50 of 56.2%, versus 51.5% for a Mean-Teacher baseline built on Grounding DINO and 44.8% for Direct Test. The authors report it outperforms the previous state-of-the-art STFAR by 11.0%.
• Corruption-type robustness: Prior approaches BN, TENT and T3A actually hurt performance relative to direct testing with Faster R-CNN on Pascal-C, whereas self-training based methods (SHOT, Mean-Teacher, STFAR) improve it. Simply applying self-training to Grounding DINO gives a 6.7% average AP50 improvement over direct testing.
• COCO-C results: On COCO-C (built from the COCO val2017 set of 5k images covering 80 categories), the method achieves the highest average mAP of 26.0% without using any source-domain data, and attains state-of-the-art performance on 8 out of 15 corruption types.
• Cross-dataset results: On the 13 ODinW-13 datasets, the method reaches an average mAP of 54.2%, a 1.4% improvement over Direct Test's 52.8%, while Mean-Teacher with Grounding DINO yields only a 0.3% gain. Improvements appear on almost all sub-datasets except Mu and Pa, which have extremely few test samples (5 for Mu, 4 for Pa).
• Text prompts alone are weak: Using only Text Prompt Tuning improves average AP50 by just 0.6% over direct testing (45.4%).
• Visual prompts alone can fail: Applying Visual Prompt Tuning alone drops performance by 3.4% (41.4%), and on some corruptions such as Gaussian noise and Pixelate it causes catastrophic failure; the Test-time Warm-start strategy addresses this.
• Components are complementary: Adding both prompt types with warm-start reaches 53.4%; Memory Enhancement (a training-free strategy) brings it to 53.9%; adding Memory Hallucination to the Mean-Teacher framework adds a further 1.0% (53.9% to 54.9%); using all components gives 56.2%.
• Number of visual prompts: Sweeping the number of visual prompts m on Gaussian noise corruption of Pascal-C gives AP50 of 42.6 (2), 44.0 (4), 45.0 (6), 45.3 (8), 45.4 (10), 45.2 (15), 45.3 (20), 45.1 (30), 44.2 (50); performance declines once m exceeds 30, and m = 10 is chosen as the balance point.
• IDM capacity matters both ways: Too small a capacity prevents Memory Enhancement from building representative category prototypes; too large a capacity risks admitting noisy pseudo-labels. A capacity of 20 is used on Pascal-C (and 3 on the cross-dataset benchmark).
• Efficiency: Compared with Full Fine-tuning, the framework requires only 0.05% of the learnable parameters (0.079M versus 164.964M). It also has lower per-image latency (582.9 ms versus 635.0 ms) and peak GPU memory (18.0 GB versus 20.9 GB), while achieving higher average AP50 (53.9% for the full Multi-modal Prompt-based Mean-Teacher framework under that measurement versus 51.5% for Full Fine-tuning).
• Why Memory Enhancement helps: Plotting True Positive and False Positive distributions for the category "bicycle" under Gaussian noise on Pascal-C shows Memory Enhancement reorders predictions so that true positives rank above false positives.
• Memory Enhancement threshold: Because Grounding DINO produces 300 predictions per image, applying enhancement to all of them slows inference and can hurt accuracy; below a threshold of 0.2 the latency rises notably and performance declines relative to direct testing, so 0.3 is chosen.
• Not reported: The provided content is truncated at the start of Table 8 ("Sensitivity an..."), so any further sensitivity analysis results are not available in this text.

Methodology in Plain English

The approach starts from Grounding DINO, a detector that links images and text, pre-trained on Objects365, GoldG and Cap4M, using Swin-Tiny as the visual backbone with DINOv2 (ViT-L) as a separate feature extractor inside the memory module.

Rather than fine-tuning the whole detector on test data — which the authors say harms generalization and overfits when target data is scarce — the method adds small learnable prompt vectors in two places. In the language branch, a learnable vector is added to the text tokens (with the class names joined into a text string, mapped to at most 256 tokens), initialized at zero. In the vision branch, learnable tokens are inserted alongside the image tokens at each transformer layer. To avoid breaking the vision branch, a Test-time Warm-start initializes these visual prompts by average-pooling the image tokens from the very first test image.

These prompts are trained in a mean-teacher setup: a teacher with one set of prompts produces pseudo-labels on weakly augmented test data, and a student with another set of prompts learns from strongly augmented data, with a classification-score threshold filtering noisy pseudo-labels. The teacher prompts are updated by exponential moving average of the student prompts. Only prompts are optimized; the rest of the detector stays frozen.

To keep pseudo-label quality high across batches, an Instance Dynamic Memory keeps a queue per category. A high-scoring detection is cropped, encoded with DINOv2, and stored as an (image, feature, score) triplet; if the queue is full, the new instance replaces the lowest-scoring one only if its score is higher. Memory Enhancement then computes a class prototype by averaging the queue contents and adds a memory-based affinity score to the detector's high-confidence predictions, refining the ranking. Memory Hallucination handles test images that produce no usable pseudo-labels: it pastes up to three stored high-quality instances at random positions, with random scaling and an IoU check (retrying up to 10 times on overlap), so that even these "negative" images contribute to adaptation.

Why This Matters

Impact on research: The work moves TTAOD away from source-data dependence and closed-set assumptions, which the authors argue had limited the practical applicability of the task. It also shows that parameter-efficient prompt tuning — not full fine-tuning — is the viable route for adapting vision-language detectors online, and it adds a memory mechanism to keep pseudo-label quality stable across a test stream. The paper mentions that storing only prompts per target domain while sharing one copy of the pre-trained weights also makes extension to Continual Test-time Adaptive Object Detection easier. Code is released at https://github.com/gaoyingjay/ttaod_foundation.

Real-world applications (the first two are the scenarios named in the paper's introduction; the rest correspond to the ODinW-13 domains the method is tested on):

• Autonomous driving, where weather, sensor noise and lighting shift continuously at inference time.
• Robotics, where a deployed vision system must adapt online without a labeled source set.
• Aerial and maritime drone imagery, pothole and road inspection, and vehicle monitoring (Ae, Po, Ve in the benchmark).
• Specialized sensing such as thermal imaging of dogs and people, wildlife and livestock monitoring (raccoons, cottontail rabbits, shellfish, aquarium imagery), and warehouse package handling.

Industry relevance: Because the method needs no source data, an organization can adapt a deployed detector to a new site or sensor without shipping proprietary training data. Requiring roughly 0.05% of the parameters of full fine-tuning and storing only prompts per domain reduces memory, latency and storage costs, and the open-vocabulary property means the deployed model is not locked to a fixed category list.

Future Directions

• Extending the prompt-only adaptation scheme to Continual Test-time Adaptive Object Detection, which the authors note is facilitated by storing per-domain prompts over shared pre-trained weights.
• Handling target domains with extremely few test samples, since the method failed to improve on Mu (5 test samples) and Pa (4 test samples), preventing the detector from adapting adequately.
• Tuning the memory mechanism further — capacity, prototype construction and how to avoid admitting noisy pseudo-labels at large capacities remain open trade-offs between detection performance and storage cost.
• Completing the sensitivity analysis that appears to begin at Table 8 in the paper, plus broader study of threshold choices (the pseudo-label threshold, the Memory Enhancement threshold and the IoU threshold) across benchmarks, and generalization beyond the two benchmark families evaluated here.

Target Audience

Researchers and engineers working on test-time adaptation, unsupervised domain adaptation, and object detection; practitioners deploying detectors to new domains without labeled source data; and readers interested in parameter-efficient adaptation of vision-language foundation models. Some familiarity with detection architectures and self-training is helpful, but the individual mechanisms are described concretely enough for newcomers to follow.

Authors’ abstract

In recent years, test-time adaptive object detection has attracted increasing attention due to its unique advantages in online domain adaptation, which aligns more closely with real-world application scenarios. However, existing approaches heavily rely on source-derived statistical characteristics while making the strong assumption that the source and target domains share an identical category space. In this paper, we propose the first foundation model-powered test-time adaptive object detection method that eliminates the need for source data entirely and overcomes traditional closed-set limitations. Specifically, we design a Multi-modal Prompt-based Mean-Teacher framework for vision-language detector-driven test-time adaptation, which incorporates text and visual prompt tuning to adapt both language and vision representation spaces on the test data in a parameter-efficient manner. Correspondingly, we propose a Test-time Warm-start strategy tailored for the visual prompts to effectively preserve the representation capability of the vision branch. Furthermore, to guarantee high-quality pseudo-labels in every test batch, we maintain an Instance Dynamic Memory (IDM) module that stores high-quality pseudo-labels from previous test samples, and propose two novel strategies-Memory Enhancement and Memory Hallucination-to leverage IDM's high-quality instances for enhancing original predictions and hallucinating images without available pseudo-labels, respectively. Extensive experiments on cross-corruption and cross-dataset benchmarks demonstrate that our method consistently outperforms previous state-of-the-art methods, and can adapt to arbitrary cross-domain and cross-category target data. Code is available at https://github.com/gaoyingjay/ttaod_foundation.