TOMCAT: Test-time Comprehensive Knowledge Accumulation for Compositional Zero-Shot Learning

AI summary

Overview

Research area: Computer Vision — Compositional Zero-Shot Learning (CZSL) and test-time adaptation with vision-language models.

Technical level: Advanced (assumes familiarity with CLIP, prompt tuning, prototype-based classification, and entropy minimization).

Scope: The paper proposes TOMCAT, a test-time framework that accumulates textual and visual knowledge from unlabeled test images to update multimodal prototypes and counter label-space distribution shift in CZSL.

What This Paper Is About

Compositional Zero-Shot Learning requires a model to recognize attribute-object pairs (for example, "browned cheese") that were never seen during training, by recombining attributes and objects it already knows. Existing methods freeze their parameters and class prototypes after training, so at test time they face a mismatch between the label distribution they learned and the one they actually encounter. TOMCAT's goal is to let the model keep learning during testing, using unlabeled test images to adjust its prototypes without forgetting previously learned knowledge and without large inference latency.

Key Contributions

1. TOMCAT framework: A test-time approach that accumulates multimodal knowledge from unsupervised test data and updates prototypes to bridge label distribution shift. The authors state they are the first to leverage unsupervised data at test time to improve models in CZSL.
2. Priority queue and knowledge accumulation modules: A dynamic priority queue stores high-confidence historical images for each composition to compute visual prototypes, and knowledge accumulation modules (KAMs) adaptively update both textual and visual prototypes.
3. Adaptive update weight strategy: A weighting mechanism controls how strongly prototypes are adjusted for each incoming image, based on similarity between the image and the prototype.
4. State-of-the-art results across four benchmarks: Experiments on UT-Zappos, MIT-States, C-GQA, and Clothing16K in both closed-world and open-world settings, plus ablation studies on modules, losses, initialization, test order, and hyperparameters.

Main Findings

• Closed-world gains on three datasets: TOMCAT reaches AUC 48.3 and HM 60.2 on UT-Zappos, AUC 16.0 and HM 34.0 on C-GQA, and AUC 99.5 and HM 98.4 on Clothing16K. The paper reports improvements of 1.7% and 1.3% in AUC and 1.7% and 2.1% in HM on UT-Zappos and Clothing16K respectively, and 1.1% in AUC and 1.2% in HM on C-GQA.
• Second-best on MIT-States: TOMCAT attains AUC 22.6 and HM 39.5 on MIT-States, behind ClusPro (AUC 23.8, HM 40.7). The authors attribute this to the substantial noise in that dataset, with approximately 70% of labels being incorrect.
• Larger open-world gains: TOMCAT surpasses the second-best method by 4.2%, 1.2%, and 4.0% in AUC and 3.8%, 2.6%, and 3.0% in HM on UT-Zappos, C-GQA, and Clothing16K respectively. Open-world Clothing16K results reach AUC 95.8 and HM 95.0.
• Open-world C-GQA and UT-Zappos: AUC 4.2 / HM 14.2 on C-GQA and AUC 43.7 / HM 57.9 on UT-Zappos; MIT-States open-world reaches AUC 8.2 and HM 21.7.
• Module ablation: On MIT-States all proposed modules contribute to improvement; on UT-Zappos, adding the priority queue and visual KAM alone causes degradation relative to the base model (base: AUC 43.57, HM 55.54 on UT-Zappos, AUC 22.12, HM 38.97 on MIT-States), which the authors attribute to the uniform shoe style and subtle inter-composition differences. Combining the queue, textual KAM, visual KAM, and adaptive update weights yields the best result: AUC 48.31 and HM 60.18 on UT-Zappos, AUC 22.55 and HM 39.45 on MIT-States.
• Loss ablation: Using only the prediction entropy loss gives AUC 44.59 / HM 57.29 on UT-Zappos and AUC 22.35 / HM 39.32 on MIT-States; using only the multimodal collaborative representation learning loss gives AUC 42.46 / HM 53.97 and AUC 22.29 / HM 39.42. Combining both losses gives the best numbers.
• Zero initialization wins: Initializing KAMs to all zeros (AUC 48.31, HM 60.18 on UT-Zappos; AUC 22.55, HM 39.45 on MIT-States) outperforms uniform random, normal random, and random walking initializations, because random initialization at test time makes the model forget training knowledge.
• Test order sensitivity is small: Across three random sample orders on UT-Zappos, AUC ranges from 46.44 to 48.52 and HM from 58.72 to 60.82; the paper states the differences are not statistically significant.
• Hyperparameters: TOMCAT performs best when the priority queue holds K = 3 images on both UT-Zappos and MIT-States; a small K causes unstable visual prototypes while a large K may include low-confidence images. Performance first improves then deteriorates as the update control factor theta increases, because a small theta under-adapts and a large theta over-updates prototypes.
• Qualitative analysis: A case study on UT-Zappos and MIT-States compares TOMCAT with the trained base model, marking successes in green and failures in red. A similarity heatmap on MIT-States examines unseen compositions built from the attribute "cooked" and its objects (such as chicken and meat). The paper also states it visualizes how top-1 accuracy evolves with test sample size, but the conclusion of that analysis is not included in the provided content.

Methodology in Plain English

The approach has two stages.

Training stage. The authors take CLIP (ViT-L/14) and insert learnable adapters into the visual encoder and learnable prompt tokens for the composition labels in the text encoder, keeping the original CLIP weights frozen. They train with a standard cross-entropy loss that pulls an image's visual feature toward the text embedding of its correct attribute-object composition. This produces a simple base model.

Test stage. The textual prototypes — the embeddings of all seen and unseen composition labels — are kept from the trained text encoder. In parallel, the model maintains a small "confidence-feature" queue per composition holding the K most confident images seen so far, ranked by prediction entropy (lower entropy means higher confidence). New images are inserted if the queue is not full, or replace the least confident entry if they are more confident. Averaging the features in a queue produces a visual prototype for that composition.

To adapt to the shifted label distribution, two sets of learnable parameters called knowledge accumulation modules (KAMs), one textual and one visual, start at zero. For each incoming test image, an adaptive weight based on the cosine similarity between the image feature and each prototype decides how strongly that prototype should be nudged: images that already match a prototype strongly (likely seen compositions) are adjusted less, while images that differ greatly (possibly unseen compositions) permit stronger updates. The updated prototypes are then combined with the visual prototype scores through a Tip-Adapter-style formulation.

Test-time learning is driven by two losses: an entropy minimization loss that encourages confident predictions over the test label space, and a contrastive multimodal collaborative representation learning loss that pulls matching textual and visual prototypes closer and pushes non-matching ones apart. Importantly, the base model stays frozen and only the KAM parameters are updated. The gradient update is deferred until after the prediction for a sample is produced, to keep latency low.

Why This Matters

This work reframes CZSL as an online, continual process rather than a one-shot frozen deployment, and it does so without access to labels or to any unseen-composition data during training. It shows that visual knowledge from earlier test images can complement frozen textual prototypes, and that doing so helps most in the harder open-world setting where the search space includes all possible attribute-object pairs.

Real-world applications:

• Fashion and e-commerce retrieval: Recognizing fine-grained attribute-object combinations such as colors and garment types (the UT-Zappos and Clothing16K benchmarks are drawn from this domain), where new combinations appear continuously and catalog labels lag behind.
• Robotics and assistive systems: Describing or fetching objects by novel attribute-object descriptions that were never in the training vocabulary.
• Visual surveillance and scene understanding: Adapting to new object states in a live camera stream without re-annotation or retraining.
• Interactive user-facing search: Systems that improve as users interact with them, where low latency matters and the paper explicitly targets time efficiency.

Industry relevance: The method avoids full fine-tuning of CLIP, updates only lightweight modules, and keeps the base model frozen, which makes it attractive for deployment pipelines where a model must adjust to a customer's stream of data on modest hardware — the paper reports experiments on a single NVIDIA RTX 3090 GPU.

Future Directions

• Robustness to noisy datasets: TOMCAT falls behind ClusPro on MIT-States, which the authors link to roughly 70% label noise. Better handling of inaccurate label supervision is an open problem.
• Fine-grained composition discrimination: Visual prototypes hurt performance on UT-Zappos because of subtle inter-composition differences; improving fine-grained semantic distinction, or gating visual knowledge when it is unreliable, is a natural next step.
• Reducing order sensitivity: Results vary across test orders, and though the authors call the differences not statistically significant, a more order-invariant accumulation mechanism would strengthen the approach.
• Extending beyond CZSL: The idea of test-time multimodal knowledge accumulation from unlabeled data could be transferred to other settings with label-space shift at deployment, which the paper frames as a broader motivation.

Target Audience

Researchers and graduate students working on zero-shot learning, vision-language models, and test-time or continual adaptation, as well as practitioners who need a deployed image-recognition model to keep improving from unlabeled user data without full retraining. Readers need a working understanding of CLIP-style contrastive embeddings, prototype classification, and entropy-based test-time adaptation to follow the method details.

Authors’ abstract

Compositional Zero-Shot Learning (CZSL) aims to recognize novel attribute-object compositions based on the knowledge learned from seen ones. Existing methods suffer from performance degradation caused by the distribution shift of label space at test time, which stems from the inclusion of unseen compositions recombined from attributes and objects. To overcome the challenge, we propose a novel approach that accumulates comprehensive knowledge in both textual and visual modalities from unsupervised data to update multimodal prototypes at test time. Building on this, we further design an adaptive update weight to control the degree of prototype adjustment, enabling the model to flexibly adapt to distribution shift during testing. Moreover, a dynamic priority queue is introduced that stores high-confidence images to acquire visual knowledge from historical images for inference. Considering the semantic consistency of multimodal knowledge, we align textual and visual prototypes by multimodal collaborative representation learning. Extensive experiments indicate that our approach achieves state-of-the-art performance on four benchmark datasets under both closed-world and open-world settings. Code will be available at https://github.com/xud-yan/TOMCAT .