Open Ad-hoc Categorization with Contextualized Feature Learning

AI summary

Overview

Research area: Computer vision — specifically open-vocabulary recognition, generalized category discovery (GCD), and vision-language model adaptation (CLIP prompt/context tuning).

Technical level: Advanced. The paper assumes familiarity with CLIP, Vision Transformers (ViT), contrastive learning, semi-supervised clustering, Hungarian matching, and t-SNE visualization.

Scope: The paper introduces a new problem setting called open ad-hoc categorization, proposes a method (OAK) that learns context-specific tokens to modulate a frozen CLIP backbone, and benchmarks it against semantic-only and visual-only baselines on the Stanford and Clevr-4 datasets.

What This Paper Is About

Most image classifiers assume a fixed set of categories, like plants or animals. But many real-world goals require ad-hoc categories — for example, "things to sell at a garage sale" — which depend heavily on context and may have no visual or semantic similarity. This paper defines the task of open ad-hoc categorization: given a few labeled exemplars and lots of unlabeled images under an unknown context, the model must infer that latent context and then expand the categories both semantically (naming new concepts) and visually (finding coherent clusters). The proposed model, OAK, does this by learning small sets of context tokens at the input of a frozen CLIP encoder.

Key Contributions

1. A new task formulation. The paper introduces open ad-hoc categorization, which unifies context learning with representation switching — the model must both infer the organizing principle (e.g., Action, Location, Mood) and expand categories around it.

2. The OAK model. A simple architecture that prepends a small set of learnable context tokens to a frozen CLIP ViT encoder, trained jointly with CLIP's image-text alignment loss and GCD's visual clustering loss (self-supervised plus supervised contrastive losses).

3. A new evaluation metric, Omni accuracy. A measure that only counts a prediction as correct when the model predicts labels correctly from all contexts for the same image, capturing the ability to switch representations across contexts.

4. Empirical results plus interpretability. OAK reports state-of-the-art results on the Stanford and Clevr-4 benchmark contexts, produces context-aligned saliency maps without direct supervision, and can name discovered novel clusters by matching image embeddings to text embeddings.

Main Findings

• Stanford benchmark performance: OAK achieves 87.4% novel accuracy on Stanford Mood, which the paper describes as surpassing CLIP and GCD by over 50% and as a 50% gain over both in that context. Per-context novel accuracies for OAK are 85.1 (Action), 88.4 (Location), and 87.4 (Mood). Overall accuracies are 86.9, 85.9, and 78.4 respectively.

• Omni accuracy on Stanford: OAK reaches 70% Omni accuracy, compared to 43% for CLIP-ZS + LLM vocab and 52% for GCD. The paper notes the Omni metric is computed over 127 overlapping images across all contexts, and that the known-class Omni result is based on only 8 overlapping images, which the table tones down in light gray as unreliable.

• Clevr-4 results: OAK's overall accuracies are 64.6 (Texture), 100.0 (Color), 99.8 (Shape), and 44.4 (Count), with an Omni accuracy of 28.5. GCD's overall accuracies on the same contexts are 58.2, 96.6, 99.1, and 42.1, with Omni accuracy of 22.6. OAK's novel accuracies are 47.8, 100.0, 99.8, and 43.7.

• Full-shot setting: Using 2K labels per context on Clevr-4, OAK reaches an average novel accuracy of 83.3, compared with 71.1 for GCD, 71.6 for SimGCD, and 78.0 for μGCD. OAK reports a 66.5 on Texture, which the paper describes as an 11% gain over μGCD (55.5), and 99.9 on Color.

• Interpretable attention that follows context: Saliency maps (computed with the method of Chefer et al.) show OAK focusing on hands for Action, the entire scene for Location, and faces for Mood — all without direct supervision. On Clevr-4, OAK attends to small regions for Texture and Color, multiple 2D views for Shape, and all objects for Count.

• CLIP and GCD failure modes: CLIP consistently focuses on statistically salient objects (typically humans) regardless of context, while GCD often attends to irrelevant regions — for instance, attending to a bike rather than the human action when distinguishing "fixing a bike" from "riding a bike."

• Feature-space separation: t-SNE visualizations show that CLIP forms diffuse, poorly separated clusters under the Action context, whereas OAK produces clear, well-separated clusters, and that images close together under Action become far apart under Location or Mood.

• Cluster naming: OAK can assign plausible names to discovered clusters, including cases where its prediction differs from the annotation but is reasonable — for example, predicting "jumping" for images annotated as "dancing." The paper reports that OAK also performs well on the challenging Count context, a concept CLIP finds difficult to understand.

Methodology in Plain English

The approach starts from the observation that ad-hoc and ordinary categories rely on the same perceptual machinery, but ad-hoc ones additionally require contextual interpretation. CLIP already provides strong perceptual and semantic knowledge, but it assumes a single fixed semantic space. Rather than modifying CLIP's weights, OAK keeps the image and text encoders frozen and adds a small set of trainable context tokens as extra inputs to the ViT alongside the image patch tokens. These tokens are learned per context, so switching context means swapping tokens only.

Training combines two objectives. The first is GCD's clustering loss — a weighted combination of self-supervised contrastive loss on unlabeled data and supervised contrastive loss on labeled data — which pushes visually similar images together at the bottom-up level. The second is a text guidance loss: a classification loss between image embeddings and text embeddings over the known class names plus candidate novel class names generated by prompting an LLM. Labeled images use their ground-truth labels; unlabeled images use pseudo-labels produced by semi-supervised K-means with Hungarian matching between clusters and text embeddings, recomputed each epoch. The two losses are added together with a weighting hyperparameter.

At inference, the model obtains visual clusters and their pseudo-labels in the same way, giving predictions for both known and novel classes. Unlike plain GCD, text guidance lets OAK assign a text label to each cluster. Experiments use a ViT-B/16 image encoder with OpenAI's released CLIP weights, and baselines are reimplemented under a consistent setup, adjusting only learning rates and epochs per dataset.

Why This Matters

Impact on research: The paper reframes category discovery as a context-switching problem rather than a single fixed clustering problem, and provides a metric (Omni accuracy) for measuring consistency across contexts. It also offers a lightweight alternative to fine-tuning: a frozen backbone plus per-context tokens. Its full-shot results on Clevr-4 suggest the approach is compatible with, rather than a replacement for, the existing GCD line of work.

Real-world applications:

• Household robots that alternate between reasoning about Actions (for assistance), Locations (for navigation), and Moods (for emotional responsiveness).
• Assistive or organizational tools that group objects by goal rather than by visual similarity, such as the paper's motivating garage-sale example of "things to sell."
• Retrieval or recommendation systems that must re-interpret the same images under different user-defined criteria.
• Annotation and dataset-curation pipelines, where the model can propose names for novel clusters and thereby reduce manual labeling effort.

Industry relevance: The few-shot setting (16 images per class in the paper's benchmarks) is motivated by the practical cost of collecting labels for many ad-hoc contexts. The reported full-shot comparison includes baseline numbers taken from prior work and shows OAK at 83.3 average novel accuracy on Clevr-4, which is relevant for teams that already rely on GCD-style pipelines and want semantic guidance without abandoning their existing clustering objectives. Code is released at the project's GitHub repository, and the work was supported in part by Bosch gift funds.

Future Directions

• Beyond static images. The paper explicitly names extending ad-hoc categorization to settings beyond static images as an exciting direction, though it does not specify which modalities.
• Reducing dependence on LLM-generated vocabulary. OAK's text guidance relies on prompting an LLM for candidate novel class names; the paper notes alternatives such as textual inversion could be used, but does not explore them.
• Understanding when text guidance helps. The paper observes that text guidance is most useful when the CLIP image encoder is less familiar with a concept, as in Location or Mood, and that CLIP is weaker on Clevr-4 due to limited exposure to synthetic images — leaving open how to predict or target that benefit.
• Cross-context consistency at scale. Omni accuracy is measured on 127 overlapping Stanford images, and the known-class Omni figure rests on only 8 overlapping images — a limitation the paper itself flags as unreliable, suggesting larger multi-context evaluation sets are needed.

Target Audience

Researchers and graduate students working on open-vocabulary recognition, generalized category discovery, vision-language models, and parameter-efficient adaptation (prompt tuning, visual prompt tuning, register tokens). It is also relevant to practitioners who need classifiers that adapt to shifting, user-defined criteria without retraining a backbone, and to those interested in model interpretability through attention-based saliency analysis. Readers should already be comfortable with CLIP, ViT architecture, contrastive losses, and semi-supervised clustering.

Authors’ abstract

Adaptive categorization of visual scenes is essential for AI agents to handle changing tasks. Unlike fixed common categories for plants or animals, ad-hoc categories are created dynamically to serve specific goals. We study open ad-hoc categorization: Given a few labeled exemplars and abundant unlabeled data, the goal is to discover the underlying context and to expand ad-hoc categories through semantic extension and visual clustering around it. Building on the insight that ad-hoc and common categories rely on similar perceptual mechanisms, we propose OAK, a simple model that introduces a small set of learnable context tokens at the input of a frozen CLIP and optimizes with both CLIP's image-text alignment objective and GCD's visual clustering objective. On Stanford and Clevr-4 datasets, OAK achieves state-of-the-art in accuracy and concept discovery across multiple categorizations, including 87.4% novel accuracy on Stanford Mood, surpassing CLIP and GCD by over 50%. Moreover, OAK produces interpretable saliency maps, focusing on hands for Action, faces for Mood, and backgrounds for Location, promoting transparency and trust while enabling adaptive and generalizable categorization.