Thinking Beyond Labels: Vocabulary-Free Fine-Grained Recognition using Reasoning-Augmented LMMs

AI summary

Overview

Research area: Computer vision, specifically fine-grained image recognition and open-world/multimodal foundation models.

Technical level: Intermediate. The paper assumes familiarity with vision-language models (CLIP-style encoders), embeddings, zero-shot classification, and basic prompt engineering, but the core ideas are explained with enough context for readers outside the subfield.

Scope: The paper introduces FiNDR, a fully automated pipeline that uses a reasoning-capable large multimodal model to invent its own category names for fine-grained images, then verifies and deploys those names as a lightweight classifier — all without any predefined label vocabulary.

What This Paper Is About

Fine-grained recognition — telling a Cocker Spaniel from a Springer Spaniel, or a Dodge Sprinter from a Mercedes Sprinter — normally requires a large, human-curated list of category names prepared by domain experts. That assumption breaks down in open-world settings where nobody knows in advance which categories exist. This paper asks whether a modern reasoning-augmented large multimodal model (LMM) can generate its own fine-grained vocabulary from a handful of unlabelled images, and whether that self-generated vocabulary can be turned into a classifier that works at inference time on unseen data.

Key Contributions

1. First application of reasoning-augmented LMMs to vocabulary-free fine-grained recognition. The authors identify and fill a gap in the literature, showing that step-by-step reasoning models are a natural fit for the problem of naming categories without supervision.

2. A fully automated three-stage framework (FiNDR) requiring no prior knowledge. The pipeline discovers candidate names via an LMM, filters them with a vision-language model, and fuses text and visual prototypes into a classifier — no manual labelling, no fixed vocabulary, no retraining of large models.

3. State-of-the-art results with a large margin, including a challenge to the "upper bound" assumption. FiNDR improves average clustering accuracy by +9.5 points over the previous best vocabulary-free method, and outperforms a CLIP zero-shot classifier that has access to ground-truth human-written class names.

4. A systematic study of prompt design and reasoning strategies across open and closed models. The paper isolates what makes LMM naming reliable (meta-category context, expert personas) and shows open-source models can approach proprietary ones with the right prompts.

Main Findings

• Substantial gains over prior vocabulary-free methods. Across five benchmarks (CUB-200, Stanford Cars, Stanford Dogs, Oxford Flowers, Oxford Pets), FiNDR reaches an average 67.9% clustering accuracy and 70.6% semantic accuracy, versus 58.4% and 66.3% for the previous best (E-FineR). The largest relative gain reported is 18.8%.

• The strongest results appear on datasets with subtle visual distinctions. On Oxford Pets, FiNDR achieves 86.5% clustering accuracy (a 13.6-point absolute gain over the best prior method) and correctly names specific breeds such as Egyptian Mau and Staffordshire Bull Terrier where earlier pipelines produced partially correct answers like "American Cocker Spaniel" for an English Cocker Spaniel.

• It beats a zero-shot baseline that uses ground-truth class names. On average clustering accuracy, FiNDR (67.9%) exceeds CLIP zero-shot with human-curated labels (65.8%), though it remains below that baseline on semantic accuracy (70.6% vs. 77.6%). The claim is strongest for grouping images correctly; matching a single canonical name is a separate, harder bar.

• Semantic accuracy understates the method on some datasets. On Oxford Flowers and Stanford Cars, FiNDR finds valid alternative names — "Helianthus Annuus" instead of "Sunflower", "Mercedes Sprinter" instead of "Dodge Sprinter" — that the single-canonical-label evaluation protocol marks as wrong. This indicates the metric design, not the model, limits reported semantic accuracy.

• Prompt design contributes measurably. Adding meta-category context (what kind of object, what granularity, what expert studies it) raises clustering accuracy by up to 5.2 points and semantic accuracy by up to 8.9 points over a naive "what is the main object?" prompt. Adding an expert persona such as "You are an ornithologist" adds a further 0.3 to 1.5 points of semantic accuracy.

• Open-source LMMs can approach proprietary ones with explicit reasoning prompts. Qwen2.5-VL-72B with the authors' prompts reaches 76.65% semantic accuracy, close to Gemini 2.5-Flash's 78.04% using its built-in internal reasoning. Carefully engineered explicit chain-of-thought prompting largely substitutes for a proprietary reasoning mode.

• Text and visual prototypes are complementary. The final classifier fuses text embeddings and averaged visual features (from 10 augmentations per image) with a coefficient of 0.7 weighting the text side, which helps absorb the noise inherent in guessed class names.

Methodology in Plain English

The framework operates in two broad phases: discovering a vocabulary and building a classifier from it.

Vocabulary discovery. From a tiny discovery set of just three unlabelled images per class, a reasoning-capable LMM is queried in two steps. First it sees a small sample of images and reports dataset-level meta-information: the broad category (e.g. "birds"), the unit of granularity within it (e.g. "species"), and the relevant expert (e.g. "ornithologist"). Second, each individual image is passed along with this context to elicit a specific candidate name. This two-step structure gives the model taxonomic scaffolding before it commits to a name.

Refinement. Raw LMM outputs are cleaned for formatting, pluralisation, and genericness. Each surviving candidate is then scored against the discovery images using a CLIP-style vision-language encoder: the average cosine similarity between the candidate's text embedding and the images' visual embeddings. Candidates that align poorly with the actual images are dropped.

Classifier construction. The refined names are pseudo-labelled onto the discovery images by nearest text embedding. Because only a handful of images exist per class, each image is augmented 10 times (random crops, horizontal flips) and the resulting visual features are averaged into a prototype. Text and visual prototypes are then mixed into a single per-class representation.

Inference. A test image is encoded and assigned to the class whose fused prototype has the highest cosine similarity. The output is a human-readable name rather than an opaque cluster index.

Everything runs without manual supervision, and the only models called at inference time are cheap CLIP-scale encoders (ViT-B/16 for the classifier, ViT-L/14 for refinement), with the large LMM needed only during discovery.

Why This Matters

Impact on research. The paper challenges a widely held assumption in the field: that a human-curated vocabulary is the best possible label set and therefore an unreachable ceiling for unsupervised methods. It also suggests that current semantic-accuracy metrics under-credit valid alternative names, which has implications for how the community evaluates open-vocabulary systems. The demonstration that explicit prompting can bridge much of the gap to proprietary reasoning models is practically useful for labs without access to closed APIs.

Real-world applications.

• Biodiversity monitoring: automatically cataloguing plant and animal species from camera traps or citizen-science photos where the set of relevant taxa is not known ahead of time.
• E-commerce and retail catalogue management: generating and organising fine-grained product taxonomies (vehicle trims, furniture styles, apparel variants) without a pre-built labelling scheme.
• Scientific and museum archives: labelling specimen photographs, herbarium sheets, or historical collections where specialist naming conventions vary by region and era.
• Visual search and recommendation: building interpretable, human-readable tags on the fly for image corpora, avoiding index-only clusters that users cannot reason about.

Industry relevance. The pipeline is attractive for deployment because the expensive LMM runs once during discovery, while the production classifier is a small CLIP-scale model that is fast and cheap at inference. It also removes the annotation bottleneck for companies that need to organise image data in domains where expert labelling is scarce or expensive.

Future Directions

• Evaluation protocols that accept multiple valid names. The Flowers and Cars results show that a single canonical label per class penalises correct but differently phrased predictions. Building semantic metrics that properly credit synonyms, scientific names, and regional variants is a clear next step.

• Scaling and robustness of the discovery stage. The current setup uses only three unlabelled images per class. How the method degrades or improves with more or fewer discovery samples, with heavy class imbalance, or with genuinely novel categories not present in the LMM's training data remains open.

• Extending beyond the closed-set assumption. FiNDR still assumes the discovered vocabulary covers the test set. Handling open-set scenarios — where test images may belong to classes the discovery set never hinted at — is a natural extension.

• Reducing reliance on large proprietary-scale LMMs. The paper shows prompt engineering helps open models, but smaller or locally runnable models were not fully explored. Whether a compact reasoning model can run the entire discovery phase on-device would broaden accessibility.

Target Audience

This paper is most valuable to computer vision and multimodal machine learning researchers working on fine-grained recognition, open-vocabulary and zero-shot classification, or LMM-based pipelines. It is also relevant to applied ML engineers building image cataloguing, taxonomy generation, or visual search systems who need to operate without pre-built label sets, and to practitioners interested in prompt-engineering strategies for extracting reliable structured knowledge from large multimodal models.

Authors’ abstract

Vocabulary-free fine-grained image recognition aims to distinguish visually similar categories within a meta-class without a fixed, human-defined label set. Existing solutions for this problem are limited by either the usage of a large and rigid list of vocabularies or by the dependency on complex pipelines with fragile heuristics where errors propagate across stages. Meanwhile, the ability of recent large multi-modal models (LMMs) equipped with explicit or implicit reasoning to comprehend visual-language data, decompose problems, retrieve latent knowledge, and self-correct suggests a more principled and effective alternative. Building on these capabilities, we propose FiNDR (Fine-grained Name Discovery via Reasoning), the first reasoning-augmented LMM-based framework for vocabulary-free fine-grained recognition. The system operates in three automated steps: (i) a reasoning-enabled LMM generates descriptive candidate labels for each image; (ii) a vision-language model filters and ranks these candidates to form a coherent class set; and (iii) the verified names instantiate a lightweight multi-modal classifier used at inference time. Extensive experiments on popular fine-grained classification benchmarks demonstrate state-of-the-art performance under the vocabulary-free setting, with a significant relative margin of up to 18.8% over previous approaches. Remarkably, the proposed method surpasses zero-shot baselines that exploit pre-defined ground-truth names, challenging the assumption that human-curated vocabularies define an upper bound. Additionally, we show that carefully curated prompts enable open-source LMMs to match proprietary counterparts. These findings establish reasoning-augmented LMMs as an effective foundation for scalable, fully automated, open-world fine-grained visual recognition. The source code is available on github.com/demidovd98/FiNDR.