Fine-R1: Make Multi-modal LLMs Excel in Fine-Grained Visual Recognition by Chain-of-Thought Reasoning

AI summary

Overview

Research area: Computer Vision / Multi-modal Large Language Models (MLLMs), specifically fine-grained visual recognition (FGVR) and reinforcement-learning-based reasoning.

Technical level: Advanced. The paper assumes familiarity with MLLM fine-tuning, contrastive vision-language models (CLIP/SigLIP), chain-of-thought prompting, and policy-gradient RL (GRPO/DAPO/KL divergence).

Scope: The paper introduces Fine-R1, a two-stage training framework (Chain-of-Thought supervised fine-tuning plus a new Triplet Augmented Policy Optimization algorithm) that makes multi-modal LLMs competitive with, and often better than, dedicated contrastive classifiers on fine-grained recognition—using only four training examples per category.

What This Paper Is About

Fine-grained visual recognition—telling apart visually near-identical sub-categories like Acadian versus Least Flycatcher—is hard for multi-modal LLMs, which typically lag behind dedicated contrastive models such as CLIP and tend to overfit to the categories they were trained on. This paper asks whether an LLM can be taught to reason its way through fine-grained distinctions ("visual analysis, candidate sub-categories, comparison, prediction") and to deploy the sub-category knowledge it already possesses, rather than trying to memorize new classifiers from large labeled datasets. The goal is a model that works in both closed-world (pick from a list) and open-world (name the species freely) settings, and that generalizes to sub-categories it has never seen during training.

Key Contributions

1. Fine-R1, a two-stage training framework for FGVR: a Chain-of-Thought supervised fine-tuning stage that teaches structured "visual analysis, candidate sub-categories, comparison, final prediction" reasoning, followed by a reinforcement-learning stage that shapes how the model deploys that knowledge.
2. Triplet Augmented Policy Optimization (TAPO): a policy-gradient algorithm built on DAPO that adds intra-class augmentation (mixing rollouts from an anchor image and a same-category positive image to handle high intra-class variance) and inter-class augmentation (maximizing KL divergence between the response distributions conditioned on an anchor/positive image versus a visually similar negative image from a different sub-category, to handle low inter-class variance).
3. A high-quality open-world FGVR CoT dataset: 404 examples generated by Qwen2.5-VL-32B with image-level visual concept extraction, filtered for exact sub-category match, corrected for language mixing, and manually verified for consistency with ground truth.
4. Extensive empirical validation across six FGVR benchmarks under a few-shot base-to-new generalization protocol (60% seen / 40% unseen categories, 4 shots per seen category), plus a mechanistic analysis showing the gains come from better deployment of existing knowledge rather than new visual features or new taxonomic knowledge.

Main Findings

• State-of-the-art closed-world accuracy: Fine-R1-7B reaches 91.71% on seen categories and 85.70% on unseen categories, beating Qwen2.5-VL-7B by +7.73% / +9.28% and even the strong contrastive SigLIP-L by +3.38% / +5.16%, despite being trained only on open-world FGVR tasks.
• Large open-world gains: Fine-R1-7B achieves 74.80% average relative semantic similarity, a +23.75% improvement over Qwen2.5-VL-7B and +30.98% over the reasoning model DeepPerception-7B; even the 3B variant (67.32%) outperforms far larger general MLLMs.
• Few-shot is enough, and generalization is real: With only 4 shots per seen category, the model does not merely memorize: it outperforms plain SFT by +15.59%, CLS-RL by +10.28%, and No-Thinking RL by +10.05% on unseen categories.
• SFT alone overfits; RL alone underperforms: Supervised fine-tuning improves seen accuracy (+3.98%) but hurts unseen accuracy (−6.12%), while classification-RL degrades seen accuracy by 5.92%, motivating the combined two-stage design.
• Both augmentations matter, and complementary: CoT SFT gives a +13.30% relative-semantic-similarity boost; adding DAPO adds +1.60%; intra-class and inter-class augmentation each help individually and together produce the best result (67.32%).
• More rollouts isn't the reason intra-class augmentation works: A 1:1 anchor-to-positive ratio beats 10:0, 8:2, and 2:8, indicating diversity—not volume—drives the gain.
• Quality beats quantity in CoT data: Going from 404 to 1199 CoT samples improves average performance (58.66 → 62.64), showing the model does not overfit the small synthetic set and that high-quality data is more valuable than large-scale data.
• The gain is about deployment, not new knowledge—via three hypotheses: Linear probing shows Fine-R1's visual features are essentially unchanged (H1 rejected), and intra-genus versus inter-genus embedding similarity differences are unchanged (H2 rejected), but PCA on the final hidden state shows positive and negative image–category contexts become more linearly separable in Fine-R1, confirming H3—the model deploys existing fine-grained knowledge better.
• Broader benefits beyond classification: Fine-R1 improves accuracy on non-classification questions that depend on object recognition (e.g., +3.60% over Qwen2.5-VL-3B on ImageWikiQA) while preserving or improving general VQA performance.
• Cross-architecture generality: TAPO also improves Qwen2-VL-2B (62.32% → 64.65% over CoT SFT + DAPO), showing the method is not tied to one base model.

Methodology in Plain English

The authors start from the observation that generic MLLMs already "know" a lot about fine-grained categories from pretraining, but fail to use that knowledge at inference time. Their fix is a two-step training recipe.

First, they build a small, carefully curated set of worked examples. For each chosen image, they use a large model (Qwen2.5-VL-32B) to generate multiple captions emphasizing different visual traits, distill those into a compact set of discriminative visual concepts, and then prompt the model to produce a structured reasoning chain: analyze the image, list the most likely candidate sub-categories, compare them using text knowledge, and give a final prediction. Only chains that reach the correct answer, are in English, and are internally consistent (predicted sub-category is among the candidates and matches ground truth) survive, and a human checks the rest. The model is then fine-tuned on these chains, learning to imitate the reasoning pattern.

Second, they refine the model with reinforcement learning. Drawing on DAPO, they reward correct final answers, but they add two twists specific to fine-grained recognition. For intra-class robustness, they sample additional rollouts using a different image of the same sub-category, pooling those rollouts into the same reward computation so the policy sees more varied appearances of the class. For inter-class discrimination, they also feed the model a "confusable" image from the most visually similar different sub-category and add a loss term (a KL divergence) that pushes the model's output distribution to differ sharply between the correct image and the confusable one—so the model is rewarded for actually reacting to the fine-grained cues rather than producing a generic answer.

Finally, they diagnose why it works by probing the model's internal representations: linear probes on image embeddings, similarity spread across taxonomic genera, and PCA of the final hidden state for positive versus negative image–category pairs.

Why This Matters

Impact on research. The paper challenges the assumption that good fine-grained recognition requires either massive annotation or a dedicated contrastive architecture. It shows that chain-of-thought reasoning plus carefully designed RL can push a generative MLLM past CLIP-family discriminative models, and it offers a mechanistic explanation (better deployment of latent knowledge, not new knowledge), which is a useful diagnostic for the broader MLLM reasoning literature. It also extends RL-with-verifiable-rewards into classification, an area largely dominated by CLIP-style approaches.

Real-world applications.

• Biodiversity monitoring and citizen-science apps, where users photograph birds, plants, or insects and thousands of species must be distinguished with little labeled data per species.
• E-commerce and automotive product identification, where subtly different models (e.g., car trims, aircraft variants) must be resolved from user photos.
• Medical and agricultural sub-type classification, where expert annotations for each subtype are scarce but the model may already hold relevant domain knowledge.
• Visual question answering and assistants that need to name an object correctly before answering downstream questions about it.

Industry relevance. The 4-shot training regime drastically reduces labeling cost, making FGVR deployable in domains where new categories appear continuously (new species, new product SKUs, new regulations). The finding that gains transfer to downstream non-classification tasks and generalize across base architectures (Qwen2-VL, Qwen2.5-VL) suggests the method can be layered onto existing production MLLMs rather than requiring an architecture swap.

Future Directions

• Scaling and systematizing CoT data generation. The paper shows quality outweighs quantity, but the process still relies on a large teacher model plus manual verification; automating and broadening this pipeline (e.g., to domains beyond birds, cars, flowers) is an open problem.
• Extending TAPO beyond static image classification. How the triplet construction—anchor, positive, negative—would work for video, 3D, or multi-label fine-grained tasks is unexplored.
• Better negative mining. The current approach uses "the most visually similar distinct sub-category"; improving how negatives are chosen (learned, retrieval-based, taxonomy-aware) could further sharpen inter-class discrimination.
• Deeper understanding of the deployment mechanism. H3 is demonstrated via PCA separability, but the precise mechanism by which RL changes how latent knowledge is surfaced remains an open interpretability question—and could suggest more direct training objectives.
• Combining Fine-R1 with retrieval or external knowledge bases. Since the model's internal taxonomy knowledge is unchanged, pairing it with retrieval could improve recognition of truly novel categories lacking any pretraining exposure.

Target Audience

Researchers and graduate students in computer vision and multi-modal machine learning, especially those working on fine-grained recognition, chain-of-thought reasoning, or RL post-training for MLLMs. Practitioners building visual recognition systems with tight annotation budgets (biodiversity, retail, medical imaging) will find the few-shot results and the base-to-new generalization protocol directly actionable. Readers seeking an entry point should be comfortable with MLLM fine-tuning and policy-gradient RL, since the TAPO objective is mathematically dense.

Authors’ abstract

Any entity in the visual world can be hierarchically grouped based on shared characteristics and mapped to fine-grained sub-categories. While Multi-modal Large Language Models (MLLMs) achieve strong performance on coarse-grained visual tasks, they often struggle with Fine-Grained Visual Recognition (FGVR). Adapting general-purpose MLLMs to FGVR typically requires large amounts of annotated data, which is costly to obtain, leaving a substantial performance gap compared to contrastive CLIP models dedicated for discriminative tasks. Moreover, MLLMs tend to overfit to seen sub-categories and generalize poorly to unseen ones. To address these challenges, we propose Fine-R1, an MLLM tailored for FGVR through an R1-style training framework: (1) Chain-of-Thought Supervised Fine-tuning, where we construct a high-quality FGVR CoT dataset with rationales of "visual analysis, candidate sub-categories, comparison, and prediction", transition the model into a strong open-world classifier; and (2) Triplet Augmented Policy Optimization, where Intra-class Augmentation mixes trajectories from anchor and positive images within the same category to improve robustness to intra-class variance, while Inter-class Augmentation maximizes the response distinction conditioned on images across sub-categories to enhance discriminative ability. With only 4-shot training, Fine-R1 outperforms existing general MLLMs, reasoning MLLMs, and even contrastive CLIP models in identifying both seen and unseen sub-categories, showing promise in working in knowledge-intensive domains where gathering expert annotations for all sub-categories is arduous. Code is available at https://github.com/PKU-ICST-MIPL/FineR1_ICLR2026.