Start Small, Think Big: Curriculum-based Relative Policy Optimization for Visual Grounding

AI summary

Overview

• Research area: Computer vision / vision-language models — specifically reinforcement-learning-based fine-tuning of Chain-of-Thought (CoT) reasoning for Visual Grounding.
• Technical level: Intermediate to Advanced. Readers should be comfortable with reinforcement learning for language models (GRPO, PPO-style clipped objectives, KL regularization), bounding-box metrics (IoU, gIoU, mAP, mIoU), and curriculum learning.
• Scope in one sentence: The paper shows that explicitly generating Chain-of-Thought reasoning can hurt visual grounding, then proposes CuRPO, a curriculum training strategy that orders training examples from short-CoT/simple to long-CoT/hard based on CoT length and gIoU reward, and evaluates it on RefCOCO, RefCOCO+, RefCOCOg, and LISA.

What This Paper Is About

Chain-of-Thought prompting is widely credited with improving reasoning, and reinforcement learning methods such as GRPO have been used to fine-tune it. This paper reports the counterintuitive finding that in Visual Grounding — where a model must output the bounding box of an object described in text — forcing the model to generate explicit CoT steps can reduce localization accuracy, particularly when the reasoning chains get long. The researchers also find that simply adding more training data does not reliably help, because the added examples vary in difficulty. Their goal is a training scheme that controls how examples are presented (simple to complex) rather than only how many are presented, and that improves accuracy, stability, and few-shot performance.

Key Contributions

1. An empirical identification that explicit CoT generation can degrade visual grounding. The authors fine-tune a visual grounding model with GRPO and compare a "no CoT" direct-output condition against a condition that explicitly generates intermediate reasoning steps, finding the direct-output model performs better.
2. CoT length and reward as novel complexity indicators. They propose using the average CoT length per sample and the gIoU-based reward as signals of task difficulty, and provide both theoretical reasoning and statistical correlation analysis supporting this use.
3. CuRPO, a curriculum-based relative policy optimization strategy. A GRPO-based training pipeline that sorts examples by complexity and progressively unlocks easier to harder phases, with variants using random, length-based, and reward-based sorting.
4. Broad empirical validation. Experiments on RefCOCO, RefCOCO+, RefCOCOg, and LISA show consistent gains over baselines including Visual-RFT, with improvements of up to +12.52 mAP on RefCOCO, plus strong few-shot behavior and per-category breakdowns.

Main Findings

• CoT hurts grounding in their setup. At a small dataset size of 40 samples, the model without CoT generation reaches an mIoU of 35.6, outperforming the CoT-outputting counterpart, which reaches only mIoU 34.3 even with a much larger dataset of 239 samples. The direct-output model reaches a maximum mIoU of 39.6.
• More data is not automatically better. The CoT-generating model shows unstable accuracy — rising, then dropping and stagnating as more data is added — while the direct-output model improves consistently with data size.
• Longer CoT correlates with lower reward. The paper reports a Pearson correlation coefficient of −0.4395 (p = 1.04 × 10⁻¹²), Spearman's rank correlation of −0.4268 (p = 5.34 × 10⁻¹²), and Kendall's Tau of −0.2981 (p = 6.81 × 10⁻¹²) between CoT length and reward.
• A theoretical rationale for longer chains being harder. Modeling a reasoning chain of C independent steps each with success probability p_c, the paper gives the success probability as the product of p_c over c = 1 to C, which decreases exponentially in C when p_c < 1.
• Short-CoT subsets train faster, long-CoT subsets catch up. Models trained on shorter-CoT subsets initially achieve higher accuracy than those trained on longer-CoT subsets, but the gap diminishes with larger training datasets.
• CuRPO beats baselines on LISA. On LISA, CuRPO with only 50 training examples reaches mIoU 37.4, exceeding Visual-RFT trained on all 239 examples by +3.0; with 200 examples it reaches 38.7 (+4.3 over Visual-RFT at 239), and with 239 examples it reaches 38.4 (+4.0).
• Strong gains on RefCOCO. On RefCOCO validation, CuRPO (Length) achieves mAP 33.80 versus Visual-RFT's 21.28 (+12.52); on RefCOCO test, CuRPO (Length) reaches 31.42 versus 20.38 (+11.04). The pretrained Qwen2-VL-2B baseline is 11.57 and 10.70 respectively.
• Different sorting strategies win on different datasets. CuRPO (Length) is best on RefCOCO; on RefCOCO+, CuRPO (Reward) leads validation with 26.85 (+8.44) while CuRPO (Length) leads test with 25.10 (+4.20); on RefCOCOg validation, CuRPO (Reward) leads with 32.65 (+9.26).
• Even random ordering helps. CuRPO (Random) still delivers large gains — 33.09 on RefCOCO val, 29.92 on RefCOCO test, 26.82 on RefCOCO+ val, 24.34 on RefCOCO+ test, 27.98 on RefCOCOg val — suggesting curriculum RL without explicit CoT generation is itself beneficial.
• Most generated CoTs are low quality. Of 239 × 8 = 1912 CoT instances, 59.52% fall in the low-similarity/low-reward region, and 97.35% of the low-similarity group fail to produce correct predictions. Only 5.23% of all samples are high-similarity/high-reward.
• Reasoning and localization may compete for capacity. At 200 training samples, a Raw+CoT prompt (original question concatenated with its CoT) achieves mIoU 38.56, beating the original-text-only setting (37.51) and the CoT-only setting (37.21).
• Per-category gains. Appendix tables report per-category mAP for RefCOCO, RefCOCO+, and RefCOCOg, showing consistent improvements on difficult categories such as chair, bed, and toilet.

Methodology in Plain English

The researchers start from a surprising observation: a model asked to "think out loud" before pointing at an object does worse than one that just points. They dig into why and find two clues. First, longer reasoning chains are statistically associated with lower localization reward. Second, dumping more training data into the model does not reliably help, which suggests the data contains a mix of easy and hard examples that matters more than the raw count.

CuRPO turns those clues into a training schedule. For each image–question pair, the authors have a pretrained VLM generate several candidate CoTs (typically 8 per sample) and record their average length. That average length becomes a difficulty score. They also use the visual reward itself as a second difficulty signal. Training examples are then sorted from easiest (shortest CoT) to hardest (longest CoT) and split into curriculum phases; in the refined setup, examples are sorted first by CoT length and then, within each 50-token length bin, by reward value. The model trains through phases in order, seeing short-CoT, high-reward examples first and longer, lower-reward ones later.

The learning algorithm underneath is GRPO, the same reinforcement learning backbone used by the Visual-RFT baseline. For each question the policy generates G candidate outputs; each output receives a reward combining a visual term derived from gIoU (linearly rescaled from [−1, 1] to [0, 2] to soften negative feedback) and a format-correctness term. Rewards are converted into group-normalized advantages, and the policy is updated with a clipped surrogate loss plus a KL-divergence term keeping it close to a reference policy. The base model is Qwen2-VL-2B, and the baseline is the same model fine-tuned with GRPO in the "with CoT" setting. Evaluation uses mIoU on LISA and class-wise Average Precision / mAP on the RefCOCO family.

Why This Matters

Impact on research. The paper challenges a broadly held assumption — that explicit intermediate reasoning steps are generally beneficial — by demonstrating a task where they are not, and it reframes data curation as a sequencing problem rather than a volume problem. It also offers a practical, low-cost difficulty signal (CoT length, reward) that does not require manual annotation of example difficulty, and it adds to the growing literature on "over-thinking" and length-adaptive reasoning.

Real-world applications (note: the paper does not itself enumerate deployment scenarios; the following follow from the visual grounding task it studies):

• Referring-expression interfaces, where a user points at something using natural language ("the mug next to the laptop") and a system must return its location.
• Robotics and assistive systems that need to locate an object named in a spoken or typed instruction before acting on it.
• Image and video editing or annotation tools that need to select an object from a description.
• Content moderation, retrieval, or cataloging pipelines that need to localize described items in images.

Industry relevance. CuRPO's headline promise for practitioners is efficiency: it outperforms a strong RL baseline using only 50 training examples on LISA (37.4 mIoU versus 34.4 for Visual-RFT at 239), and it builds on a small 2B-parameter model (Qwen2-VL-2B). For teams with limited labeled grounding data or limited compute, that combination of a small base model plus a curriculum schedule is directly actionable. The authors also release code at https://github.com/qyoung-yan/CuRPO.

Future Directions

• Extending the paradigm beyond visual grounding. The conclusion states that future work will focus on evaluating and adapting the curriculum–GRPO paradigm to other vision–language and multimodal reasoning tasks to assess generalization beyond visual grounding.
• Better handling of noisy reasoning traces. Given that 59.52% of generated CoTs were low-similarity/low-reward and 97.35% of low-similarity CoTs failed to localize correctly, how to filter, repair, or down-weight such traces remains open.
• Resolving the reasoning-versus-localization capacity competition. The Raw+CoT result (38.56 mIoU versus 37.51 and 37.21) suggests CoT is better used as an auxiliary prompt than as a generated output; how far this can be pushed is unresolved.
• Scaling and architecture sensitivity. Appendix C reports additional experiments with Qwen2-VL-7B-Instruct intended to isolate training strategy from architecture, but the provided content is truncated, so the conclusions of that larger-model study are not reported here.

Target Audience

This paper is most valuable to reinforcement-learning and vision-language researchers who work on grounding, referring expression comprehension, or RL fine-tuning of multimodal models, and to practitioners who need strong localization results from small models and small labeled datasets. Readers without a background in policy-gradient methods or object-detection metrics will need to consult the cited GRPO, Visual-RFT, gIoU, and AP/mAP references first, since the method section assumes that vocabulary.

Authors’ abstract

Chain-of-Thought (CoT) prompting has recently shown significant promise across various NLP and computer vision tasks by explicitly generating intermediate reasoning steps. However, we find that reinforcement learning (RL)-based fine-tuned CoT reasoning can paradoxically degrade performance in Visual Grounding tasks, particularly as CoT outputs become lengthy or complex. Additionally, our analysis reveals that increased dataset size does not always enhance performance due to varying data complexities. Motivated by these findings, we propose Curriculum-based Relative Policy Optimization (CuRPO), a novel training strategy that leverages CoT length and generalized Intersection over Union (gIoU) rewards as complexity indicators to progressively structure training data from simpler to more challenging examples. Extensive experiments on RefCOCO, RefCOCO+, RefCOCOg, and LISA datasets demonstrate the effectiveness of our approach. CuRPO consistently outperforms existing methods, including Visual-RFT, with notable improvements of up to +12.52 mAP on RefCOCO. Moreover, CuRPO exhibits exceptional efficiency and robustness, delivering strong localization performance even in few-shot learning scenarios, particularly benefiting tasks characterized by ambiguous and intricate textual descriptions.The code is released on https://github.com/qyoung-yan/CuRPO.