Connecting the Dots: Training-Free Visual Grounding via Agentic Reasoning

AI summary

Overview

Research area: Computer vision and vision–language integration, specifically zero-shot visual grounding (referring expression comprehension) using pretrained detectors, multimodal large language models (MLLMs), and large language models (LLMs).

Technical level: Intermediate. Readers will benefit from familiarity with object detection, bounding boxes, vision-language models, and standard grounding benchmarks, but the core idea is conceptually straightforward.

Scope: The paper proposes GroundingAgent, a training-free framework that combines an open-vocabulary detector, an MLLM, and an LLM in an iterative reasoning pipeline to localize image regions from natural language queries, and evaluates it on RefCOCO, RefCOCO+, and RefCOCOg.

What This Paper Is About

Visual grounding means matching a phrase such as "the white chair by the fireplace" to the correct region of an image. Existing methods typically require large, expensive sets of task-specific bounding-box annotations and fine-tuning, which limits how well they transfer to novel or out-of-distribution concepts. This paper asks whether the same task can be solved with no task-specific training at all, by orchestrating already-pretrained detectors and language models into a step-by-step reasoning agent.

Key Contributions

1. A fully training-free visual grounding framework. GroundingAgent is presented as the first fully training-free visual grounding framework that uses a structured, agentic reasoning pipeline, integrating pretrained open-vocabulary detectors with multimodal and large language models without any task-specific fine-tuning.

2. State-of-the-art zero-shot performance. The framework reports an average zero-shot grounding accuracy of 65.1% on RefCOCO, RefCOCO+, and RefCOCOg, which the authors describe as surpassing previous zero-shot methods by significant margins and setting a new baseline for training-free visual grounding.

3. Strong interpretability and modularity. The pipeline explicitly shows each reasoning step, and its modular design allows individual pretrained vision or language models to be swapped or upgraded.

4. An ablation-driven diagnosis of the remaining bottleneck. By replacing MLLM-generated captions with the original query text, selection-stage accuracy reaches approximately 90%, showing that the framework's reasoning mechanism is not the limiting factor—caption quality is.

Main Findings

• Zero-shot accuracy of 65.1% on average. GroundingAgent scores 67.1 / 73.3 / 60.1 on RefCOCO val / testA / testB, 62.4 / 67.6 / 53.8 on RefCOCO+ val / testA / testB, and 67.9 / 68.8 on RefCOCOg val / test. No task-specific fine-tuning is used.

• Large margin over prior zero-shot methods. The authors report accuracy improvements of approximately 12–27% across different subsets relative to fully zero-shot competitors such as VGDiffZero, ReCLIP, and Red Circle.

• Superior to REG despite REG's implicit supervision. REG (Wang et al. 2024) trains with synthetically generated grounding annotations and has a reported average of 59.5%, but GroundingAgent outperforms it on all evaluation splits, including RefCOCO+ testB (53.8%) and RefCOCOg test (68.8%).

• Caption quality is the main bottleneck, not the reasoning. Using MLLM-generated captions yields 65.1% average accuracy. Substituting query-enhanced text (Query+) raises the average to 85.0%; substituting the original query directly (Caption → Query) raises it to 90.6%.

• Selection stage approaches supervised performance. With Caption → Query+, the 85.0% average is close to the 84.1% average of a reported SOTA fine-tuned model (Zheng et al. 2025). With Caption → Query, the 90.6% average matches the 90.3% average of a reported SOTA pre-trained model (Team 2025b).

• Candidate generation is robust across detectors. Recall is approximately 98.3% for APE, 98.2% for GroundingDINO, 94.9% for OWL, and 93.8% for YOLO-World on average across benchmarks, using only the bounding boxes and the vocabularies extracted by the method.

• Global captions improve candidate vocabulary generation. Removing caption information leads to a significant drop in recall, indicating that the global image description constrains the LLM's vocabulary generation and reduces semantic divergence.

• Reasoning ability matters more than model size. On RefCOCO testA / testB, DeepSeek-R1 reaches 75.9% / 60.3% and DeepSeek-V3 reaches 73.3% / 60.1%, whereas base Llama3.1-8B reaches 55.0% / 44.0%. DeepSeek-R1-Llama-8B, which incorporates explicit reasoning training, outperforms base Llama3.1-8B by 4.7 and 3.7 percentage points. Qwen2.5-7B is substantially lower at 52.0% / 41.6%.

• Low rejection rates and few reasoning steps. Quantitative analysis reports low rejection rates on RefCOCO+ (0.77% on val, 0.73% on testA, 1.69% on testB). The average number of reasoning steps is 3.4.

• Detector behavior on small targets diverges. For ground-truth instances below 5% and below 20% of image area, APE scores 96.9 / 98.1, GroundingDINO 95.3 / 96.9, OWL 94.9 / 93.9, and YOLO 90.2 / 92.5.

• Self-consistency helps. Sampling five independent instance descriptions per candidate (n=5) and having the LLM consolidate them raises RefCOCO-val accuracy from 67.1% to 68.5%.

• Stable across runs. Three independent runs across all benchmarks show an average standard deviation of approximately 0.55% across all splits.

• Segmentation extension works without retraining. Adding a Segment-Anything-Model refinement module to the selected bounding box yields 57.3 mIoU on RefCOCO-val, 51.2 mIoU on RefCOCO+-val, and 56.5 mIoU on RefCOCOg-val.

Methodology in Plain English

The pipeline has four stages, all inference-only:

1. Describe the scene. An MLLM generates a global caption of the input image, which is concatenated with the user's query to form an enriched context.

2. Propose candidate concepts. An LLM reads that enriched context and infers a set of semantically relevant candidate concepts (for example, "chair," "fireplace").

3. Detect and describe. Each concept is fed to an open-vocabulary object detector, which returns candidate bounding boxes. The MLLM then produces a detailed description of each candidate region, capturing visual attributes and context. Boxes are deduplicated with non-maximum suppression and sorted in descending order of area to prioritize prominent objects. Regions smaller than 2.5% of the image area are filtered out, and at most 10 primary candidates per image are retained.

4. Reason and select. An LLM receives the query, the global context, each bounding box, and its description, and applies chain-of-thought reasoning to decide whether each candidate matches the query. For Referring Expression Comprehension, the output is constrained to a one-hot decision (a single r_i = 1); the authors note the framework can be extended to tasks with no referred region or multiple referred regions.

The default configuration uses YOLO-World as the detector (chosen because it is not trained on RefCOCO, for unbiased zero-shot evaluation), Llama-3.2-11B-Vision for global and region-level descriptions, and DeepSeek-V3 (0324) for semantic reasoning. Four detectors—APE, Grounding DINO, OWL-ViT, and YOLO-World—are evaluated in total. For visualization, candidate regions are outlined in red and the background is blurred with a Gaussian blur of standard deviation 10.0. Success is measured as top-1 accuracy with an IoU greater than 0.5 against the ground-truth box.

Why This Matters

Impact on research. The paper separates two questions that are usually entangled: how well a model localizes, and how well it reasons about what to localize. Its ablations show that with a strong reasoning model the selection stage can reach roughly 90% accuracy—near supervised levels—even with no grounding-specific training, and that the residual gap traces back to MLLM caption quality. That reframing suggests the field's bottleneck may be in fine-grained multimodal description rather than in grounding supervision.

Real-world applications:

• Visual question answering, where the system must first locate the region a question refers to before answering.
• Human-robot interaction, where a person names an object by attributes and spatial relations and a robot must identify it in a cluttered scene.
• Interactive image retrieval, where users describe what they want in natural language rather than using tags or filters.
• Referring expression segmentation, which the paper's SAM-based extension shows is directly reachable from the grounded bounding box.

Industry relevance. The approach requires no annotation pipeline and no fine-tuning—only pretrained components and inference compute. Its modular design means an organization can swap in its own detector or LLM as better ones appear, without retraining. The explicit reasoning trace also supports debugging and auditing, which matters for deployments where a system must justify why it selected a region.

Future Directions

• Improving MLLM caption quality. The paper identifies inaccurate or hallucinated instance captions as the dominant error source; the n=5 self-consistency experiment is presented as an initial mitigation rather than a solution.
• Extending to general referring tasks. The one-hot constraint is specific to Referring Expression Comprehension; the authors note the framework can be extended to cases with no referred region or with multiple referred regions, but do not evaluate those settings.
• Handling very small referents. Appendix B shows detectors diverge in robustness on ground-truth instances below 5% and 20% of image area, and the authors call for proposal refinement and multi-scale handling for tiny referents.
• Transferring the paradigm to other domains. The conclusion suggests that similar training-free, reasoning-based frameworks may generalize to other domains requiring transparent and reliable decision-making.

Target Audience

Researchers and engineers working on visual grounding, referring expression comprehension, and vision-language systems; practitioners who need to deploy localization without assembling task-specific bounding-box annotations; and readers interested in agentic pipelines that orchestrate pretrained detectors and LLMs. It is also useful for those studying interpretability in multimodal systems, since the paper's reasoning traces are a central part of its argument rather than an afterthought.

Authors’ abstract

Visual grounding, the task of linking textual queries to specific regions within images, plays a pivotal role in vision-language integration. Existing methods typically rely on extensive task-specific annotations and fine-tuning, limiting their ability to generalize effectively to novel or out-of-distribution scenarios. To address these limitations, we introduce GroundingAgent, a novel agentic visual grounding framework that operates without any task-specific fine-tuning. GroundingAgent employs a structured, iterative reasoning mechanism that integrates pretrained open-vocabulary object detectors, multimodal large language models (MLLMs), and large language models (LLMs) to progressively refine candidate regions through joint semantic and spatial analyses. Remarkably, GroundingAgent achieves an average zero-shot grounding accuracy of 65.1 % on widely-used benchmarks (RefCOCO, RefCOCO+, RefCOCOg), entirely without fine-tuning. Furthermore, by substituting MLLM-generated captions with the original query texts, the accuracy at the selection stage alone reaches approximately 90 %, closely matching supervised performance and underscoring the critical role of LLM reasoning capabilities. GroundingAgent also offers strong interpretability, transparently illustrating each reasoning step and providing clear insights into its decision-making process.