MRD: Multi-resolution Retrieval-Detection Fusion for High-Resolution Image Understanding

AI summary

Overview

Research area: multimodal large language models (MLLMs) and high-resolution (HR) image understanding, specifically retrieval-augmented perception and open-vocabulary object detection.

Technical level: Intermediate. Readers should already be comfortable with MLLM pipelines, retrieval-augmented generation (RAG), patch/crop-based visual encoding, and open-vocabulary detection.

One-sentence scope: The paper introduces MRD, a training-free framework that fuses multi-resolution semantic similarity maps with open-vocabulary detection confidence maps to improve how MLLMs localize and reason about targets in high-resolution images.

What This Paper Is About

MLLMs struggle with high-resolution images because their visual inputs are constrained to fixed, relatively low resolutions. A leading fix is "retrieval-augmented perception," which splits an HR image into crops and retrieves the query-relevant ones — but the retrieval step is unreliable: large objects get split across patches (object fragmentation), the crop resolution is a hard-to-tune hyperparameter, and cluttered backgrounds produce high-similarity false positives. The paper's goal is to make that retrieval step more accurate and robust without any training, by combining cross-scale semantic matching with explicit object-level localization.

Key Contributions

1. A training-free unified multi-scale framework (MRD) for high-resolution image understanding that jointly models local semantic integrity and global spatial grounding to improve retrieval accuracy.

2. Multi-resolution Semantic Fusion: instead of a single fixed crop scale, semantic similarity is computed at multiple proportional resolutions, spatially aligned, and fused to calibrate single-resolution bias and mitigate object fragmentation.

3. Open-vocabulary Detector Enhancement: the open-vocabulary detector LLMDet is integrated as a global localization prior, using LLM-based in-context learning to extract target entities from the query and a sliding-window strategy to build a target-aware detection confidence map that suppresses background false positives.

4. A synergistic linear fusion of the multi-resolution semantic similarity map and the detection confidence map (weight w), which amplifies genuine target responses and speeds up the downstream retrieval-exploration search. Code is stated to be released at https://github.com/yf0412/MRD.

Main Findings

• Two failure modes dominate existing retrieval-based methods. A case-by-case analysis of RAP on V* Bench with LLaVA-ov classified failures as Object Fragmentation (FRAG), Background Interference (BG), and other causes. For RAP, FRAG affects 10.7% of the full dataset (65.2% of all failure cases) and BG affects 8.9% of the full dataset (54.3% of all failure cases). 89.6% of all failure cases are associated with either FRAG or BG.

• Crop resolution strongly affects MLLM performance. Experiments on V* Bench with RAP across several MLLMs showed a resolution of 112 yielded the best performance on both single- and multi-object tasks, but degraded recognition accuracy in several cases due to object fragmentation — motivating adaptive, multi-resolution partitioning.

• MRD reaches state-of-the-art results on both benchmarks. With LLaVA-v1.5-7B, MRD scores 97.4 (Attribute) / 96.1 (Spatial) / 95.6 (Overall) on V* Bench; 76.8 (FSP) / 42.7 (FCP) / 59.9 (Overall) on HR-Bench 4K; and 73.0 / 36.5 / 54.8 on HR-Bench 8K. With LLaVA-ov-0.5B it scores 89.6 / 85.6 / 88.9, 84.0 / 45.2 / 64.6, and 81.8 / 47.3 / 64.5 respectively.

• Absolute gains over the vanilla backbone. On V* Bench, MRD delivers a 46.9% absolute overall improvement over the LLaVA-v1.5-7B baseline (nearly doubling it), plus up to 23.8% and 22.7% absolute overall gains on HR-Bench 4K and HR-Bench 8K.

• Consistent gains over the SOTA RAP baseline. MRD surpasses RAP across all datasets and model settings, with an average overall improvement of 2.8%. On the V* Bench single-object attribute sub-task, MRD gains 53.9% absolute over vanilla LLaVA-v1.5-7B and 7.0% over RAP, attributed to the detection module's precise localization.

• Both modules are complementary and necessary. Ablation on V* Bench with LLaVA-ov-0.5B: OVD alone gives 85.3 overall (+1.7 over RAP's 83.6) but falls on the multi-object spatial task (82.9, −1.3); RAP+Multi-Res gives 85.8 (+2.2); RAP+OVD gives 86.7 (+3.1); full MRD gives 88.9 (+5.3). Error rates drop to BG 4.0 (−62.6%), FRAG 4.4 (−50.6%), and BG/FRAG 7.6 (−48.3%). The full framework outperforms RAP by 5.3% overall.

• The fused map enables a shorter search. MRD fixes the maximum number of search steps to 50 while preserving full model performance. On HR-Bench 4K, search time drops from 55.6 s to 27.5 s (−50.5%) for LLaVA-ov-0.5B and from 52.8 s to 15.2 s (−71.2%) for LLaVA-v1.5-7B; total runtime drops from 67.0 s to 59.6 s (−11.0%) and from 63.4 s to 53.4 s (−26.2%). RAG time rises 53.1% and 48.0% respectively, and peak memory rises from 18.9 GB to 21.2 GB (+12.2%) and from 21.2 GB to 23.4 GB (+10.4%) — the paper states MRD introduces additional GPU memory and compute cost but yields a shorter overall runtime.

• MRD is more stable across crop resolutions. In additional ablations on V* Bench, MRD showed consistently stable performance across tested crop resolutions for both backbones, while RAP was noticeably unstable, particularly with LLaVA-ov-0.5B. MRD was slightly inferior to RAP only for the multi-object task with LLaVA-v1.5-7B at a very small maximum number of search steps.

• More resolution scales do not always help. With LLaVA-ov-0.5B, fusing three scales (k = 1, 2, 4) gave 83.5 / 88.2 / 86.7 (Attribute / Spatial / Overall) versus 83.5 / 85.5 / 85.8 for k = 1, 2. With LLaVA-v1.5-7B, k = 1, 2 gave 94.8 / 96.1 / 94.2 versus 93.9 / 96.1 / 94.2 for k = 1, 2, 4. Multi-resolution fusion beat single-resolution in all cases; the paper adopts k = 1, 2 in practice for performance and efficiency.

Methodology in Plain English

MRD is a training-free wrapper around an existing localize-and-zoom pipeline. It builds two maps over a grid of image crops and combines them.

Local branch — multi-resolution semantic fusion. The HR image is partitioned into two crop sets at proportional resolutions: a low-resolution set P at base resolution l, and a high-resolution set P̂ at l̂ = k·l. Because the resolutions scale proportionally, each HR crop lines up with exactly k² LR crops in a k × k grid. Query and crop embeddings are produced with a vision-language model (following VisRAG), and cosine similarity is computed for both sets. The HR scores are projected onto their corresponding k² LR positions, then fused element-wise with the LR scores via the geometric mean. The result is reshaped into a 2D semantic similarity map. This cross-scale fusion repairs LR scores that were degraded by objects being split across crops.

Global branch — open-vocabulary detector enhancement. An LLM extracts target object entities from the free-form query using in-context learning with a system prompt and few-shot examples. LLMDet then runs sliding-window open-vocabulary detection over the whole HR image, aligned to the same crop grid. Low-confidence boxes are filtered by a threshold τ; each crop receives the maximum confidence of the filtered boxes covering it, giving per-window maps that are averaged into one global detection confidence map to smooth boundary artifacts.

Fusion and search. The semantic similarity map and detection confidence map are linearly combined with balancing weight w to produce the final fused map, which guides target-aware crop selection in the downstream retrieval-exploration search pipeline described in RAP. Default settings: crop resolution 112 (V* Bench), 224 (HR-Bench-4K), 448 (HR-Bench-8K); k = 2; window size/stride pairs of (1232, 896), (2240, 1792), and (3136, 2688) respectively; detection threshold 0.3; w = 0.4; maximum search steps 50.

Why This Matters

Impact on research: the paper reframes HR image understanding as a joint local-semantic and global-spatial problem, and provides a quantified failure taxonomy (FRAG and BG) plus evidence that resolution choice is a major uncontrolled variable in retrieval-based pipelines. Because MRD is training-free, it can be attached to off-the-shelf MLLMs without the cost, convergence time, or cross-architecture transfer problems the paper attributes to SFT and RL approaches.

Real-world applications:

• Document, chart, and screenshot analysis where text and small structures are lost at low resolution.
• Remote sensing and aerial or satellite imagery, where targets of very different scales coexist.
• Medical and microscopy imaging, where fine-grained detail and crowded backgrounds are common.
• Industrial inspection and retail/warehouse scene understanding, where small or partially occluded objects must be located reliably.

Industry relevance: the efficiency results matter for deployment — the fused map halves search time for a 0.5B backbone and cuts it by 71.2% for a 7B backbone on HR-Bench 4K, at the cost of roughly 10 to 12% more peak GPU memory. Any product that already uses a localize-and-zoom or visual-RAG loop over high-resolution images can adopt this as a drop-in retrieval stage.

Future Directions

• Adaptive scale selection. The paper fixes k = 2 and shows that adding a third scale helps the smaller backbone but not the larger one. An open question is whether the number of resolutions should be chosen per image, per query, or per backbone rather than fixed globally.

• Reducing the extra cost. MRD adds 53.1% / 48.0% to the RAG stage and introduces a detection stage that RAP does not have (16.1 s and 15.8 s), plus about 10 to 12% more peak memory. Making the detector branch cheaper without losing the background suppression is a clear next step.

• Improving multi-object behavior. OVD alone and RAP+OVD both trail RAP on the V* Bench spatial (multi-object) task, and MRD is reported as slightly inferior to RAP for multi-object tasks with LLaVA-v1.5-7B at very small search budgets. The paper attributes the OVD weakness to missing objects in complex multi-object scenes, which the detection module does not fully solve.

• Generalizing beyond the tested settings. The hyperparameter ablations are conducted on V* Bench with LLaVA-ov-0.5B and LLaVA-v1.5-7B; the truncated content does not report results for the sliding-window size or detection-confidence-threshold ablations, nor for other backbones or benchmarks, so the robustness of those hyperparameters is not established.

Target Audience

Researchers and engineers working on multimodal large language models, high-resolution or fine-grained visual understanding, retrieval-augmented perception, and open-vocabulary detection. It is most useful to practitioners who need a training-free drop-in improvement for an existing localize-and-zoom or visual-RAG pipeline over high-resolution images, and to researchers interested in the measured failure taxonomy of crop-based retrieval.

Authors’ abstract

Understanding high-resolution (HR) images remains a critical challenge for multimodal large language models (MLLMs). Recent approaches leverage vision-based retrieval-augmented generation (RAG) to retrieve query-relevant crops from HR images, improving understanding capacity of MLLMs. However, this paradigm often leads to object fragmentation, resulting in semantic bias and incomplete retrieval, while also introducing false positives from irrelevant background patches. To address these issues, we propose Multi-resolution Retrieval-Detection (MRD), a training-free framework that enhances HR image understanding from both local and global perspectives. Locally, MRD enforces cross-scale semantic consistency via multi-resolution semantic fusion to mitigate single-resolution bias and alleviate object fragmentation. Globally, it integrates open-vocabulary object detection (OVD) as localization priors within a unified framework. Extensive experiments across multiple MLLMs on HR image benchmarks demonstrate that MRD achieves state-of-the-art (SOTA) performance on both single-object and multi-object understanding tasks. Code will be available at: https://github.com/yf0412/MRD.