Spatial Preference Rewarding for MLLMs Spatial Understanding

AI summary

Overview

Research area: Multimodal large language models (MLLMs), specifically fine-grained spatial understanding (region description, referring expression comprehension, phrase grounding) and preference-based alignment.

Technical level: Advanced. The paper assumes familiarity with MLLM training pipelines, CLIP-style contrastive embeddings, object detection, IoU-based evaluation, and Direct Preference Optimization (DPO).

Scope: The paper introduces SPR (Spatial Preference Rewarding), a DPO-based framework that builds its own preference data from MLLM-generated grounded region descriptions and uses combined semantic and localization scores to reward precise object localization over vague or inaccurate responses.

What This Paper Is About

MLLMs can describe and localize objects, but their responses are often vague, distracted by objects outside the queried region, or imprecisely localized—and they frequently ignore the user's request for fine-grained spatial detail. The authors argue this happens because existing training methods (supervised instruction tuning) only teach models to imitate positive examples and never penalize inaccurate localization during real inference. SPR closes this gap by scoring MLLM responses with semantic and localization rewards, then training the model with DPO to prefer accurate, detailed grounded descriptions over weak ones.

Key Contributions

1. A Spatial Preference Rewarding (SPR) framework that applies direct preference optimization to fine-grained spatial understanding, explicitly contrasting high-quality object localization (positive samples) against inaccurate localization (negative samples)—mirroring the positive/negative sample mechanism long used in traditional object detection.

2. An automated preference-data pipeline with no human annotators or external MLLMs. It constructs random multi-object region queries from object annotations, prompts MLLMs in diverse ways for grounded region descriptions, and ranks responses using a semantic score and a localization score.

3. Introduction of a semantic score (CLIP-based, averaging a cropped-region similarity with a full-image similarity using a local-attention layer) and a localization score (ground-truth-matched average IoU computed with Grounding DINO), combined as S = λ·S_sem + (1−λ)·S_loc with λ = 0.8.

4. Grounded region description refinement, which replaces matched predicted boxes (IoU > 0.5) in the preferred descriptions with their matched ground-truth boxes to widen the localization gap between preferred and rejected data, plus extensive experiments across three MLLMs and multiple benchmarks.

Main Findings

• Referring expression comprehension improves on all three tested MLLMs. On RefCOCO val, Ferret-7B rises from 87.49 to 88.39, Ferret-13B from 89.48 to 89.94, LLaVA-OneVision (LLava-OV-7B) from 74.77 to 76.66, and CogVLM-Grounding-17B from 92.76 to 92.95.

• Gains grow as the IoU threshold rises. On the average of RefCOCO/+/g, Ferret-7B improves from 43.25 to 52.21 (an 8.96 gain) and Ferret-13B from 49.55 to 53.61 (a 4.06 gain) at IoU 0.9, compared with much smaller gains at IoU 0.5 (83.91→84.93 and 85.56→86.18).

• Phrase grounding benefits, especially for the smaller model. Averaged over the Flickr30k validation and test sets, the maximum gains are 4.01 Recall@1 for Ferret-7B (29.98→33.99 at IoU 0.9) and 1.58 for Ferret-13B (32.60→34.18). The paper reports that Ferret-7B with SPR becomes comparable to the 13B model. One caveat: on the single Ferret-13B Flickr30k test split in Table 1, the score decreases slightly (84.76 → 83.75).

• Region captioning text quality improves. On RefCOCOg, Ferret-7B's METEOR rises from 12.3 to 13.5 and ROUGE_L from 15.6 to 20.4; Ferret-13B's ROUGE_L rises from 26.4 to 27.2. On Visual Genome, gains are smaller (Ferret-7B METEOR 17.4→17.6, ROUGE_L 29.6→29.7).

• Fine-grained conversational ability improves on Ferret Bench. Ferret-7B's average rises from 64.5 to 65.5; Ferret-13B's from 66.3 to 67.9, with the largest single gain in Referring Reasoning (68.7 → 72.6, about 3.9 for the 13B model).

• General and hallucination benchmarks do not degrade. LLaVA-OneVision improves on TextVQA (75.89→76.07), GQA (62.21→62.42), LLaVA-Bench (88.9→91.4), and POPE F1 (88.12→88.49); Ferret-7B improves on LLaVA-Bench (64.7→66.3) and POPE (85.36→85.69). The authors attribute the hallucination reduction to the semantic and localization scores rejecting descriptions of content outside the queried region.

• DPO outperforms plain instruction fine-tuning on the same data. For Ferret-7B, instruction fine-tuning gives 84.35 REC / 81.72 phrase grounding, while DPO gives 84.93 / 82.44, against a baseline of 83.91 / 81.30.

• Ablations support the design choices. Removing description refinement lowers the REC score from 84.93 to 84.41 (the paper's printed phrase-grounding value for this row, 91.38, is inconsistent with the surrounding results). Varying λ shows that 0.8 is best on both tasks (REC 84.93, grounding 82.44), with λ = 0.0 (84.25 / 81.83) and λ = 1.0 (84.45 / 81.87) both weaker.

• Training overhead is minimal. Preference data uses 10k randomly selected Objects365 training images; LORA tuning runs on a single A100 GPU, taking around 3 hours for Ferret-7B and 5 hours for Ferret-13B.

Methodology in Plain English

The pipeline has three stages. First, it manufactures hard query regions instead of using existing region-description datasets, which the authors found too simple (often one or two objects and short phrases). Starting from annotated bounding boxes in Objects365, it picks a random object and grows the region by adding nearest neighbors, stopping randomly once more than four objects are involved. It then prompts the MLLM with a variety of prompts, including cropped region images and object references, to produce several grounded region descriptions per image.

Second, it ranks those candidate descriptions using two scores. The semantic score asks a CLIP model how well the description text matches the region image; to avoid losing context, the authors average the score from the cropped region with a score from the full image where the vision encoder's final aggregation layer is replaced by a local-attention layer. The localization score asks how many of the objects mentioned in the description are correctly boxed: Grounding DINO extracts boxes for the mentioned objects and for the plain text, these are merged with the annotation boxes, and the score is the average IoU over ground-truth boxes after filtering matches below 0.5. The two are combined with λ = 0.8 weighting the semantic score.

Third, the top-scoring refined description becomes the "preferred" response and the lowest-scoring one the "rejected" response. Before pairing, the authors refine the preferred response by replacing matched predicted boxes with their ground-truth counterparts, reinserting the corrected coordinates, and removing duplicates. The MLLM is then tuned with DPO using LORA, where the frozen instruction-tuned model acts as the reference policy and only the LORA weights are updated.

Why This Matters

Research impact. The paper argues that instruction fine-tuning alone cannot penalize bad localization, and it is one of the first works to apply preference optimization specifically to fine-grained spatial alignment rather than image-level hallucination reduction. It also shows that preference data for spatial tasks can be generated automatically from annotations, without human annotators or a stronger judge model, which makes the approach scalable.

Real-world applications (as motivated by the paper's task set):

• Visual assistants that describe a user-specified region of an image in detail, such as a highlighted area of a photo or screenshot.
• Robotic or embodied systems that need to locate and refer to multiple named objects in a scene accurately.
• Accessibility tools that narrate and point to regions of an image for blind or low-vision users.
• Document or scene inspection tools that must ground every object mentioned in a generated description to a precise location.

Industry relevance. The method trains with LORA on a single A100 GPU in roughly 3 to 5 hours, so it is a low-cost add-on that can be layered onto existing grounded MLLMs such as Ferret, LLaVA-OneVision, and CogVLM-Grounding. The paper reports that the approach also reduces hallucination (POPE F1 improvements) and does not hurt general VQA or conversational benchmarks, which matters for deployment.

Future Directions

• Combining SFT and DPO. The authors state explicitly that DPO depends on a well-trained SFT model and that they plan to explore integrating the two training stages.
• Extending the preference pipeline beyond region description. Open question: whether the same semantic-plus-localization scoring could reward other spatial tasks, such as dense detection or grounded reasoning at scale.
• Scaling and generalizing the preference data. The current setup uses 10k Objects365 images and three MLLMs; whether larger preference sets or other datasets produce further gains is not reported.
• Reducing reliance on auxiliary models. Semantic scoring uses CLIP and localization scoring uses Grounding DINO; although the paper emphasizes no human labor or external MLLMs are needed, it is not reported how sensitive results are to the quality of these auxiliary components.

Target Audience

Researchers and engineers working on multimodal LLMs, visual grounding, and preference optimization, particularly those interested in aligning model outputs to human preferences for fine-grained perception rather than broad image-level captions. It is also relevant to practitioners who need to add spatial grounding to an existing MLLM under tight compute budgets, and to readers interested in how traditional object-detection ideas (positive and negative samples, IoU thresholds) can be transplanted into language-model alignment.

Authors’ abstract

Multimodal large language models~(MLLMs) have demonstrated promising spatial understanding capabilities, such as referencing and grounding object descriptions. Despite their successes, MLLMs still fall short in fine-grained spatial perception abilities, such as generating detailed region descriptions or accurately localizing objects. Additionally, they often fail to respond to the user's requirements for desired fine-grained spatial understanding. This issue might arise because existing approaches primarily focus on tuning MLLMs to model pre-annotated instruction data to inject spatial knowledge, without direct supervision of MLLMs' actual responses. We address this issue by SPR, a Spatial Preference Rewarding~(SPR) approach that enhances MLLMs' spatial capabilities by rewarding MLLMs' detailed responses with precise object localization over vague or inaccurate responses. With randomly selected image regions and region descriptions from MLLMs, SPR introduces semantic and localization scores to comprehensively evaluate the text quality and localization quality in MLLM-generated descriptions. We also refine the MLLM descriptions with better localization accuracy and pair the best-scored refinement with the initial descriptions of the lowest score for direct preference optimization, thereby enhancing fine-grained alignment with visual input. Extensive experiments over standard referring and grounding benchmarks show that SPR improves MLLM spatial understanding capabilities effectively with minimal overhead in training. Data and code will be released at https://github.com/hanqiu-hq/SPR