Stitch and Tell: A Structured Multimodal Data Augmentation Method for Spatial Understanding

AI summary

Overview

Research area: Computer Vision / Vision-Language Models (multimodal data augmentation for spatial understanding). Technical level: Intermediate — readers should be familiar with vision-language model training stages (pretraining and supervised fine-tuning) and standard multimodal benchmarks. Scope: A single paper proposing an annotation-free image-stitching and caption-templating method (SiTe) that injects explicit spatial supervision into multimodal training data, evaluated on three model backbones, two training sets, and thirteen benchmarks.

What This Paper Is About

Vision-language models frequently produce "spatial hallucinations" — incorrect statements about where objects are relative to one another. The authors argue this happens because images carry rich multidimensional spatial structure while their captions are spatially sparse: a survey of common datasets (Table 1) shows spatially-aware samples make up only 0.0168 of Conceptual Captions, 0.0201 of blip_laion_cc_sbu_558K, 0.0288 of VQA_v2, 0.0511 of COCO_2017, 0.0582 of SBU_Captions, 0.0611 of Visual_Genome, 0.0732 of Flickr30K, and 0.1898 of VSR. The goal is to close that gap without manual annotation or costly generative models by stitching images along a spatial axis and generating spatially-aware captions and question-answer pairs from the resulting layout.

Key Contributions

1. A simple, annotation-free, plug-and-play augmentation pipeline called Stitch and Tell (SiTe), composed of two steps: Image-Spatial Stitching and Text-Spatial Telling. It requires no human annotators and no large generative model rewrites, and needs no architecture modification.
2. Two pairing strategies for stitched samples: SiTe_rand (random selection of two images for horizontal concatenation) and SiTe_ratio (filters vertically dominant images with a height-to-width ratio greater than 1.2, buckets them by similar aspect ratio, and pairs within buckets).
3. Automatic generation of two supervision forms from stitched pairs: structured captions built by inserting the original texts into spatial templates (35 templates for horizontal stitching, 29 for vertical stitching), and spatial question-answer pairs built from nouns extracted from the two captions, with answers inferred automatically from which region an entity came from.
4. A demonstration that swapping spatial expressions in the structured captions (for example exchanging "left" and "right") yields hard negative / contrastive samples while preserving global semantics, tested on HALVA.

Main Findings

• Spatial benchmarks improve under pretraining (LLaVA-v1.5-7B, 558K): LLaVA_SiTe-rand raises MME_Pos by +5.50, Spatial-MM by +1.76, COCO-QA_Spat by +1.03, and MM-Vet_Spat by +1.91 over the corresponding baseline; LLaVA_SiTe-ratio raises COCO-QA_Spat by +2.34, Spatial-MM by +2.13, MME_Pos by +4.97, and MM-Vet_Spat by +2.16.
• General benchmarks also improve for LLaVA (pretraining, 558K): LLaVA_SiTe-rand gains +1.02 on COCO-QA, +0.93 on MMBench, +0.07 on VQA-v2, and +0.29 on MM-Vet.
• Flickr30K pretraining: LLaVA_SiTe-rand^flickr gains +2.46 on COCO-QA_Spat and +1.99 on COCO-QA; the paper notes LLaVA_SiTe^flickr improves from 68.75% to 71.42% on COCO-QA_Spat relative to the 558K SiTe setting.
• Widest fine-tuning gain reported for LLaVA: with 5K samples per stitching direction, LLaVA_SiTe-ratio^5K raises Spatial-MM to 48.58 (baseline 42.02), i.e., +6.56, and MME_Pos by +13.17 to 141.00; LLaVA_SiTe-rand^5K raises Spatial-MM from 42.97% to 46.21%, described as +4.19%.
• Qwen2-1.5B fine-tuning: Qwen2_SiTe-ratio^5K improves MM-Vet_Spat by +4.38 and MMBench by +5.33, while COCO-QA drops by 1.86 and COCO-QA_Spat by 2.03.
• Qwen2-1.5B pretraining: Qwen2_SiTe-rand achieves +10.42 on MME_Pos and +5.01 on MMBench; Qwen2_SiTe-ratio achieves +7.25 on MME_Pos and +5.90 on MMBench. On Flickr30K, Qwen2_SiTe-ratio^flickr reaches +1.14 on Spatial-MM and +1.53 on VQA-v2.
• HALVA contrastive setting: HALVA_SiTe beats HALVA_Baseline^* on COCO-QA_Spat (+1.61), Spatial-MM (+1.08), MM-Vet_Spat (+0.40), COCO-QA (+1.42), and MM-Vet (+0.80), but drops on MME_Pos (−11.67), VQA-v2 (−0.64), and MMBench (−0.90). HALVA_SiTe used 20K data versus 21.5K for the baseline.
• Baselines behave differently: LLaVA_Rotate^flickr loses 0.93% on COCO-QA_Spat versus LLaVA^flickr, while LLaVA_VSR reduces MME_Pos to 121.67 (a −6.16 drop from 127.83) in the 558K setting.
• Small-model sensitivity: the authors attribute Qwen2's fine-tuning drop on COCO-QA to over-adaptation to the binary Yes/No format of SiTe QA data, which they say is more pronounced in smaller models like Qwen2-1.5B.
• Real-world and high-resolution benchmarks (Table 3): LLaVA_SiTe-ratio improves CV-Bench 2D (+3.02), CV-Bench 3D (+3.10), RealWorldQA (+0.14), HRBench-8K (+2.82), and V-Star (+1.79); LLaVA_SiTe-ratio^flickr improves CV-Bench 3D by +11.41 and RealWorldQA by +1.74; Qwen2_SiTe-ratio^flickr improves RealWorldQA by +6.54.
• Training efficiency: the paper reports a time reduction of over 20% in the default SiTe setting, and states that an epoch in the pretraining stage takes 77.4% of the baseline's time.
• Qualitative attention analysis: using MLLM-Know visualizations, the authors show cases where LLaVA attends only to the hand and neglects the AI chip, while SiTe attends to both entities, and a case involving the left side of the cake.
• Not reported: no explicit limitations section or conclusion appears in the provided content, and several appendix tables (full template lists) are referenced but not included.

Methodology in Plain English

The method takes pairs of existing image-caption samples and builds new training examples from them in two steps.

First, image stitching: two images are placed on a blank canvas along a horizontal or vertical axis. For horizontal stitching, the canvas height is the maximum of the two heights and the width is the sum of the two widths; the first image is pasted at (0,0) and the second at (w1,0). For vertical stitching, the canvas width is the maximum of the two widths and the height is the sum of the two heights, with pastes at (0,0) and (0,h1). The random variant pairs arbitrary images; the ratio variant first keeps only vertically dominant images (height-to-width ratio greater than 1.2), buckets them by aspect ratio, and pairs within buckets so both halves have comparable geometry.

Second, text telling. The two original captions are inserted into spatial templates, producing a structured caption such as "The left part shows T1, and the right part displays T2," which makes the layout explicit. Separately, nouns are extracted from each caption, overlapping nouns shared by both sides are removed, and entity pairs from the disjoint sets are sampled to form questions like "Is the oL on the left of the oR?" Because each entity's side is known by construction, the answer is assigned automatically without human labeling. Noun phrases for the fine-tuning data are extracted with Qwen2.5-72B-Instruct using a prompt that asks for concrete, visible physical objects and ignores abstract terms like "type", "color", "time", or actions.

The structured captions are intended for the pretraining stage and the QA pairs for instruction tuning. For contrastive learning, swapping the spatial expressions (for example, turning "The bottom image contains T1, and the top image shows T2" into "The top image contains T1, and the bottom image shows T2") creates spatially inconsistent negatives.

Implementation details: horizontal and vertical stitching are applied in a 1:1 ratio, stitched data by default make up one-third of the total set, and image-caption pairs used for stitching are removed from the raw set so each image appears only once. More than 20 question templates were defined, most using camera-perspective phrasing such as "from the camera's point of view, is left_obj located to the left of right_obj?" A pretraining comparison variant substitutes 5K spatially-focused VSR samples for part of the original data. Rotate augments by rotating images randomly between 0° and 360° with captions unchanged; Crop retains 80%–100% of the image from a random region.

Training used LLaVA-v1.5-7B (LLaVA), LLaVA-Qwen2-1.5B (Qwen2) and HALVA-7B (HALVA); LLaVA and Qwen2 were evaluated in both pretraining and fine-tuning, HALVA only in fine-tuning. Pretraining used batch size 16 per GPU for LLaVA and 64 per GPU for Qwen2 on 8 L20Z GPUs for 1 epoch; fine-tuning used batch sizes of 4, 16 and 128 per GPU for HALVA, LLaVA-v1.5-7B and LLaVA-Qwen2-1.5B for 1 epoch. Each setting was run five times and averages are reported. All evaluations are zero-shot.

Why This Matters

The paper's central claim is that explicitly encoding spatial structure into training text is a cheap way to reduce spatial hallucination while keeping general capabilities, and its approach needs no annotators, no generative model rewrites, and no architecture changes.

Real-world applications:

• Robotics and embodied AI, where agents must judge the relative positions of objects to act.
• Autonomous driving and navigation, where correct left/right/front/back judgments are safety-relevant.
• Visual question answering and assistive scene description, where wrong spatial statements mislead users.
• Any pipeline that already trains a vision-language model, since SiTe appears to plug into existing pretraining or fine-tuning data without extra compute — the paper reports epoch time at 77.4% of baseline in its default setting.

Industry relevance: the method is designed for scale, replacing costly spatial annotation or model-generated rewrites with deterministic image concatenation and template filling, which matters for teams that cannot afford large-scale annotation or high-cost synthetic data generation. Its demonstrated benefit on high-resolution and 3D-style benchmarks (HR-Bench 8K, V-Star, CV-Bench, RealWorldQA) suggests applicability beyond the narrow spatial QA setting.

Future Directions

• Fixing the supervision bias seen in small models: Qwen2-1.5B lost ground on COCO-QA during fine-tuning, and the authors suggest the binary Yes/No QA format conflicts with open-ended question styles — a natural next step is a mixed-format or format-aware QA construction.
• Extending stitching beyond left/right and top/bottom layouts, since the current design uses only horizontal and vertical axes and a 1:1 mixing ratio.
• Scaling the contrastive hard-negative construction, which was only tested on HALVA with 20K SiTe samples versus 21.5K baseline samples.
• Understanding why some benchmarks move in opposite directions within the same model, such as HALVA's MME_Pos drop of 11.67 alongside gains elsewhere, and whether spatial supervision transferred during pretraining is more robust than that added only during instruction tuning.

Target Audience

Researchers and engineers working on vision-language models, multimodal data augmentation, and spatial reasoning or hallucination mitigation. It is most useful to practitioners who train or fine-tune VLMs and need a low-cost data-side intervention, and to readers studying how data composition (rather than model architecture) affects spatial grounding. Readers without background in multimodal training pipelines may need to consult the cited benchmarks and prior models first.

Authors’ abstract

Existing vision-language models often suffer from spatial hallucinations, i.e., generating incorrect descriptions about the relative positions of objects in an image. We argue that this problem mainly stems from the asymmetric properties between images and text. To enrich the spatial understanding ability of vision-language models, we propose a simple, annotation-free, plug-and-play method named $\text{Stitch and Tell}$ (abbreviated as SiTe), which injects structured spatial supervision into data. It constructs stitched image-text pairs by stitching images along a spatial axis and generating spatially-aware captions or question answer pairs based on the layout of stitched image, without relying on costly advanced models or human involvement. We evaluate SiTe across three architectures including LLaVA-v1.5-7B, LLaVA-Qwen2-1.5B and HALVA-7B, two training datasets, and eight benchmarks. Experiments show that SiTe improves spatial understanding tasks such as $\text{MME}_{\text{Position}}$ (+5.50%) and Spatial-MM (+4.19%), while maintaining or improving performance on general vision-language benchmarks including COCO-QA (+1.02%) and MMBench (+4.76%). Our findings suggest that explicitly injecting spatially-aware structure into training data offers an effective way to mitigate spatial hallucinations and improve spatial understanding, while preserving general vision-language capabilities.