Enhancing Multi-Image Understanding through Delimiter Token Scaling

AI summary

Overview

• Research area: Multimodal vision-language modeling, specifically large vision-language models (LVLMs) that accept multiple images as input, with a secondary connection to multi-document and multi-table text understanding.
• Technical level: Intermediate. Understanding the work requires familiarity with how LVLMs process interleaved image and text tokens, but the core idea is a targeted, training-free intervention rather than a new architecture.
• Scope: The paper diagnoses why the delimiter tokens that separate images fail to prevent information from bleeding between images, and proposes scaling those tokens' hidden states as a fix evaluated across multi-image and text-only benchmarks.

What This Paper Is About

Large vision-language models handle single images well but degrade when several images are provided at once. The authors identify cross-image information leakage as a major cause: the model mixes information belonging to different images and cannot reliably tell which content came from which input.

Every one of these models already inserts delimiter tokens to mark where each image starts and ends. The paper's central claim is that these existing delimiters do not actually block the leakage, and its goal is to make them work by scaling their hidden states — without any additional training or inference cost.

Key Contributions

1. An analysis of delimiter token failure. The authors examine existing LVLM delimiter tokens and report that, despite being present in the input, they do not effectively stop cross-image information leakage.
2. A delimiter hidden-state scaling method. They propose amplifying the hidden states of delimiter tokens so that the boundary signal is reinforced, strengthening interaction within an image and suppressing interaction across images.
3. Evaluation on multi-image benchmarks. The method is tested on Mantis, MuirBench, MIRB, and QBench2, with the abstract reporting performance gains.
4. Evidence of transfer beyond images. The same method is evaluated on text-only tasks that require clear separation of sources — multi-document and multi-table understanding on TQABench, MultiNews, and WCEP-10 — where it also improves performance. The approach is described as requiring no additional training or inference cost.

Main Findings

• Delimiter tokens are insufficient as-is: The abstract states that the delimiter tokens LVLMs already use to mark image boundaries fail to effectively block cross-image information leakage, which the authors identify as a major reason multi-image performance lags single-image performance.
• Scaling delimiter hidden states changes model behavior: Amplifying these hidden states reinforces intra-image interaction and limits undesired cross-image interaction, helping the model preserve image-specific information.
• Better discrimination and reasoning over images: As a consequence of that preservation, the model is described as better able to distinguish between images and reason over them more accurately.
• Gains on multi-image benchmarks: Improvements are reported on Mantis, MuirBench, MIRB, and QBench2. The abstract does not give the size of these gains or any per-benchmark breakdown.
• Gains on text-only, source-separation tasks: Improvements are also reported on multi-document and multi-table benchmarks — TQABench, MultiNews, and WCEP-10 — suggesting the mechanism is not specific to images.
• No added cost: The method is claimed to require no additional training or inference cost, meaning it is an intervention applied to an existing model rather than a new model or fine-tuning procedure. The abstract provides no latency or memory measurements to accompany this claim.

Methodology in Plain English

The researchers start from an observation rather than a new architecture. LVLMs already place special delimiter tokens around each image to tell the model where one image ends and the next begins. The authors inspect what happens inside the model at those positions and conclude the delimiters are not doing their job — information still flows between images.

Their fix is to take the internal representation (the hidden state) at each delimiter token and scale it up. Delimiter tokens are the natural place to act because they sit exactly at the boundaries the model needs to respect. By making those boundary representations stronger, the model's attention is pushed to keep information within a single image and to avoid pulling content across image boundaries. The result is that each image's information stays more distinct as it moves through the network.

Because the change is applied to representations at inference time rather than to the model's weights, no retraining is needed and the paper claims no additional computational cost. The authors then verify the idea on multi-image benchmarks, and additionally check whether the same boundary-strengthening effect helps in purely textual settings where documents or tables must be kept apart — a test of whether the problem is about vision at all, or about separating distinct information sources in general.

Why This Matters

Impact on research. The paper reframes a well-known weakness of LVLMs — degraded multi-image performance — as a boundary-enforcement problem located at specific, already-existing tokens, rather than a capability gap requiring more data or a larger model. That is a mechanistic framing: a small, precise intervention is claimed to recover performance that would otherwise be pursued through training. It also suggests the distinction between "visual" and "textual" source separation may be less sharp than assumed, since the same fix reportedly helps multi-document and multi-table tasks.

Real-world applications (per the abstract's scope):

• Multi-image visual question answering and comparison, where a user submits several photos and asks the model to reason across them while keeping each photo's content distinct.
• Multi-document summarization and question answering, such as synthesizing several news articles or reports without conflating facts from different sources.
• Multi-table analysis, where several tables must be read together but their figures kept separate.
• Any setting where provenance matters, meaning the model must be able to say which input a given piece of information came from.

Industry relevance. Because the method is claimed to add no training or inference cost, it is attractive for deployment: it can in principle be applied to an already-served model rather than requiring a retraining cycle or additional hardware. Provenance and source attribution are practical requirements in document-heavy workflows, and a cheap intervention that improves them is easier to justify than a fine-tuning program.

Future Directions

• Quantifying the effect. The abstract reports gains without magnitudes, per-benchmark detail, or ablations; establishing how large and how consistent the improvements are, and how sensitive they are to the scaling amount, is the obvious next step. (The abstract does not state a scaling factor or a sensitivity analysis.)
• Understanding the mechanism more deeply. The paper argues that scaled delimiters reinforce intra-image and suppress cross-image interaction; a fuller account of why delimiter hidden states are weak to begin with, and whether leakage originates in specific layers or attention heads, would strengthen the diagnosis.
• Testing generality across models and modalities. The abstract covers specific multi-image and text benchmarks; whether the same delimiter scaling helps other LVLM families, video or interleaved document inputs, or non-visual modalities with explicit boundary markers is left open.
• Reconciling the no-cost claim with real serving conditions. The abstract claims no additional training or inference cost but gives no measurements; verifying this under realistic latency and memory constraints matters for adoption.

Target Audience

Researchers and engineers working on vision-language models, especially those concerned with multi-image inputs, interleaved multimodal context, or long-context document understanding. It is also relevant to practitioners looking for low-cost, training-free improvements to an already-deployed model, and to anyone studying attention behavior and information flow between distinct segments of a model's input. Readers seeking a rigorous empirical comparison with alternative fixes will find fewer details here, since the available abstract reports benchmark names and directions of improvement rather than numbers.

Authors’ abstract

Large Vision-Language Models (LVLMs) achieve strong performance on single-image tasks, but their performance declines when multiple images are provided as input. One major reason is the cross-image information leakage, where the model struggles to distinguish information across different images. Existing LVLMs already employ delimiter tokens to mark the start and end of each image, yet our analysis reveals that these tokens fail to effectively block cross-image information leakage. To enhance their effectiveness, we propose a method that scales the hidden states of delimiter tokens. This enhances the model's ability to preserve image-specific information by reinforcing intra-image interaction and limiting undesired cross-image interactions. Consequently, the model is better able to distinguish between images and reason over them more accurately. Experiments show performance gains on multi-image benchmarks such as Mantis, MuirBench, MIRB, and QBench2. We further evaluate our method on text-only tasks that require clear distinction. The method improves performance on multi-document and multi-table understanding benchmarks, including TQABench, MultiNews, and WCEP-10. Notably, our method requires no additional training or inference cost.