Positional Bias in Multimodal Embedding Models: Do They Favor the Beginning, the Middle, or the End?

AI summary

Overview

Research area: Multimodal representation learning — specifically positional bias in image-text retrieval models built on CLIP-style architectures.

Technical level: Intermediate. The paper assumes familiarity with CLIP-style contrastive embedding models, positional encodings, and cross-modal retrieval metrics, but the experimental logic is explained in accessible terms.

Scope: An empirical study of where positional bias appears in the text and image encoders of multimodal embedding models, how the bias varies across models and datasets, and which training and architectural factors contribute to it.

What This Paper Is About

Prior work has documented positional bias — the tendency of a model to overemphasize certain positions in an input regardless of content — in text generation models (the "lost in the middle" effect) and in text embedding models. This paper asks whether the same phenomenon exists in multimodal representation models, specifically CLIP and its variants, and how it differs between the text encoder and the image encoder. The authors separate positional bias from genuine contextual importance (some positions really do carry more useful content) and then test which factors, such as positional encoding scheme, training loss, model size, and the use of image-text pairs, drive or amplify the bias.

Key Contributions

1. First report of positional bias in multimodal representation models. The authors state that this phenomenon had not previously been reported for multimodal embedding models, to their knowledge.

2. Characterization of distinct bias patterns per modality. Text encoders consistently perform best when a segment is placed at the beginning of the sequence; image encoders show bias at both the beginning and the end, with the beginning bias typically stronger.

3. Demonstration that the bias is widespread. Experiments across many models show the bias persists regardless of data distribution, positional encoding, training length, model size, resolution, patch size, loss function, or vision encoder architecture.

4. Identification of contributing factors. The paper isolates several drivers or amplifiers: positional encoding scheme, training loss, contextual importance of the data, and the use of image-text pairs during multimodal training.

Main Findings

• Text encoders favor the beginning. On Urban1K with Long-CLIP (ViT-B/16), captions were split into six segments (six is the modal number of sub-captions in the data). Retrieval accuracy is consistently highest when any segment is placed at position 0, regardless of where that segment originally appeared. The same trend appears under a text-perturbation strategy using Lorem Ipsum placeholders.

• Image encoders favor the beginning and the end. Using seven image splits (an odd number so the central segment occupies a distinct middle position), Long-CLIP's image encoder showed bias toward both ends for 5 out of 7 segments, with the beginning bias notably stronger. This is unexpected because the central region is the most semantically informative.

• Contextual importance does not explain the bias. Contextual importance analysis showed that for text, accuracy is highest with the first segment and declines monotonically; for images, the central region is most important with a gradual drop toward the edges. The image-side bias (beginning and end) therefore does not align with the image-side importance profile (center).

• Shuffled training does not eliminate text bias but weakens it. Shuffled Long-CLIP was trained on captions whose sub-captions were randomly reordered at sentence boundaries. Shifting the first segment from the beginning to the end caused a drop of 0.199 (from 0.581 to 0.382), compared to a larger drop of 0.303 (from 0.581 to 0.278) for the original Long-CLIP. Contextual importance contributes to the bias to some extent but does not fully account for it.

• Absolute positional encoding mitigates but does not remove bias. TULIP, which uses relative (rotary) positional encoding, showed a stronger bias than Long-CLIP. On Urban1K, shifting the first segment from position 0 to position 2 dropped TULIP's retrieval accuracy from 0.66 to 0.065, while Long-CLIP declined only from 0.582 to 0.485. Both text and image encoders in Long-CLIP use the same absolute encoding, yet they exhibit different bias patterns.

• Bias persists with short captions. On COCO, where captions average 11.53 valid tokens, CLIP-ViT-B/16 was evaluated on the first 12 tokens split into three segments and shifted across five positions. Positional bias remained, favoring early positions in text and both ends in images, though the effect was weaker.

• Larger models can show stronger bias. CLIP-ViT-L/14 exhibited stronger positional bias in both encoders than CLIP-ViT-B/16, as measured by higher coefficient of variation across positions (reported in the supplementary Table 1). CLIP-ViT-L/14 also retained more positional information when a segment was shifted back to its original position, a pattern shared with TULIP, which is also based on ViT-L/14.

• Smaller patches and higher resolution reduce vision-side bias. CLIP-ViT-B/32 showed stronger image-encoder bias than ViT-B/16. CLIP-ViT-L/14@336 exhibited a lower coefficient of variation across visual segment positions in 5 out of 7 cases compared to CLIP-ViT-L/14.

• Loss function changes the bias pattern. SigLIP, which uses a sigmoid cross-entropy loss instead of softmax contrastive loss, still shows positional bias in both encoders, but its image-side pattern differs: it shows a prominent bias toward the beginning rather than both ends.

• Bias is not exclusive to transformers. CLIP-ResNet-50, whose vision encoder is a CNN, still showed positional bias with patterns resembling the ViT-based image encoders.

• Image-text training amplifies visual bias. On ImageNet Top-1 classification, a standalone ResNet classification model showed relatively stable accuracy across positions (consistent with CNN translation invariance), while CLIP-ResNet-50 zero-shot classification showed clear beginning-and-end bias. The same comparison with ViT-B/16 showed significantly more pronounced bias under the CLIP framework, suggesting the image-side bias originates from or is amplified by training on image-text pairs.

Methodology in Plain English

The authors separate two confounded ideas: positional bias (a model favoring a position irrespective of content) and contextual importance (some positions genuinely hold more useful content). To measure contextual importance, they split an input into uniform segments, keep only one segment, mask the rest, and measure retrieval accuracy — holding the other modality fixed. Text is masked with the model's padding token; images are masked using CLIP's RGB mean values ([0.481, 0.458, 0.408]). Different split counts produce different granularities, so values are interpolated onto a common scale for comparison.

To measure positional bias, they take a single segment and move it across positions while masking or replacing the rest — either with the padding token or with Lorem Ipsum dummy text. Prior work shuffled whole documents while preserving each document's content; this paper deliberately uses a single segment per input because the models have fixed context windows, and because in their setting all segments are relevant to the query, so treating any as an irrelevant distractor would be inappropriate. Absolute retrieval scores are not the point; the point is how accuracy changes with position.

They evaluate several models selected to isolate specific factors: Long-CLIP (absolute positional encoding, 248-token window), TULIP (relative/rotary encoding), Shuffled Long-CLIP (a re-trained variant with reordered sub-captions), the official CLIP checkpoints (ViT-B/32, B/16, L/14, L/14-336, ResNet-50) for architecture, patch size, resolution, and scale comparisons, and SigLIP-Base to test the role of loss function. Long-CLIP was reproduced with mixed precision disabled and training on 4 GPUs with batch size 128. Evaluation uses Recall@1 for image-to-text and text-to-image retrieval on long-caption datasets, Recall@10 on COCO where Recall@1 would be too low for meaningful comparison, and Top-1 accuracy for classification.

Why This Matters

Impact on research. Positional bias has been extensively studied in text generation and text embedding, but this paper shows the problem carries over into multimodal representation models, sometimes in modality-specific ways that existing explanations do not predict. In particular, the finding that image encoders favor the beginning and end while the most informative visual content is central challenges the common assumption that positional bias simply mirrors data statistics. The result that a CNN vision encoder inside CLIP still shows bias also weakens purely architectural explanations and points toward training data and objective as key drivers.

Real-world applications:

• Text-to-image and image-to-text search: systems that rank results by embedding similarity can systematically miss relevant content depending on where that content sits in a caption or image, affecting user-facing search quality.

• Retrieval-augmented generation (RAG): cross-modal retrieval feeds context into generation pipelines; biased retrieval can supply the wrong visual or textual context and degrade downstream generation.

• Content and media organization: automatic tagging, cataloging, and accessibility captioning for image libraries rely on embedding models whose ranking behavior varies with content position.

• Zero-shot classification and monitoring: CLIP-style backbones used for zero-shot labeling can produce position-dependent predictions, which matters for datasets where subjects are not centered.

Industry relevance. CLIP-style models are widely used as backbones for larger multimodal systems, so bias in their embeddings propagates into downstream products. Understanding which factors reduce the bias — smaller patch size, higher resolution, absolute rather than relative positional encoding — gives practitioners concrete knobs to consider.

Future Directions

1. Investigate the underlying causes of positional bias more deeply, since the paper shows multiple contributing factors (positional encoding, training loss, contextual importance, image-text pair training) rather than a single root cause.

2. Develop mitigation strategies. The authors suggest adapting mechanisms like the Differential Transformer, which uses differential attention to reduce attention noise in causal models, to bidirectional multimodal representations.

3. Test whether the non-transformer findings generalize. The paper notes that future work could explore whether the patterns hold when both the image and text encoders are non-transformer-based, since only the vision side was tested with a CNN.

4. Explain why loss functions produce different image-side patterns. Contrastive loss and sigmoid loss led to different bias shapes in the image encoder, a difference the paper reports but does not fully resolve.

Target Audience

Researchers and engineers working on multimodal representation learning, cross-modal retrieval, and vision-language systems built on CLIP-style architectures. It is also relevant to practitioners deploying image-text search or RAG pipelines who need to understand failure modes in embedding models, and to researchers studying positional bias in language models who want to see how the phenomenon transfers to the multimodal setting.

Authors’ abstract

Positional bias - where models overemphasize certain positions regardless of content - has been shown to negatively impact model performance across various tasks. While recent research has extensively examined positional bias in text generation models, its presence and effects in representation models remain underexplored. Even less is known about such biases in multimodal models. In this work, we investigate positional bias in multimodal representation models, specifically in the context of image-text retrieval. We begin by distinguishing between context importance and positional bias, and then assess the presence and extent of positional bias across different models and datasets. Our experiments demonstrate that positional bias is prevalent in multimodal models, but manifests differently across modalities: text encoders tend to exhibit bias toward the beginning of the input, whereas image encoders show bias at both the beginning and end. Furthermore, we find that this bias arises from, or is amplified by, a combination of factors, including the positional encoding scheme, training loss, context importance, and the nature of using image-text pairs in multimodal training.