Enhancing Compositional Reasoning in CLIP via Reconstruction and Alignment of Text Descriptions

AI summary

Overview

Research area: Vision-language models (VLMs), specifically compositional reasoning in contrastively trained image-text models such as CLIP.

Technical level: Intermediate. The paper builds on contrastive learning and encoder-decoder reconstruction, and assumes familiarity with CLIP-style image-text embedding spaces.

Scope: This paper proposes READ (REconstruction and Alignment of text Descriptions), a fine-tuning method that adds a token-level reconstruction loss and a sentence-level alignment loss to the standard contrastive objective in order to improve the text encoder's compositional reasoning, and evaluates the resulting model, READ-CLIP, on five compositional reasoning benchmarks.

What This Paper Is About

Vision-language models trained with standard contrastive objectives tend to match words to objects in an image but fail to capture how those words relate to each other, so they can rank an incorrect caption such as "the grass is eating the horse" above the correct "the horse is eating the grass". Prior work identifies the text encoder as the main bottleneck, yet most auxiliary-objective methods supervise the image encoder, or both encoders jointly, rather than targeting the text encoder specifically. This paper's goal is to fix the text encoder directly by adding two complementary auxiliary objectives during fine-tuning: reconstructing alternative captions from the caption embedding, and aligning paraphrased captions in the embedding space.

Key Contributions

1. A fine-tuning method (READ) that targets the text encoder. READ augments the standard contrastive loss of CLIP with two auxiliary losses: a token-level reconstruction loss and a sentence-level alignment loss, with the combined objective L_READ = L_Contrastive + α·L_Token Reconstruction + β·L_Sentence Alignment.

2. A token-level reconstruction objective. Using a frozen pre-trained decoder and a learnable projection W from the text embedding, the text encoder is trained so that the decoder can reconstruct each token of an alternative caption (not the original caption) that describes the same image.

3. A sentence-level alignment objective. Paraphrases of a caption are treated as positives in the text embedding space and paraphrases of other samples in the batch as negatives, so the encoder must place differently worded but semantically equivalent captions close together.

4. READ-CLIP and its evaluation. The paper introduces READ-CLIP (READ applied to pre-trained CLIP) and reports state-of-the-art performance across five compositional reasoning benchmarks, plus consistent gains when READ is applied on top of other CLIP variants (CLIP, NegCLIP, FSC-CLIP).

Main Findings

• State-of-the-art average accuracy: READ-CLIP achieves the highest average accuracy of 64.1% across WhatsUp, VALSE, CREPE, SugarCrepe, and SugarCrepe++, improving over the pre-trained CLIP by 12.1%, over NegCLIP by 4.5%, and over the second-best model FSC-CLIP by 1.7%. It ranks first on four benchmarks and second on the remaining one.

• The abstract reports a larger margin in the best case: READ-CLIP is described as outperforming "the strongest conventional fine-tuning baseline by up to 4.1%."

• Both auxiliary losses are complementary: In the ablation study, a contrastive-only baseline averages 62.2% (row 1). Adding only token-level reconstruction reaches 63.9% (row 2) and notably improves WhatsUp by 3.1%, VALSE by 1.9%, and CREPE by 3.2%. Adding only sentence-level alignment reaches 63.8% (row 3) and substantially improves SugarCrepe++ ITT to 70.2% and TOT to 67.0%. Combining both gives the highest average of 64.1% (row 4).

• Reconstruction encourages relational understanding: The paper observes that the reconstruction loss lowers the ranking of negative captions that involve subtle structural edits — swapping, replacing, or inserting single words or short phrases — indicating the encoder is capturing relationships between words.

• Alignment encourages semantic consistency: The alignment loss improves the ranking of positive captions in SugarCrepe++, where each image is paired with two paraphrases (Pos1 and Pos2) that share meaning but differ in wording, by embedding paraphrases closer together.

• Reconstructing an alternative caption beats reconstructing the original: On the SugarCrepe++ TOT (text-to-text) benchmark, reconstructing an alternative caption yields much more stable accuracy across training epochs, whereas reconstructing the original caption produces a gradual decline and a steady drop in similarity between Pos1 and Pos2, which the authors interpret as overfitting to exact wording.

• READ transfers to other fine-tuning methods: Applying READ to naive CLIP raises average accuracy from 57.0% to 58.2%; to NegCLIP from 61.5% to 62.5%; and to FSC-CLIP from 62.8% to 63.6%, with improvements described as up to 2.4% and gains on WhatsUp, VALSE, CREPE, and SugarCrepe while SugarCrepe++ stays comparable.

• Robust to hyperparameters: Performance is stable across values of α (token reconstruction weight), β (sentence alignment weight), and the number K of target sequences, and it remains strong even with a single target sequence (K = 1) and a T5-Large decoder, indicating low computational overhead for the gain.

• Robust to paraphrase noise: Replacing 10% or 20% of LLM-generated paraphrases with unrelated captions drops average accuracy by only 1.2–2.0% (from 64.1% to 62.9% and 62.1%, respectively).

• Paraphrase diversity gives marginal benefit: Increasing the number of LLM-generated paraphrases per caption from 1 to 3 or 5 leaves the average essentially unchanged (64.1%, 64.0%, 64.1%), though SugarCrepe++ improves slightly, suggesting one paraphrase per caption is already sufficient for alignment.

Methodology in Plain English

The authors start from the standard CLIP training setup, where a batch of B image-text pairs is pushed together or apart based on embedding similarity, and modify the image-to-text part of that loss to include M hard negative captions per positive caption.

On top of this contrastive loss they add two auxiliary losses:

• Token-level reconstruction. For each image, they take an original caption, run it through the text encoder to get an embedding, project that embedding with a learnable linear layer into the decoder's input space, and ask a frozen pre-trained decoder to produce a different caption that describes the same image (an "alternative caption"). The loss is the negative log-likelihood of the tokens of that alternative caption. Because the decoder is frozen and only sees the caption embedding, the encoder must preserve the relationships between words in the original caption.

• Sentence-level alignment. They generate a paraphrase of each caption through augmentation and treat the (original, paraphrase) pair as a positive pair, with paraphrases from other samples in the batch as negatives. The loss is a contrastive-style term that pulls the paraphrase embedding toward the original embedding.

The two losses are weighted by hyperparameters α and β and summed with the contrastive loss.

For experiments, all models including the baselines are fine-tuned on a 100K subsample of MS-COCO using the Karpathy split, for 5 training epochs, with a batch size of 256 and the ViT-B/32 architecture. Evaluation uses accuracy on five benchmarks — WhatsUp, CREPE, VALSE, SugarCrepe, and SugarCrepe++ — measuring whether positive image-text pairs are ranked above all negatives, averaged over subtasks where a benchmark has several. Baselines include NegCLIP, GNM-CLIP, Triplet-CLIP, CE-CLIP, and FSC-CLIP. Analysis experiments vary the reconstruction target (original versus alternative caption), inject noise into paraphrases, vary the number of generated paraphrases, and sweep α, β, K, and T5 decoder size.

Why This Matters

Impact on research: The paper argues that the text encoder, though repeatedly identified as the bottleneck for compositional reasoning, has received limited attention from auxiliary-objective approaches. It shows that text-only auxiliary objectives are effective and compose with existing hard-negative-based fine-tuning methods, which suggests a low-cost add-on path for improving VLMs without redesigning the whole training pipeline.

Real-world applications: The introduction notes that VLMs trained with the standard contrastive objective are used for open-vocabulary object detection, semantic segmentation, cross-modal retrieval, and multi-modal generation. Better compositional reasoning would matter in such settings as:

• Image search and retrieval, where queries differ in wording from the captions associated with images.
• Grounding and detection systems that must respect word order and relations, such as "the horse is eating the grass" versus the reversed statement.
• Caption ranking and annotation pipelines for accessibility or media indexing, where paraphrase-invariant embeddings are useful.
• Multi-modal generation and editing pipelines that depend on a text encoder's understanding of relations between described elements.

Industry relevance: The method is a fine-tuning recipe applied on top of existing pre-trained checkpoints rather than a new architecture, and the paper reports that strong results hold with a single target sequence and a moderately sized T5 decoder, which keeps the extra cost modest. It also plugs into NegCLIP and FSC-CLIP without hyperparameter retuning, which lowers the barrier to adoption. Code is available at a GitHub repository linked in the paper.

Future Directions

• Reducing dependence on multi-caption data. The authors note their method is designed to use multiple captions per image; for datasets with only one caption per image, extra paraphrases must be generated with LLMs, which adds complexity.
• Exploring other generative architectures for the decoder. The reconstruction loss was implemented only with a T5 decoder; alternative generative architectures were not explored.
• Training the decoder as well. The authors did not assess what happens if the decoder is fine-tuned instead of kept frozen, which they say may affect compositional reasoning.
• Testing wider transfer and scale. The paper fine-tunes ViT-B/32 on a 100K MS-COCO subsample; whether READ's gains persist across other architectures, dataset scales, and non-COCO training data is left open.

Target Audience

Researchers and practitioners working on vision-language models, contrastive learning, and multimodal representation learning — particularly those interested in compositional reasoning, hard-negative fine-tuning, or auxiliary training objectives. It is also relevant to engineers who want to improve an existing CLIP-style checkpoint for downstream tasks such as retrieval, detection, or captioning, and who need a fine-tuning recipe that is robust to hyperparameter choices and compatible with existing methods. Readers without background in contrastive learning or encoder-decoder language models will need to consult the cited CLIP, NegCLIP, and T5 work first.

Authors’ abstract

Despite recent advances, vision-language models trained with standard contrastive objectives still struggle with compositional reasoning -- the ability to understand structured relationships between visual and linguistic elements. This shortcoming is largely due to the tendency of the text encoder to focus on individual words rather than their relations, a limitation reinforced by contrastive training that primarily aligns words with visual objects. In this paper, we introduce REconstruction and Alignment of text Descriptions (READ), a fine-tuning method designed to enhance compositional reasoning by adding two auxiliary objectives to the contrastive learning: (1) a token-level reconstruction objective, where a frozen pre-trained decoder reconstructs alternative captions based on the embedding of the original caption; and (2) a sentence-level alignment objective, which explicitly aligns paraphrased sentences in the embedding space. We show that READ-CLIP, a model derived by applying the READ method to the pre-trained CLIP model, achieves the state-of-the-art performance across five major compositional reasoning benchmarks, outperforming the strongest conventional fine-tuning baseline by up to 4.1%. Furthermore, applying the READ to existing CLIP variants (including NegCLIP and FSC-CLIP) also improves performance on these benchmarks. Quantitative and qualitative analyses reveal that our proposed objectives -- reconstruction and alignment -- offer complementary benefits: the former encourages the encoder to capture relationships between words within a caption, while the latter ensures consistent representations for paraphrases expressed with different wording.