Leveraging Textual Compositional Reasoning for Robust Change Captioning

AI summary

Overview

Research area: Computer vision, specifically change captioning (generating natural-language descriptions of what changed between two images taken at different times).

Technical level: Intermediate. The paper builds on established vision-language architectures and assumes familiarity with cross-attention, contrastive/loss-based alignment, and standard captioning metrics, but its central idea is explained conceptually.

Scope: The paper introduces CORTEX, a plug-and-play framework that adds explicit textual compositional reasoning (from a Vision Language Model) to existing visual-only change captioning models, and reports gains on three datasets.

What This Paper Is About

Change captioning models must describe meaningful differences between a "before" image and an "after" image while ignoring irrelevant variation such as viewpoint or lighting. Existing methods rely on visual features alone, and the authors argue these features do not explicitly represent structured information such as object relationships, relative attributes, and spatial arrangements. The goal is to supply that missing compositional information as text and fuse it with the visual difference features, improving the accuracy of the generated change descriptions.

Key Contributions

1. A plug-and-play framework (CORTEX). The authors propose CORTEX (Compositional Reasoning-aware Text-guided), which enhances existing visual-only change captioning approaches by injecting explicit textual compositional reasoning that was previously only implicit in images.

2. A Reasoning-aware Text Extraction (RTE) module. This module uses a frozen Vision Language Model (InternVL2 in the main experiments) with a structured prompt to generate compositional reasoning sentences for each image, encoding object attributes and inter-object relationships. The authors state the generated sentences will be publicly released.

3. An Image-Text Dual Alignment (ITDA) module. This module aligns image and text features through static alignment (within the same scene) and dynamic alignment (across scenes), so the model captures both individual scene composition and the differences between scenes.

4. An augmented text dataset (supplementary material). The paper releases RTE-augmented versions of three benchmarks: CLEVR-Change-RTE, CLEVR-DC-RTE, and Spot-the-Diff-RTE, with maximum captions per image of 15, 13, and 16 respectively.

Main Findings

• Consistent improvement on CLEVR-Change (total setting): CORTEX(DIRL) reached BLEU-4 57.4, METEOR 43.0, ROUGE-L 76.2, CIDEr 130.7, and SPICE 34.2, versus DIRL's reproduced baseline of 55.5, 40.8, 73.4, 125.3, and 33.4. CORTEX(SCORER) scored 57.0, 42.7, 75.9, 128.8, 33.9 over SCORER's 56.3, 41.2, 74.5, 126.8, 33.3, and CORTEX(SMART) scored 56.5, 42.1, 75.7, 130.2, 34.0 over SMART's 56.1, 40.8, 74.2, 127.0, 33.4.

• Semantic change setting on CLEVR-Change: CORTEX(DIRL) reached 55.4, 39.6, 74.6, 131.1, 33.5 against DIRL† 55.4, 38.4, 72.1, 123.2, 32.7. The paper defines "semantic change" as cases with actual changes, while "total" includes changed and unchanged cases.

• Robustness to viewpoint shifts (CLEVR-DC): CORTEX improved all three baselines. CORTEX(DIRL) reached BLEU-4 55.3, METEOR 32.9, ROUGE-L 67.8, CIDEr 89.7, SPICE 17.0, versus DIRL's 51.4, 32.3, 66.3, 84.1, 16.8.

• Real-world generalization (Spot-the-Diff): CORTEX(DIRL) reached BLEU-4 11.6, METEOR 13.9, ROUGE-L 33.4, CIDEr 49.5, SPICE 21.4, versus DIRL's 10.3, 13.8, 32.8, 40.9, 19.9.

• Both modules matter: On CLEVR-Change (total, DIRL baseline), the baseline scored 55.5/40.8/73.4/125.3/33.4; adding RTE alone gave 55.8/41.6/74.8/128.5/33.9; adding RTE and ITDA gave 57.4/43.0/76.2/130.7/34.2. ITDA was not tested alone because it depends on the text generated by RTE.

• Both alignment losses contribute: With neither loss, 56.6/41.5/75.1/127.9/33.5; with static loss only, 56.3/41.8/75.5/128.4/34.0; with dynamic loss only, 56.6/41.8/75.6/128.9/33.7; with both, 57.4/43.0/76.2/130.7/34.2.

• Compositional prompts beat generic prompts: Generic descriptions gave 56.5/41.6/75.3/129.5/33.5, while compositional reasoning sentences gave 57.4/43.0/76.2/130.7/34.2, showing that structured cues matter more than having text at all.

• Works across VLMs: Using LLaVA yielded 56.9/42.3/75.7/130.0/34.1 and InternVL2 yielded 57.4/43.0/76.2/130.7/34.2, both above the visual-only baseline of 55.5/40.8/73.4/125.3/33.4.

• Single-image text beats asking a VLM to compare image pairs: On Spot-the-Diff, feeding paired images directly to the VLM for change prediction scored 2.7/10.7/21.0/12.3/12.5, whereas using single-image compositional text as auxiliary context scored 11.6/13.9/33.4/49.5/21.4. The authors attribute this to VLMs being optimized for single-image captioning rather than comparison.

• Modest computational overhead: Offline VLM captioning takes 3.94s per image for CORTEX(DIRL) versus none reported for DIRL. Training is 0.79s per iteration versus 0.77s, inference is 0.008s versus 0.007s, and learnable parameters are 18.2M versus 14.4M.

• Visual relevance: Qualitative comparison against DIRL showed CORTEX generating outputs closer to ground truth by better capturing compositional structure.

Methodology in Plain English

The framework takes a "before" image and an "after" image, extracts visual features with a ResNet-101 backbone, and passes them to an existing image-level change detector (the authors use three publicly available ones: SCORER, SMART, and DIRL). In parallel, a frozen Vision Language Model—InternVL2-8B, with LLaVA also tested—looks at each image separately and, following a structured prompt, writes several sentences per image describing each object's color, shape, size, and its spatial relationship to other objects. The prompt asks for at least one sentence per distinct object and for each sentence to include relationships to at least one other object, so the number of sentences per image varies with scene complexity.

A BERT text encoder turns those sentences into feature vectors. A new alignment module then connects text and images in two ways. Static alignment uses cross-attention to match each image's visual features with the sentences describing that same image, teaching the model the composition of each scene on its own. Dynamic alignment uses cross-attention to match each image's visual features with the sentences describing the other image, which highlights what actually changed. In both cases the text-augmented features are trained to match purely visual cross-attended features via a squared-error alignment loss, so text is used as guidance rather than as a replacement for visual evidence. The aligned features are concatenated and passed to a transformer decoder that writes the change caption.

Training combines the standard captioning loss with the alignment loss, weighted by a balancing term λ, set to 10^-3 for SCORER and 10^-4 for SMART and DIRL, with 8 attention heads. Training used the Adam optimizer on a single RTX 4090 GPU. VLM captions are generated offline before training to avoid runtime cost.

Why This Matters

Impact on research: The work argues that the bottleneck in change captioning is not visual feature quality but the absence of explicitly structured relational information, and shows that a modality-level addition—text—improves three different published baselines. It also provides evidence that VLMs are better used as single-image scene describers than as direct change detectors, an insight relevant beyond this task.

Real-world applications:

• Surveillance, where the paper notes fine-grained difference detection is critical.
• Medical imaging, also cited by the paper as an application where fine-grained visual differences matter.
• Autonomous driving, cited in the related-work discussion of VLM applications.
• Robustness under viewpoint and illumination variation, which the CLEVR-DC experiments address directly, relevant to camera networks and infrastructure monitoring.

Industry relevance: Because CORTEX is designed as two plug-and-play modules that attach to existing change detectors, it can be added to deployed visual-only systems without architectural replacement. The overhead is a one-time offline VLM pass plus a 3.8M increase in learnable parameters relative to DIRL, and the code is available at https://github.com/VisualAIKHU/CORTEX.

Future Directions

1. Reducing VLM overhead. The authors state in the limitations that using VLMs incurs computational overhead and that future work will aim to reduce it for practical applications.

2. Improving on direct paired-image VLM reasoning. Since directly prompting a VLM with image pairs performed poorly on Spot-the-Diff (CIDEr 12.3 versus 49.5 for the auxiliary-text approach), the paper leaves open how or whether paired-image VLM reasoning can be made competitive.

3. Recruiting more VLMs and text encoders. The main experiments use InternVL2-8B and BERT, with LLaVA evaluated as an alternative VLM; the supplementary material lists an effect-of-text-encoder study and a human evaluation of the generated compositional text, indicating more systematic evaluation of these components.

4. Determining the amount of generated text. The number of sentences per image is decided dynamically by scene, with observed maxima of 15, 13, and 16 for the three RTE-augmented datasets, and the paper notes this is governed by prompt tuning rather than a fixed rule.

Target Audience

Researchers and graduate students working on change captioning, image difference description, or vision-language fusion will get the most from this paper, especially those who already work with SCORER, SMART, or DIRL and want a lightweight add-on. It is also useful for practitioners who need to detect and describe changes in surveillance or monitoring imagery and who want to know whether prompting a VLM to compare two images works, or whether single-image text descriptions are the better route. Readers without background in cross-attention or captioning metrics will find the ideas accessible but the equations and metric comparisons dense.

Authors’ abstract

Change captioning aims to describe changes between a pair of images. However, existing works rely on visual features alone, which often fail to capture subtle but meaningful changes because they lack the ability to represent explicitly structured information such as object relationships and compositional semantics. To alleviate this, we present CORTEX (COmpositional Reasoning-aware TEXt-guided), a novel framework that integrates complementary textual cues to enhance change understanding. In addition to capturing cues from pixel-level differences, CORTEX utilizes scene-level textual knowledge provided by Vision Language Models (VLMs) to extract richer image text signals that reveal underlying compositional reasoning. CORTEX consists of three key modules: (i) an Image-level Change Detector that identifies low-level visual differences between paired images, (ii) a Reasoning-aware Text Extraction (RTE) module that use VLMs to generate compositional reasoning descriptions implicit in visual features, and (iii) an Image-Text Dual Alignment (ITDA) module that aligns visual and textual features for fine-grained relational reasoning. This enables CORTEX to reason over visual and textual features and capture changes that are otherwise ambiguous in visual features alone.