Conflict Adaptation in Vision-Language Models

AI summary

Overview

• Research area: Cognitive science-inspired evaluation of vision-language models (computer vision / multimodal LLMs), specifically sequential Stroop conflict adaptation combined with mechanistic interpretability.
• Technical level: Advanced. The behavioral experiments are accessible, but the second half relies on sparse autoencoders (transcoders), coactivation-based feature grouping, and causal ablation.
• Scope: The paper tests whether 13 vision-language models show human-like conflict adaptation on a sequential Stroop task, then uses sparse autoencoders to locate the internal features that carry color, text, and conflict information in one of those models.

What This Paper Is About

In humans, performance on a hard (incongruent) trial improves when it follows another hard trial rather than an easy one — a phenomenon called conflict adaptation, thought to reflect how a limited pool of cognitive control gets recruited. The paper asks whether vision-language models show the same sequential pattern, and whether the internal features responsible can be identified and causally verified. It does this by running models through a two-word sequential Stroop task and then dissecting one model's activations with sparse autoencoders and feature ablation.

Key Contributions

1. A sequential Stroop benchmark for VLMs. The authors build a task in which each image contains two color words printed in colored fonts, arranged left-to-right or top-to-bottom, where each word's font color may match or mismatch its meaning. The color set is red, blue, green, yellow, pink, and brown, extending beyond the traditional red-blue-green paradigm, and the two words share no colors in either modality. This yields 30 CC, 120 CI, 120 IC, and 360 II sequences per arrangement.
2. A multi-model behavioral survey. Thirteen leading open-source VLMs from the Gemma, InternVL, Molmo, and Qwen families are evaluated on all sequences in both spatial arrangements, with performance measured by the log probabilities assigned to correct second color tokens.
3. A feature-level account of color and text processing. Using transcoders (a variant of sparse autoencoders) on InternVL 3.5 4B, the authors extract "supernodes" for the color red and the text RED, showing partially overlapping but partly modality-specific feature sets in both early and late layers.
4. A causal conflict feature. They isolate a conflict-modulated supernode in layers 24–25 whose activation is more elevated under II than CI, and show that ablating it selectively damages incongruent performance.

Main Findings

• Conflict adaptation is widespread. 12 of 13 tested VLMs show higher log probabilities for the correct second color token under II (incongruent following incongruent) than under CI (incongruent following congruent), and this pattern holds for both the left-right and top-down arrangements.
• One model reverses the pattern, likely due to a ceiling. Qwen2.5 VL 72B Instruct shows the opposite pattern. Both it and Qwen2.5 VL 32B Instruct, a model half its size from the same family, achieve 100% accuracy across all conditions, so the authors suspect the task became trivially easy, creating a ceiling effect.
• Spatial arrangement does not matter. The observed behavior pattern is not meaningfully affected by whether words are arranged left-to-right or top-to-bottom.
• Color and text form separate but overlapping supernodes. The color red yields supernodes in layers 8–11 and 19–33; the text RED yields supernodes in layers 3–15 and 19–33. They partially overlap in both early and late layers, meaning there are both shared and modality-specific features.
• The size asymmetry mirrors human automaticity. In early layers the text supernode contains more features than the color supernode; in late layers the reverse is true. The authors note this mirrors reading being more automatic than color naming in humans even without interference.
• Conflict features look different from color/text features. Color and text features are sparse and interpretable (their top activating texts largely mention the corresponding colors), whereas the conflict-modulated features are dense, activate frequently across diverse contexts, and lack clear interpretable descriptions — suggesting a general rather than task-specific function.
• Ablation of the conflict supernode selectively breaks incongruent trials. Zeroing the layers 24–25 conflict-modulated supernode increases Stroop errors 3.38-fold on CI trials (17.5% to 59.2%) and 8.33-fold on II trials (2.5% to 20.8%), with minimal effect on congruent trials (CC, IC).
• A new failure mode appears after ablation. Post-ablation, the model sometimes repeatedly outputs the word "ink" in place of the second color — a failure mode entirely absent pre-ablation.

Methodology in Plain English

The authors first designed a task that mimics the classic Stroop test but with a sequential twist: models see two colored words and must report the ink colors, not the words. Because each image is built so the two words never share colors, the first trial's properties can be varied independently of the second, letting the researchers compare "hard after easy" against "hard after hard." Models were given explicit system prompts telling them to name ink colors only, not read the words, and to answer in exactly two words; models without system prompt support received abbreviated instructions after the image but still performed well.

Performance was measured not by exact-match accuracy alone but by the log probabilities the model assigned to the correct second color token, which gives a finer-grained signal than right/wrong.

For the interpretability half, the authors used transcoders — a variant of sparse autoencoders that decompose dense neural activations into sparse features responding to specific patterns — on InternVL 3.5 4B, whose language component Qwen 3 4B has published transcoders available. They built "summary tensors" by averaging activations over specific trial types and taking differences (for example, trials where the first color is red versus trials where it is not), then applied coactivation-based grouping to find sets of features that activate together across tokens and layers — these connected components are called supernodes. Finally, they validated importance causally: setting a supernode's features to zero during the forward pass and measuring how the output distribution changes.

Why This Matters

The paper connects a classic signature of human cognitive control to the internal workings of multimodal models, and it does so with a causal test rather than only a behavioral one. Showing that a specific, localized set of features drives conflict handling — and that removing them selectively harms incongruent trials — gives a mechanistic foothold for claims about "conflict adaptation" in artificial systems. It also contributes a richer benchmark: sequential design, six colors, and log-probability-based measurement rather than binary accuracy.

Real-world applications:

• Multimodal agent reliability: Models that resolve conflicting visual and textual signals well are more dependable in settings where an image and its caption disagree.
• Evaluation design for VLMs: The sequential paradigm and fine-grained log-probability metric offer a template for stress-testing models beyond single-trial accuracy.
• Interpretability-guided debugging: The ablation method shows how to locate and test the specific features responsible for a behavior, useful for auditing models before deployment.
• Cognitive-science modeling: The finding that feature asymmetry mirrors human reading-versus-color-naming automaticity suggests VLMs could serve as testbeds for theories of cognitive control.

Industry relevance: The work speaks to teams building multimodal assistants that must follow instructions over distracting inputs, and to anyone who needs interpretable evidence, not just benchmark scores, that a model is handling conflicting information for the right reasons.

Future Directions

• Disentangling conflict adaptation from task comprehension. The paper's own stated limitation is that it cannot rule out that conflict helps models better understand what the task demands; what looks like adaptation may partly be conflict clarifying instructions.
• Testing whether the conflict supernode generalizes. The conflict-modulated features are dense and lack interpretable descriptions, so it remains open what general function they serve beyond this task.
• Extending the analysis beyond a single model. The feature-level work covers only InternVL 3.5 4B; whether the same supernode structure appears in the other 12 models is not addressed.
• Explaining the Qwen2.5 VL 72B reversal. The ceiling-effect explanation is a suspicion rather than a demonstration; a harder task variant could test it directly.

Target Audience

Researchers working at the intersection of cognitive science and multimodal AI, interpretability practitioners interested in sparse autoencoders and causal ablation on vision-language models, and evaluation engineers who want finer-grained behavioral probes than single-trial accuracy. Readers without background in sparse autoencoders or feature attribution will find the behavioral results accessible but the interpretability half demanding.

Authors’ abstract

A signature of human cognitive control is conflict adaptation: improved performance on a high-conflict trial following another high-conflict trial. This phenomenon offers an account for how cognitive control, a scarce resource, is recruited. Using a sequential Stroop task, we find that 12 of 13 vision-language models (VLMs) tested exhibit behavior consistent with conflict adaptation, with the lone exception likely reflecting a ceiling effect. To understand the…