Canonical Color as a Lens into Concept Decodability in Vision Encoders and VLMs

AI summary

Overview

Research area: Computer Vision / Vision-Language Models — specifically, interpretability and representation probing of frozen visual encoders and the vision towers inside VLMs.

Technical level: Intermediate. The methods (linear probing, frozen feature extraction, cross-validation) are standard in representation-analysis research, but familiarity with vision transformers (CLIP, SigLIP, DINOv2, ViT-MAE, Swin-V2) and VLM architectures is assumed.

Scope: A controlled probing study of whether "canonical" object color (e.g., bananas are yellow) is encoded as conceptual, object-level knowledge in vision encoders and where that information becomes linearly accessible across the VLM stack.

What This Paper Is About

Vision encoders turn images into representations, but it is unclear whether they encode conceptual knowledge — like "bananas are typically yellow" — or merely reflect the visible pixels in a given image. The authors use canonical color as a clean test case because color can be physically removed from the input by converting images to grayscale, while object structure remains intact. They then ask whether canonical color is still linearly decodable from frozen representations after visible chromatic cues are stripped away, how it relates to object identity, and how VLM post-training relocates this information.

Key Contributions

1. A controlled canonical-color dataset: 708 object classes, each mapped to one of 10 basic canonical colors (black, blue, brown, gray, green, pink, purple, red, white, yellow), built by extending the Visual-CounterFact object–color pairs with Wikidata has-color (P462) entries, filtered to single-valued labels and manually screened for image quality.

2. An input-side control protocol for separating canonical color from surface color: grayscale conversion plus per-image histogram equalization to remove chromatic cues and reduce luminance cues, validated against counterfactual (recolored) images.

3. Layer-wise probing of five frozen vision encoders (CLIP, SigLIP, DINOv2, ViT-MAE, Swin-V2) plus a scale study of DINOv2-S/B/L/g, for both canonical color and object identity, with an analysis of how the two accuracy curves relate across layers.

4. A cross-stack localization of semantic decodability in VLM pairs (CLIP→Molmo, CLIP→mPLUG-Owl, SigLIP→PaliGemma), comparing pre-VLM encoders, post-VLM vision towers, the visual interface, and decoder-side visual-token states.

Main Findings

• Canonical color survives grayscale conversion. All five vision encoders decode canonical color well above the 13.98% majority baseline after visible color is removed. Best-layer grayscale accuracies: CLIP 41.5%, DINOv2 43.8%, SigLIP 42.4%, ViT-MAE 40.3%, Swin-V2 40.2% (all p < .001 against baseline). RGB-to-grayscale drops were large and significant (32.8–39.5 percentage points), confirming RGB probing is heavily driven by visible pixels.

• Counterfactual validation points to canonical, not surface, color. On Visual-CounterFact images, the actual-surface-color probe stays near the majority baseline after preprocessing, while canonical-color probes on both original and recolored images remain well above baseline and trace similar layer-wise curves. The residual signal behaves like object-category knowledge, not leftover chromatic cues.

• Signal strength scales with model size. Within the DINOv2 family, canonical-color accuracy increases monotonically with scale under every input setting (Spearman ρ = 1.00): Gray accuracy rises from 46.2% (S, 22M) to 51.8% (g, 1.1B); Gray+HE rises from 41.8% to 48.0%.

• Color and object identity are related but not identical. Layer-wise Spearman correlations between color-probe and object-probe curves are strongly positive for all encoders (strongest for CLIP and DINOv2), yet the trajectories diverge — some models show high color accuracy before object recognition peaks, others show late-layer color drops despite continued object gains. When scored against the object probe's own predicted identity, correct color predictions reach roughly 55–65% for CLIP, SigLIP, DINOv2, and Swin-V2 (lower for ViT-MAE), indicating color partly tracks the model's own object concept rather than a fully independent signal.

• VLM post-training can suppress tower-level decodability — model-dependently. For the two CLIP-based VLMs, the post-training vision tower is markedly worse at linear probing than the original encoder. Molmo: color 43.9% → 36.8% (−7.1 pp), object 65.0% → 26.2% (−38.8 pp). mPLUG-Owl: color 43.0% → 30.8% (−12.3 pp), object 64.4% → 7.0% (−57.4 pp). PaliGemma is the exception: its SigLIP tower is essentially unchanged (color −0.4 pp, n.s.; object −1.9 pp).

• Information reappears downstream — reformatting, not removal. In Molmo and mPLUG-Owl, decoder-side visual-token averages recover much of the lost decodability. Probing the mPLUG-Owl path directly shows the object-recognition jump happens largely at the Q-Former-style visual abstractor, before tokens enter the LLaMA decoder. Blank-image baselines stay near majority, confirming the recovered signal depends on visual input.

• Final VLM answers are a poor window into internal representations. On grayscale prompts, all three VLMs score poorly on "what color is this?" (23.4%, 14.9%, 17.3%) despite gray being the correct answer, and "typical real-world color?" varies wildly (Molmo 44.5%, mPLUG-Owl 25.6%, PaliGemma 4.1%). Internal probes cleanly separate these effects.

Methodology in Plain English

The researchers build a dataset where every object category has a known typical color — a banana is yellow, an eggplant is purple. For each category they collect up to five images and mark the canonical color from structured Wikidata entries.

To test whether models know these colors conceptually rather than just reading pixels, they remove color from the input in two steps. First, convert to grayscale using a single luminance channel. Second, apply per-image histogram equalization to flatten brightness patterns that might otherwise leak color information. They call this processed input "grayscale."

They then freeze several pretrained vision encoders and pull out their internal features at every layer, mean-pooling across image patches to get one vector per image. On top of those frozen vectors they train simple linear classifiers (regularized logistic regression) to predict either the canonical color (10 classes) or the object identity (708 classes). If a simple linear classifier can recover a property, that property is considered "linearly accessible" in the representation.

Color probing uses strict object-class-level five-fold cross-validation: entire object categories are held out, so the probe must generalize to unseen categories, not just unseen images. Object probing uses image-level hold-out instead, because holding out whole categories would erase the target labels from training.

They validate the control with Visual-CounterFact images, where objects are recolored to unnatural colors (e.g., a blue strawberry). If the residual color signal were just leftover surface color, it should follow the recolored appearance; instead it tracks the object's canonical color in both original and recolored images.

Finally, they extend the same probing logic up the VLM stack: they compare matched pre- and post-VLM vision towers, probe the output of the visual interface, and probe averaged visual-token representations at each decoder layer.

Why This Matters

Impact on research. The paper reframes a subtle interpretability question: where in a vision-language pipeline does conceptual, world knowledge live, and how does training reshape it? Its key methodological insight — that you can dissociate a "visible" attribute from its "conceptual" counterpart by removing the visible channel at the input — is transferable to other attributes. The finding that VLM post-training can make information less linearly accessible in the vision tower yet still recoverable downstream challenges the assumption that a weaker probing readout means information was lost. It argues for representational reformatting rather than erasure, which matters for anyone auditing, editing, or compressing vision encoders before VLM training.

Real-world applications:

• Robustness auditing for deployed VLMs: understanding that object and color knowledge can be hidden behind a learned visual interface helps diagnose why a model fails on unusual inputs without falsely concluding the knowledge is absent.
• Model compression and vision-tower reuse: teams that swap or retrain vision towers for VLMs can use this probing framework to check whether object-level semantics survive the swap.
• Accessibility tooling: knowing models retain object-identity signal in grayscale supports building robust captioning or recognition for color-blind users or grayscale-captured imagery.
• Dataset and benchmark design: the method exposes when a benchmark measures visual perception versus memorized priors, useful for constructing fairer multimodal evaluations.

Industry relevance. Every major VLM pipeline involves a vision encoder followed by a learned interface and a language model. This paper gives engineering teams a cheap, model-agnostic diagnostic (linear probes on cached features) for monitoring where capabilities live after each stage of training, which is directly useful for debugging, model merging, and deciding whether to freeze, fine-tune, or replace a vision tower.

Future Directions

• Extend the control paradigm to other attributes. Material, texture, function, and affordance cannot be removed as cleanly as color; the paper's own limitations note that new controls and evaluation protocols would be needed.
• Move beyond linear probing. A nonlinear or task-specific probe could recover information invisible to the linear readouts used here, which would test whether the reported "reductions" after VLM post-training are truly representational losses or just changes in linear accessibility.
• Explain what the learned visual interface does. The mPLUG-Owl result localizes the recovery to the Q-Former-style abstractor, but why a compressed interface makes object information more linearly available is unresolved — as is why PaliGemma behaves so differently from the CLIP-based VLMs.
• Connect internal decodability to downstream behavior. Final VLM answers and internal probes diverge sharply; closing that gap would clarify when internal knowledge actually gets used in generation and when it is overridden by priors or formatting preferences.

Target Audience

Researchers and engineers working on vision-language models, multimodal representation learning, and interpretability. It is most useful to those who already work with frozen feature extraction and linear probing, or who need practical diagnostics for how training stages reshape a vision encoder's semantic content. Readers interested in cognitive-science-adjacent questions — such as whether machines represent object concepts the way humans do — will also find the canonical-color framing a useful, concrete entry point.

Authors’ abstract

Visual encoders construct a representation of the image input for Vision-Language models. How much conceptual, as opposed to immediately visible, information does this representation contain? We use canonical color as a controlled test case to ask whether vision encoders make canonical-color information linearly accessible, even when color is removed from the input image. We construct a dataset of objects with canonical colors, and probe vision encoders for both color and object identity using color and grayscale images. We find that canonical color remains decodable from grayscale images, and is tied to predicted object identity, indicating a conceptual link. Extending this analysis to full VLMs, we find that VLM post-training can have a surprisingly large effect on color decodability in the vision encoder. Overall, canonical color provides a usefully controllable lens for tracing object-level conceptual semantic information in vision encoders and VLMs.