Cross-modal Steering Vectors Exist Across Both Image and Text Generation

AI summary

Overview

Research area: Interpretability and controllable generation in unified multimodal models (UMMs) — models that autoregressively generate both text tokens and image tokens from a single shared weight space. Sits at the intersection of activation steering, mechanistic interpretability, and cross-modal representation learning.

Technical level: Intermediate. The paper assumes familiarity with hidden states, residual-stream interventions, and the idea of a "steering vector," but the core setup (extract a direction, add it, measure the effect) is explained from first principles.

Scope (one sentence): The paper tests whether a linear steering vector extracted entirely from one modality (text or image) can be added, with no adaptation, to the other modality's tokens and still causally steer generation — and finds that it often can, in both directions.

What This Paper Is About

Prior work has steered language models and image generators separately, treating text steering and image steering as two different toolkits. This paper asks whether the underlying control directions are actually modality-specific, or whether a single direction in a shared model can control a concept across both modalities.

To answer this causally rather than just correlationally, the authors extract steering vectors from minimal pairs in one modality and inject the same vector into the other modality's tokens during generation, measuring whether the concept shifts. They run this on three unified multimodal models (UniAR 8B, Emu3.5 34B, Liquid 7B) across eight concepts, testing all four combinations of source and target modality (txt→txt, txt→img, img→txt, img→img).

Key Contributions

1. A causal framing of modality unification. The authors operationalize "are concepts shared across modalities?" as cross-modal steering: does a vector found on tokens of one modality transfer zero-shot to steer generation on tokens of the other? They argue this goes beyond representational-similarity measures because steering is fundamentally causal.

2. Demonstration of bidirectional transfer in three UMMs. Contrary to prior work reporting strong asymmetry (text→image works, image→text mostly fails), they show transfer in both directions across UniAR, Emu3.5 and Liquid, for most of eight concepts.

3. Qualitative analysis of modality-unique concepts. They build purely linguistic axes (formal vs. colloquial language; English vs. French) and purely visual axes (low vs. high resolution; motion blur; camera angle) and inspect what happens when those vectors are applied in the other modality — e.g., a "formal language" vector producing formal clothing and design in images.

4. A geometric signature. They show that text- and image-derived steering vectors for the same concept often point in similar directions despite a clear modality gap, and that this cosine similarity correlates moderately with transfer success (Spearman ρ = 0.38, p < 0.01 across all model/concept/layer-range points).

Main Findings

• Transfer works in both directions, and is often close to native steering. In UniAR, a transferred vector recovers 84% of the intra-modality steering effect on images and 74% on text at middle layers. For every concept except spatial, at least one layer range lets text-derived vectors recover more than 80% of the intra-modality effect, in all three models.

• Text→image is the easier direction. Text-derived vectors steer images for nearly every concept. In Emu3.5 and Liquid they often beat image-native steering: average txt→img recovery is 1.10 and 1.20 respectively, i.e., above 1 (better than the intra-modality vector). The authors attribute this mostly to weak image-derived vectors in those models — img→img reaches an AUC of only 0.44 and 0.43 at middle layers, versus 0.83 in UniAR.

• Image→text is less uniform. UniAR recovers on average up to 0.96 of the intra-modality effect, while Emu3.5 and Liquid come close only at late layers (0.87 and 0.91). Averaged over all models and layer ranges (Table 1), the overall mean recovery is 1.04 for txt→img but 0.68 for img→txt.

• Layer location matters, and differs by direction. Txt→img transfer is strongest at middle layers in all three models. Transfer into text peaks at early layers in UniAR but at late layers in Emu3.5 and Liquid.

• Concept matters more than model. Emotion transfers most evenly, with a mean recovery of 0.95 in both directions. Age transfers into images particularly well (1.02–1.87 across models and layer ranges). Size shows the sharpest asymmetry: mean recovery 1.12 for txt→img but only 0.32 for img→txt. Spatial (left/right) fails across most settings — only moderate success for img→img in UniAR and minor txt→txt effects in Emu3.5 and Liquid — which the authors anticipated, since spatial relations are a documented failure mode of image generators and VLMs.

• Output quality does not collapse before the concept rate saturates. Using an AxBench-like judge, text fluency is still about three quarters of the unsteered level at 0.4 of the strength range (where the text concept rate peaks) and collapses beyond that; image coherence is still about 90% of its unsteered level at 0.6 of the range, where the concept rate has already reached over 90% of its maximum.

• Modality-unique concepts produce interpretable transfers. Steering with a formal-vs-colloquial language vector — defined only by sentence pairs like "I need to go" vs. "I gotta go" — shifts generated images toward formal clothing, black-and-white rather than colorful, and luxurious minimalist style. An English–French vector produces historically ornate French-style imagery (mansard roofs, Art Nouveau posters, a vintage car) and more feminine-presenting characteristics, and turns writing in the image into French-looking non-existent words. In the other direction, a low-to-high resolution image vector makes generated text shorter, less detailed, and prone to mention "blurry" or express confusion; a motion-blur vector shifts text toward calmer or faster-movement vocabulary; a camera-angle vector steers images cleanly but only shifts the next-token distribution in text without changing the predicted token.

• Aligned directions despite a modality gap. Text and image steering vectors for the same concept have cosine similarity up to 0.60 in UniAR and 0.51 in Liquid (Emu3.5 is near zero for all concepts except color, max 0.24). 30 of 42 (concept, layer range) points for UniAR and Liquid lie more than two standard deviations (0.03) above a random baseline. The happy–sad vectors have cosine 0.47 in UniAR's middle layers.

• The alignment is between relations, not between points. Activations are strongly anisotropic (two random same-modality activations have cosine ≈ 0.75 at middle layers; cross-modality ≈ 0.65), so the authors center and normalize before comparing. They find that in UniAR's middle layers, the average happy image is more than ten times closer to the average sad image than to the average happy sentence — yet the happy→sad difference directions in the two modalities have cosine similarity between 0.14 and 0.60 for five of eight concepts. Modalities occupy separate narrow subspaces that share a similar internal structure.

• The correlation between geometry and transfer is moderate and model-dependent. Overall Spearman ρ = 0.38 (p < 0.01) and Pearson r = 0.22 (n = 63 points). In UniAR the correlation is strongest for txt→img (ρ = 0.76). In all three models, the direction that correlates most is whichever direction has the stronger intra-modality baseline — i.e., where the denominator of the recovery ratio is reliable.

Methodology in Plain English

Setup. The authors need models where text and image generation share the same weights, otherwise cross-modal steering is meaningless. They pick three autoregressive UMMs that emit text and discrete image tokens in one stream: UniAR (8B, image tokens from a language-aligned vision encoder), Emu3.5 (34B, pixel-reconstruction tokens) and Liquid (7B, pixel-reconstruction tokens).

Building a steering vector. For each concept they define two opposite poles (e.g., happy/sad), generate N = 100 minimal pairs that differ only in the pole, run both members of each pair through the model, take one representation per input at each layer, and average the paired differences. Each layer's vector is normalized to unit length so a single scalar α controls steering strength comparably across layers and concepts.

• Text pairs are short declarative sentences produced by filling a fixed template, "a {subject} is {pole}", with subjects generated by Gemini 2.5 Flash — so the two sentences differ only in the final word. The hidden state of the last token is used.
• Image pairs start from baseline images generated with Gemini 2.5 Flash Image and are edited toward each pole with an instruction-based image-editing model, with the prompt asking to preserve subject, pose, framing and background. For size, depth and spatial, where edits were less reliable, they use Gemini 3 Pro Image and Reve. Hidden states are mean-pooled across all image tokens. All pairs were manually inspected; fewer than 5% failed edits were regenerated.

Applying the vector. During generation they add α·v̂ to hidden states at a chosen set of layers. The intervention is identical regardless of whether the tokens being generated are text or image, and regardless of which modality the vector came from — in transfer settings the vector is simply dropped into the other modality. They steer ranges of four consecutive layers (eight for the deeper Emu3.5) placed at one quarter, one half and three quarters of model depth — for example layers 7–10, 16–19 and 25–28 of UniAR's 36. Range steering was chosen over single layers because it steers more strongly, and over all-layer steering because it reveals where the modalities align. Strength α is swept per model and setting up to just past the point where outputs break down.

Measuring success. For each concept they use 20 neutral scene descriptions generated with Gemini 2.5 Flash. Images are generated from the description directly; text uses the prompt "Describe your day in the city. Your description must include {subject}." An LLM judge (Gemini 3.7 Flash) sees the steered and unsteered outputs in random order, blind to which is which, and is asked which shows more of a randomly chosen pole. The "concept rate" is the fraction of correct judgments over the 20 prompts; its normalized area under the curve (AUC) across strengths summarizes a concept in one number. Cross-modal recovery is the ratio of the transferred vector's AUC to the intra-modality vector's AUC — 1.0 means transfer is as good as native steering, above 1 means better. The judge agrees with human raters at Cohen's κ = 0.62.

Eight concepts. Four semantic concepts that occur naturally in both modalities (emotion happy/sad, age young/old, cleanness clean/dirty, chaos peaceful/chaotic) and four visual ones (size small/large, depth far/near, spatial left/right, color blue/red).

Why This Matters

Impact on research. The paper complicates a growing body of negative results about unification in multimodal models — near-orthogonal text and image representations, text-side knowledge edits that fail to show up in images, and largely disjoint circuits for the same task. It also directly counters a prior cross-modal steering study (Wang & Li, 2026) that reported one-directional transfer in a differently structured UMM (BAGEL, with separate language- and image-generation weights). The authors' explanation is that representations can be separated by modality while the directions of control remain shared, which suggests representational similarity is the wrong lens for asking whether a model is "actually unified" — steering is an alternative probe. The released vectors, pairs, generations and judge scores make the result easy to extend to other models.

Real-world applications (as the authors frame them):

• Safety steering that carries over to images. If a promising intervention makes a model safer in text, the results suggest it should automatically transfer to the model's visual outputs, avoiding a separate safety pipeline per modality.
• Knowledge editing with visual spillover. Updating facts in the model's text behavior could plausibly propagate to what it depicts, addressing a known failure mode where text-side edits do not appear in generated images.
• Cheaper control authoring. Practitioners can build a steering vector in whichever modality is easier to curate — minimal text pairs are far cheaper to produce than minimal image pairs — and then apply it to the other modality, which matters most for concepts that are unique to one modality.
• Modality-agnostic interpretability tooling. A single probe can be used to audit both a model's language and image behavior, rather than maintaining two disjoint sets of analysis tools.

Industry relevance. Unified models that generate images natively inside a language-model architecture (the paper cites Gemini and OpenAI systems as having moved this way) are becoming the default product architecture. If control directions are shared, teams can build one steering layer, one safety layer, and one red-teaming harness instead of one per modality — a meaningful reduction in engineering surface for anyone deploying controllable multimodal generation.

Future Directions

• Extend beyond three UMMs and eight concepts. The authors explicitly encourage testing other families of unified models and novel concepts; the failure of spatial (left/right) and the model-dependent geometry results (Emu3.5 near-zero cosine similarity) suggest the picture is not yet complete.

• Explain why image-derived vectors are weak in Emu3.5 and Liquid. The authors hypothesize this stems from differences in image tokenization (semantic, language-aligned tokens in UniAR versus pixel-reconstruction tokens in the other two) but do not establish it. Untangling tokenizer choice from transferability is an open question.

• Develop modality-agnostic theories of multimodal interaction. The finding that separate subspaces share relational structure — echoing vector offsets in word embeddings and the Platonic Representation Hypothesis — is offered as an observation, not an explanation. A mechanistic account of how a linguistic direction becomes visual formality is missing.

• Turn transfer into a general-purpose diagnostic. The authors position cross-modal steering as a way to quantify whether a unified model is actually unified, which would require validating the recovery metric against other unification measures and confirming that high recovery predicts successful transfer of other interventions (knowledge edits, safety fine-tuning) across modalities.

Target Audience

Researchers in mechanistic interpretability and multimodal representation learning will get the most from this, particularly those working on activation steering, unified multimodal architectures, or the "modality gap" debate. It is also directly useful to practitioners building controllable generation systems on UMMs who want a lightweight, training-free way to control image and text behavior with one intervention. Readers need some comfort with hidden states and residual-stream interventions; those without it will still follow the qualitative examples (formal-language vectors producing formal clothing) but will find the AUC and recovery-ratio analysis harder to parse. The paper's clear framing of causal versus correlational evidence also makes it a good methodological reference for graduate students designing interpretability experiments.

Authors’ abstract

Are concepts and mechanisms unified across modalities in current AI models, or instead implemented separately for each one? We operationalize this broad question of modality unification by studying cross-modal steering: 1) finding vectors that successfully steer generation on tokens of one modality, which then 2) transfer zero-shot to steer generation when added to tokens of another modality. Our study goes beyond simple correlational alignment (e.g. representational similarity) because steering is fundamentally causal. We study cross-modal steering in several unified multimodal models (UMMs), since they possess shared parameters for language and image generation, focusing on these two modalities. Across diverse steered concepts such as emotion and object size, we show that steering vectors transfer zero-shot in both directions (text-to-image &amp; image-to-text). Even for concepts that are highly unique to one modality (e.g. formal vs. colloquial language), steering vectors often induce a consistent, interpretable effect in the other modality, such as formal clothing and design. This transfer has a geometric signature too: despite a clear modality gap separating text and image representations, steering vectors with strong cross-modal transferability tend to point in similar directions for the same concept. Our findings shed light on how vision and language interact inside unified multimodal models and point towards more general, modality-agnostic approaches for controlling their behavior.