From Geometry to Behavior: How Instruction Exposure Unlocks Latent Rhetorical Directions in LLMs

AI summary

Overview

Research area: Mechanistic interpretability and representation engineering in large language models — specifically, whether a rhetorical style (explaining things by analogy) is encoded as a linear direction in activation space, and what training conditions make that direction actually control behavior.

Technical level: Advanced. The paper assumes familiarity with residual streams, difference-in-means vector extraction, activation steering, KL divergence, and base-vs-instruct model distinctions.

Scope (one sentence): A single-author study that extracts an "Analogy Vector" from 14 dense transformer models (seven matched base–instruct pairs across Gemma-2, Llama-3.1, and Qwen-2.5, 2B–32B) and shows that the vector exists in every base model but only becomes behaviorally usable in proportion to how much instruction data the model saw during its entire training pipeline.

What This Paper Is About

Language models absorb rhetorical habits from pretraining text — including the human tendency to explain things through analogy — but it is unclear whether that habit is stored as a distinct, findable geometric object inside the model, and why it sometimes fails to influence output even when it is present. The paper asks two questions: is an "analogy direction" already present in the residual stream before any instruction tuning, and what determines whether steering along that direction actually changes what the model writes? The goal is to map the transition from geometric existence to causal accessibility, using analogy as a test case for a broader class of continuous, generative rhetorical strategies (as opposed to binary features like refusal).

Key Contributions

1. A gradient of behavioral accessibility. Across seven matched base–instruct pairs, the analogy direction is extractable in every base model, but how much steering it changes behavior scales continuously with instruction exposure — Gemma-2 base models are geometrically present but behaviorally inert, Qwen-2.5 base models are already partially accessible, and Llama-3.1 sits in between. This challenges the binary "base vs. instruct" framing.

2. KL divergence under steering as a "wiring diagnostic." The authors propose that measuring the KL divergence produced by an intervention reveals whether a causal pathway to the output exists, even when output quality is poor — a model-agnostic probe that does not require the model to be able to follow instructions.

3. A practical, training-free lever for rhetorical amplification. For models where the direction is already accessible (notably Qwen-2.5 base), activation steering works as an inference-time, prompt-composable control on explanation style — positioned explicitly as a diagnostic complement to prompting, not a replacement.

4. A documented and fixed failure mode: lexical entanglement. At high steering coefficients the "Imagine" token embedding contributed roughly 10% of the vector's norm and caused catastrophic mode collapse; projecting the vector onto the subspace orthogonal to that token's unembedding direction reduces vector magnitude by only 1–5% and cuts collapse by 75–92%.

Main Findings

• Steering raises Analogy Quality specifically. On 300 held-out test prompts, steering improved the harmonic mean (HM) of Analogy Quality, Fluency, and Instruction-Following for all seven instruction-tuned models while fluency and instruction-following stayed stable or improved modestly — evidence the vector is semantically specific rather than a generic style perturbation. The headline case, Gemma-2-9B-IT, moves from HM 3.22 (literal prompt, no steering) to 3.94 at α = 1.0, a +22% gain.

• Prompting still wins. The explicit analogy prompt outperforms steering in every model (e.g., 4.54 HM for Gemma-2-9B-IT versus 3.94 steered), consistent with the AxBench benchmark the paper adopts conventions from.

• Per-model steering gains are uneven. ΔHM is +0.53 (Gemma-2-2B-IT), +0.71 (Gemma-2-9B-IT), +0.80 (Gemma-2-27B-IT), +0.15 (Llama-3.1-8B-IT), +0.38 (Qwen-2.5-7B-IT), +0.81 (Qwen-2.5-14B-IT), and +0.31 (Qwen-2.5-32B-IT).

• Gemma-2 base models are "behaviorally inert." Base HM stays flat and low (1.38 for 2B, 1.36 for 9B, 1.88 for 27B) with no layer producing meaningful steering, while their instruct counterparts reach HM 3.52, 3.90, and 3.87 — gaps of +2.14, +2.54, and +1.99, the largest of any family.

• Qwen-2.5 base models are pre-wired. Qwen-2.5-14B base reaches HM 3.05 versus 3.14 for its instruct counterpart (ΔHM = +0.09) — the analogy direction is already usable before any explicit instruction tuning, consistent with Qwen-2.5's 18-trillion-token pretraining curriculum that includes late-stage annealing on SFT-quality data. Qwen-2.5-32B shows a larger gap (ΔHM = +0.53), so accessibility varies with scale.

• Llama-3.1 lands in between. Base HM = 2.35 (well above Gemma-2 base) but a still-meaningful ΔHM of +0.77 after instruction tuning, placing it at the boundary between the inert and pre-wired tiers.

• Geometric corroboration via cosine similarity. Base–instruct cosine similarity of the analogy vector reaches 0.94–0.97 for Qwen pairs versus 0.37–0.66 for Gemma pairs, matching the behavioral ordering.

• KL divergence mirrors the gradient. Gemma-2 instruct models peak at KL ≈ 0.086–0.100 at the best layer, while their base counterparts stay ≤ 0.018 across all layers — the vector fails to shift the output distribution. Qwen-2.5 base and instruct KL profiles closely overlap (≈ 0.003–0.07); Qwen-2.5-32B-Instruct is only 0.004, as if steering merely reinforces an existing pathway. Llama-3.1 shows intermediate separation (KL_IT = 0.059).

• The steering coefficient itself is a probe. Gemma-2 models peak sharply at α = 1.0 and collapse at α ≥ 1.5 (all three metrics degrade together), whereas Qwen-2.5 models respond monotonically through α = 2.0 with all metrics still improving. The authors interpret this "headroom" effect as evidence that tightly wired directions sit in a narrow representational corridor and diffusely accessible ones sit in a flatter neighborhood.

• Human validation supports the LLM judge. 15 US-based Prolific raters scored Analogy Quality on 30 concepts across three model families, blind to condition; 1,318 of 1,350 possible ratings were completed (97.6%), aggregated to N = 90 concept-condition items. Overall Spearman ρ between human and judge Analogy Quality was 0.663 (p < 0.0001, 95% CI [0.534, 0.763]); per-model ρ was 0.604 for Gemma-2-9B-IT, 0.807 for Qwen-2.5-14B-IT, and 0.502 for Llama-3.1-8B-IT. Krippendorff's α across raters was 0.637 ("moderate"), and human ratings reproduced the paper's condition ordering (prompted 3.80 > steered 3.16 > literal 1.32, Mann-Whitney p < 0.0001), with 87.1% preference for non-literal responses (53.7% prompted, 33.3% steered).

• Controls confirm specificity and bidirectionality (all on Gemma-2-9B-IT). Random vectors of identical norm left HM unchanged at ≈ 3.22 across α ∈ [0, 2]. Applied to creative-writing prompts, the vector lengthened outputs (49 to 57 tokens) and increased analogy usage without hurting fluency (FL = 4.28), arguing against a mere "simplification" confound. Negative steering (α ∈ {−0.5, −1.0}) reduced Analogy Quality from 3.22 to 2.41, showing the direction acts as a continuous bidirectional dial.

Methodology in Plain English

The authors first build a paired dataset: 1,000 concepts drawn from the ELI5 community QA corpus (spanning r/explainlikeimfive, r/askscience, and r/AskHistorians), filtered to keep only concrete process-type questions, labelled into eight knowledge domains, and balanced to an imbalance ratio of ≤ 1.5×, then split 600/100/300 for train/dev/test. Concepts were extracted with a non-target model (Mistral-7B-Instruct-v0.3 via vLLM) specifically to avoid circularity. Each concept gets three literal prompt templates ("Explain [concept] technically") and three analogy templates ("Explain [concept] using an analogy").

They then run the models on both prompt types, record the residual-stream activations at the final prompt token, and subtract the mean literal activation from the mean analogy activation. That difference is the "Analogy Vector" for a given layer — the same difference-in-means trick previously used to isolate a single refusal direction. At inference, they add a scaled version of this vector back into the residual stream before a chosen layer's attention and MLP blocks.

Because the raw vector was contaminated by the literal "Imagine" token (which alone accounted for about 10% of its norm and caused degenerate output at high strength), they strip out that token's direction — a cheap fix that shrinks the vector by only 1–5% and largely eliminates the collapse.

Layer choice is treated as a hyperparameter: they sweep every layer, score outputs on a 100-prompt dev set, and keep whatever layer maximizes the harmonic mean. Evaluation uses Qwen-2.5-72B-Instruct as a judge scoring three 1–5 dimensions (Analogy Quality, Fluency/Coherence, Instruction-Following), combined via harmonic mean to penalize keyword-stuffing. Three conditions are compared: unsteered literal, explicit analogy prompt, and literal prompt plus steering vector. The key comparative move is running this identical procedure on both members of seven base–instruct pairs, so that differences in steering effectiveness can be attributed to training rather than to the vector-extraction method.

Why This Matters

Impact on research. The paper reframes instruction tuning from a binary switch into a continuous calibration process, and gives a concrete counterexample to the assumption that "base" models are a uniform category — Qwen-2.5's base models already contain instruction data, which changes what interpretability results on them mean. It also extends the causal-inertia literature (where representations exist but do not drive output) from binary features to continuous generative styles, and offers KL divergence under steering as a cheap, model-agnostic test for whether a capability is already wired to the output head.

Real-world applications:

• Inference-time style control — dialing explanation style (more or less analogical) without retraining or rewriting prompts, useful when an application wants a fixed prompting template but adjustable register.
• Auditing base models before post-training — using KL-under-steering to check which behaviors are already behaviorally accessible, complementing pretraining-curriculum evaluation.
• Educational and explanatory systems — where analogy is known to aid conceptual exploration, steering offers a composable knob alongside prompting.
• Model/label provenance checks — the base–instruct cosine similarity measure gives a quantitative signal of how much a "base" checkpoint has already been shaped by instruction-style data.

Industry relevance. The finding that a "base" checkpoint may already be partially instruction-tuned has direct consequences for anyone benchmarking, red-teaming, or fine-tuning from base weights: assumed starting points differ substantially by vendor. The orthogonalization fix for lexical entanglement is also a practical recipe for anyone deploying steering vectors in production, where high coefficients are tempting and mode collapse is costly.

Future Directions

• Composability of rhetorical directions. Testing whether analogy, formalization, and simplification vectors are orthogonal or can be combined would turn this from a single case study into a general framework for rhetorical style control.
• Cross-comparison with the refusal direction. Applying the same base–instruct gradient methodology to refusal would allow direct comparison with the single-direction safety result the paper builds on.
• Curvature as a predictor of collapse. Measuring the local curvature of the analogy direction's neighborhood at each model's best layer, and testing whether it predicts the α at which steering collapses, would connect the behavioral gradient to mechanistic geometry.
• Generalization beyond one feature and one architecture. Whether the gradient holds for formality, hedging, or Socratic questioning is untested, and the paper evaluates dense transformers only — mixture-of-experts routing is explicitly left to future work.

Target Audience

Interpretability and representation-engineering researchers will get the most from this paper, particularly those working on activation steering, linear feature extraction, or the base-versus-instruct distinction. It is also relevant to alignment researchers interested in causal inertia and to practitioners who deploy steering vectors and need to know why the same coefficient behaves differently across model families. Readers without background in residual-stream interventions will find the core argument accessible through the results tables, but the methodology section assumes prior familiarity with the technique.

Authors’ abstract

Language models do not merely memorize facts; they absorb rhetorical strategies from pretraining data, including the human tendency to reason by analogy. We ask: is this absorbed capability geometrically accessible before any instruction tuning, and what determines whether it can be activated at inference time? We extract an "Analogy Vector" from the residual stream via difference-in-means and show it enables steerable explanation style across seven instruction-tuned models (Gemma-2, Llama-3.1, Qwen-2.5; 2B-32B), improving harmonic mean from 3.22 to 3.94 (+22%) for Gemma-2-9B-IT on 300 held-out test prompts. A small human validation study confirms the LLM judge on Analogy Quality. Our central finding, from seven matched base-instruct pairs, is a gradient of behavioral accessibility: the analogy direction exists in every base model, but its behavioral accessibility under linear intervention scales continuously with instruction exposure during training. Gemma-2 base models (strict pretraining/IT separation) show flat steering landscapes: present but behaviorally inert. Qwen-2.5 base models (instruction data woven into pretraining) are already partially accessible before any explicit fine-tuning; base-instruct cosine similarity of the analogy vector reaches 0.94-0.97 for Qwen versus 0.37-0.66 for Gemma, providing geometric corroboration of the behavioral gradient. This challenges the binary "base vs instruct" framing: the relevant variable is how much instruction exposure a model has received across its full training pipeline. We position activation steering as a complementary interpretability tool that reveals how instruction exposure organizes rhetorical knowledge in activation space.