Rule vs.Consequence: Dissociable Internal Representations of Moral Reasoning in LLMs

AI summary

Overview

Research area: Natural Language Processing / mechanistic interpretability, specifically the internal representation of moral reasoning in large language models.

Technical level: Intermediate. The paper is readable without a background in moral philosophy, but it assumes familiarity with linear probes, residual-stream activations, and activation steering.

Scope: The paper tests whether an LLM's different answers to rule-framed versus consequence-framed moral questions come from two separable internal directions in the residual stream or from one shared axis, using a 2×2 factorial dataset across three instruction-tuned models.

What This Paper Is About

Large language models answer moral questions differently depending on whether a question is framed around rules and norms or around outcomes and welfare. It is unclear whether that difference reflects two genuinely distinct internal representations or just sensitivity to the words in the prompt. The authors build a balanced dataset that crosses framing style with permissive/prohibitive judgment polarity, then use linear probing and activation steering to ask whether these two factors occupy separate, causally effective directions inside the model.

Key Contributions

1. A 2×2 factorial moral reasoning dataset. Scenarios drawn from ETHICS, Moral Stories, and Social Chemistry 101 are each paired with both a rule-framed and a consequence-framed question, under matched permissive/prohibitive polarity, producing 1,452 examples with strict marginal label balance (180 per cell in training, 60 per cell in validation and benchmark test). It adds a 32-item hand-written cleanroom set including 10 conflict pairs where the two framings predict opposite gold labels.

2. A double dissociation between reasoning style and judgment polarity. Two orthogonal directions are identified in the residual stream: Dir-A tracks reasoning style and Dir-B tracks judgment polarity, each near-blind to the other factor (cross-correlation |r| < 0.04), replicated in Qwen2.5-7B-Instruct, Llama-3.1-8B-Instruct, and Mistral-7B-Instruct-v0.3 at comparable proportional depth.

3. Causal evidence via activation steering. Dir-A is causally sufficient to shift judgments (Cohen's d > 1.0, all p < 0.0001), and on conflict pairs it selectively corrects rule-framed errors while leaving consequence-framed judgments unchanged—a pattern the authors argue is inconsistent with a global acceptance bias.

4. Generalization and lexical-baseline comparisons. Cross-domain probing transfers across three of four benchmark sources (mean 0.967 accuracy excluding ETHICS Deontology), and the activation probe beats n-gram baselines on the cleanroom set, though not a hand-specified keyword detector.

Main Findings

• Two separable directions, not one strictness axis. At layer 10 of Qwen, Dir-A correlates r = +0.899 with style but only r = +0.025 with polarity; Dir-B correlates r = +0.856 with polarity but only r = +0.022 with style. The cross-correlation between projections is r = +0.030.

• Replication across three models. Qwen2.5-7B peaks at layer 10/28 (35.7% depth) with Dir-A r(style) = +0.899 and Dir-B r(polarity) = +0.856; Llama-3.1-8B at layer 11/32 (34.4%) with +0.875 and +0.888; Mistral-7B at layer 15/32 (46.9%) with +0.978 and +0.909. Cross-correlation stays below |0.04| in every model.

• Perfect in-domain separability. On the benchmark test set, the style probe reaches 1.000 accuracy with AUC = 1.000, and the polarity probe also reaches 1.000 accuracy.

• The effect is not confined to one layer. All ten layers tested in the range 6 to 15 of Qwen show significant framing selectivity (p < 0.0001). Layer 10 has the highest Fisher discriminant ratio (J = 5.58), with layer 9 second (J = 4.21) and the ratio decaying below 1.0 by layer 15.

• Large causal effects. Under Dir-A steering at α = ±60 on Qwen, logit difference shifts range from −0.640 (consequence-permissive, d = −0.398, p = 0.003) to +2.877 (rule-prohibitive, d = +1.664, p < 0.0001). Dir-B produces a rule-permissive shift of +1.113 (d = 1.060) and a consequence-permissive shift of −0.410 (d = −0.290, p = 0.030).

• Better than random directions. At |α| = 40, Dir-A produces mean |ΔLD| = 1.715, versus 0.778 and 0.640 for two random unit vectors—roughly 2.3 times more effective.

• Modest accuracy gain overall. Steering with Dir-A raised overall accuracy from 0.613 to 0.650 (3.7 percentage points), with the largest absolute gain of 13.3 percentage points on consequence-permissive examples (0.183 to 0.317).

• Framing-specific correction on conflict pairs. Rule-framed conflict examples improved from 0.400 to 0.700 accuracy (30 percentage points, 3 prediction flips), while consequence-framed examples, already at 0.800 baseline, showed zero flips despite a comparable mean logit shift of +2.61. The authors report Wilson 95% confidence intervals of [0.349, 0.910] and [0.483, 0.957] respectively, and note that n = 10 per condition makes these intervals wide.

• Partial generalization across domains. Leave-one-source-out accuracy was 1.000 on Social Chemistry 101 (n = 60, cosine similarity 0.731), 0.900 on Moral Stories (0.900), 1.000 on ETHICS Commonsense (0.777), and 0.517 on ETHICS Deontology (0.835). The Deontology failure is attributed to near-identical framing prefixes across both conditions.

• N-gram baselines collapse out of domain; keyword matching does not. On the cleanroom set, TF-IDF unigram and bigram probes fall from 1.000 to 0.500 and 0.438, the activation probe retains 0.594 (a 0.156 gain), and keyword matching scores 0.750.

• Approximate commutativity. Sequential Dir-A then Dir-B application shows mean |AB − BA| = 0.719, at the 8th percentile of a 100-pair random-direction null distribution (mean 0.915, standard deviation 0.134, t = −4.20, p < 0.0001), consistent with independent pathways.

• No necessity claim. Orthogonal ablation of either direction at layer 10 produced near-zero selective effects, with selectivity ratios below 1.0, which the authors read as redundant encoding rather than unique necessity.

• Weak link to spontaneous reasoning style. Dir-A projection does not correlate with spontaneous reasoning style on neutral prompts (r = −0.056, p = 0.79), which is why the authors frame the result as "moral reasoning style sensitivity" rather than internalized normative reasoning.

Methodology in Plain English

The researchers built a balanced set of moral scenarios, each asked two ways: once invoking an applicable rule or norm ("Given the social rule that…, is this action acceptable?") and once invoking outcomes and welfare ("Considering the outcomes and effects on those involved, is this action acceptable?"). Because each scenario appears in both styles, and because permissive and prohibitive answers are balanced evenly across cells, the two factors can in principle be separated statistically rather than collapsing into a single "strictness" axis.

They then ran three instruction-tuned models in 4-bit quantization and recorded the residual-stream activation at the final prompt token at every layer. For each layer they fit a regularized logistic regression probe to separate rule-framed from consequence-framed examples, and a second probe to separate permissive from prohibitive examples. The probe weight vectors were converted back into unit-norm directions in activation space, and layers were ranked by a Fisher discriminant ratio on held-out validation data; layer 10 of Qwen won with J = 5.58.

To test whether the directions actually drive behavior, they injected them into the residual stream at the prompt token during generation and measured the shift in logit difference between acceptable-type tokens and unacceptable-type tokens. Since heavy quantization inflates absolute activation norms, they report a perturbation ratio rather than raw α, showing that α = 60 in Qwen corresponds to a ratio of 0.09 to 0.14 and matches α ≈ 7 in an unquantized model of the same architecture. They verified that 94.2% of steered dimensions remained within ±3σ of the unsteered distribution at α = 60.

Finally, they compared these directions against three lexical baselines (keyword matching and two TF-IDF probes) on both the benchmark test set and the hand-written cleanroom set, and ran a conflict-pair experiment where the two framings predict opposite answers and only a framing-specific (not a globally permissive) mechanism can explain the pattern of prediction flips.

Why This Matters

Impact on research. The paper moves moral reasoning evaluation from behavioral benchmarks to internal mechanism, and it offers a methodological template—a factorial design that separates a framing manipulation from judgment polarity—that other interpretability work on social or normative concepts could reuse. The finding that style and polarity are geometrically orthogonal while being roughly equally strong predictors contradicts a single "normative strictness" axis, and the failure of the probe to beat a hand-written keyword detector sets a concrete bar that future generalization claims need to clear.

Real-world applications:

• Auditing deployed models in legal, medical, or safety contexts to check whether an output is being driven by how a question was phrased rather than by the underlying facts.
• Debiasing interventions: the paper notes that framing-debiasing methods such as DeFrame could use directions of this kind as building blocks.
• Monitoring or diagnosing framing sensitivity in clinical and legal decision-support tools, where question phrasing varies naturally across users.
• Providing an interpretability handle for red-teaming, so that a model's normative outputs can be tested for stability under reframing.

Industry relevance. Any organization deploying instruction-tuned models in regulated or ethically sensitive settings has an interest in knowing whether a model's answers are framing-dependent and whether that dependence is a surface artifact or an internal structure that could be monitored, steered, or corrected. The three-model replication across Qwen, Llama, and Mistral suggests the geometry is not vendor-specific.

Future Directions

1. Expand the conflict-pair set. The behavioral dissociation rests on n = 10 per condition with wide confidence intervals; the authors explicitly call for a larger set to convert this from supportive evidence into a definitive statistical claim.

2. Test at full precision and on base models. The reported α values of 40 to 60 are attributed to 4-bit quantization, and the observed rule-acceptance bias leads the authors to hypothesize that RLHF may create or amplify the style-sensitive direction—making base variants of Llama and Mistral an important comparison.

3. Connect the direction to spontaneous reasoning. Because Dir-A does not predict reasoning style on neutral prompts (r = −0.056), the next step is testing whether it predicts reasoning style in chain-of-thought settings.

4. Demonstrate control on an applied benchmark. The authors state that dissociability and steerability are established, but not that steering confers control on a downstream task of independent interest—for example holding polarity fixed while shifting the framing a model reasons from, or auditing whether deployed outputs are framing-driven. They call this the most valuable extension.

Target Audience

Mechanistic interpretability researchers studying how models represent social, normative, or abstract concepts; AI safety and alignment researchers interested in framing effects and activation steering; and applied practitioners in legal, medical, or policy-facing AI who need to know whether model judgments are stable under reframing. Readers without familiarity with linear probing or residual-stream interventions will find the methods section dense, but the framing of the problem and the conflict-pair results are accessible without that background.

Authors’ abstract

Large language models are increasingly deployed in morally and legally sensitive contexts, yet little is known about how they internally represent distinctions between moral reasoning styles. We ask whether a model's differential response to rule-framed and consequence-framed moral questions reflects distinct internal structure or merely surface lexical sensitivity. We construct a 2$\times$2 factorial dataset crossing reasoning style (rule-based vs. consequence-based framing) with judgment polarity (permissive vs. prohibitive), drawn from three established moral reasoning benchmarks and evaluated on a cleanroom test set of novel phrasings that includes conflict pairs where the two framings predict opposite labels. Using linear probing and activation steering, we identify two orthogonal directions in the residual stream: Dir-A tracks reasoning style and Dir-B tracks judgment polarity, each near-blind to the other factor (cross-correlation $|r| &lt; 0.04$). This double dissociation replicates across three instruction-tuned models from different developers at comparable proportional depth, and it is not knife-edged in layer: the directions are recoverable and causally effective throughout an extended band of the network. Dir-A can causally change judgments, and in conflicting scenarios, it selectively corrects errors in rule-based judgments without changing consequence-based judgments, suggesting that it does not simply make the model more likely to accept everything. Generalization beyond the training templates is partial: the direction transfers across benchmark sources, but its advantage over lexical baselines on novel phrasings is modest. We therefore frame the result as moral reasoning style sensitivity, a geometrically dissociable and causally effective structure, rather than internalized normative reasoning.