How Language Models Conflate Logical Validity with Plausibility: A Representational Analysis of Content Effects

AI summary

Authors: Leonardo Bertolazzi (University of Trento), Sandro Pezzelle (University of Amsterdam), Raffaella Bernardi (Free University of Bozen-Bolzano) arXiv: 2510.06700v3 [cs.CL]

Overview

• Research area: Natural Language Processing, specifically mechanistic interpretability of reasoning in large language models.
• Technical level: Intermediate. The paper assumes familiarity with LLM hidden states, linear probes, activation steering, and prompting setups such as zero-shot and chain-of-thought, though each concept is defined before use.
• Scope: The paper traces content effects in syllogistic reasoning to the geometric entanglement of validity and plausibility directions inside ten instruction-tuned LLMs, and shows that disentangling those directions reduces the bias.

What This Paper Is About

Both humans and LLMs tend to judge an argument as logically valid when its conclusion is plausible, even when the argument's structure does not support it. The paper asks why this happens inside the model: whether the abstract concepts of validity (does the conclusion follow from the premises?) and plausibility (is the statement true in the real world?) are encoded in ways that overlap in the model's hidden representation space. The goal is to connect an internal representational property to an observable behavioral bias, and then to use that connection to intervene.

Key Contributions

1. A behavioral demonstration of content effects in current LLMs. The authors evaluate Qwen-2.5, Qwen-3, and Gemma-3 models under zero-shot and chain-of-thought prompting on a balanced syllogism dataset, and introduce a Content Effect (CE) metric bounded between −1 and 1 to quantify how much plausibility drives validity judgments.

2. Evidence that validity and plausibility are each linearly represented, and that the two directions are highly aligned. Using difference-in-means vectors and steering power, the authors show that single directions can flip judgments in both tasks, and that validity and plausibility vectors share substantially higher cosine similarity than control concepts (hypernymy, harmlessness).

3. A link from geometry to behavior, plus causal cross-influence. Across model–prompt pairs, higher plausibility–validity similarity predicts stronger content effects in a mixed-effects regression, and plausibility vectors steered during the validity task (and vice versa) flip predictions, establishing cross-task causal influence.

4. A steering-based debiasing intervention. A task-difference vector computed as the difference between mean validity-task and plausibility-task activations is scaled by α = 1.5 and added at a specific layer, reducing content effects and, in one model, raising accuracy, without updating any parameters.

Main Findings

• Content effects are present but shrink sharply with chain-of-thought. For Qwen2.5-32B-Instruct, CE is 0.348 zero-shot and drops to 0.096 with CoT; for Qwen3-14B, CE is 0.213 zero-shot and drops to 0.014 with CoT. Across all models, CoT produced a significantly lower CE than zero-shot (Mann-Whitney U test, p < 0.01).

• Overall and per-subset accuracy tracks the bias. On the logical validity test split, Qwen2.5-32B-Instruct scored 100.00 on valid/plausible and 98.04 on invalid/implausible but only 67.50 on invalid/plausible and 60.92 on valid/implausible in zero-shot, for an overall accuracy of 81.62. With CoT, those four figures were 98.67, 86.64, 93.10, and 100.00, for an overall accuracy of 94.60. Qwen3-14B followed the same shape: 97.33, 90.83, 60.92, 97.06, overall 86.54 zero-shot; 95.31, 99.10, 92.50, 99.90, overall 96.70 with CoT.

• Single directions control both concepts. Validity and plausibility vectors reach steering power of approximately 1 at late layers of Qwen2.5-32B-Instruct's 63 layers, and near zero at early layers, under both zero-shot and CoT prompting. Both tasks peak at similar layers.

• Validity and plausibility are geometrically entangled. A 3D PCA projection of layer 50 (of 64) hidden states shows four separated clusters whose true/false and valid/invalid axes run in parallel. Average cosine similarity between validity and plausibility vectors is reported as 0.48 to 0.64 in the text (0.53 to 0.64 in the Figure 3 caption), versus 0.10 to 0.13 for validity versus harmlessness and −0.12 to −0.17 for validity versus hypernymy. Alignment did not differ significantly between prompting conditions (p = 0.625).

• Representational similarity predicts behavioral bias. A linear mixed-effects model fit on 57 model–prompt data points (CE ~ Prompt + AvgSim + (1|LLM)) found average similarity to be a significant positive predictor of content effect (β = 0.557, p = 0.017), and zero-shot prompting to be associated with higher CE than CoT (β = 0.188, p < 0.001). The random intercept variance for LLMs was small (0.001).

• The influence runs in both directions. Plausibility vectors applied during the validity task, and validity vectors applied during the plausibility task, both flip a large fraction of predictions at late layers, for Qwen2.5-32B-Instruct and Qwen3-14B under both prompting styles. Qwen3-14B, which shows the lower CE, has a smaller set of layers where this cross-task steering transfers effectively.

• Disentangling the concepts reduces the bias. Adding the task-difference vector μ(V−P) scaled by α = 1.5 at layer 43 for Qwen2.5-32B-Instruct raised accuracy from 81.62 to 82.21 and cut CE from 0.348 to 0.072. For Qwen3-14B at layer 26, accuracy rose from 86.54 to 96.70 while CE fell from 0.213 to 0.043.

Methodology in Plain English

The authors take an existing syllogism dataset from prior work and rebuild it as a binary classification task: 1,280 syllogisms, each with two premises and a conclusion, split 70-30 into train and test, covering all 64 syllogism types in both "plausible" and "implausible" conclusion conditions. The dataset is approximately 42% valid and 58% invalid, and was assembled from 640 plausible and 640 implausible syllogisms built on ten triples of taxonomically related terms.

For each syllogism the model must say whether the argument is valid; separately, the model must judge 1,056 unique conclusions as true or false. Two control tasks guard against the possibility that the effects are just artifacts of binary classification or shared vocabulary: a hypernym/hyponym task over 180 examples built from the same ten term triples, and a harmful/harmless classification task of 916 examples drawn from prior work.

To find the internal representations, the researchers record the model's hidden activations at the last token before it emits a label, average them separately for each class, and subtract the negative-class mean from the positive-class mean. This gives one vector per concept per layer. They measure "steering power" by adding that vector to the activations against the model's own prediction and counting how often the answer flips. They then compare the validity vector with the plausibility vector, and with the two control vectors, using cosine similarity restricted to layers where steering was effective (SP > 0.75). They also test whether the similarity predicts the strength of the content effect, whether a plausibility vector can steer validity judgments and vice versa, and whether a vector built from the difference between the two tasks' mean activations can separate the concepts and reduce the bias.

All experiments use instruction-tuned models from the Qwen-2.5, Qwen-3, and Gemma-3 families with parameter counts from 4B to 32B, tested with both zero-shot and CoT prompts, with Qwen2.5-32B-Instruct and Qwen3-14B as the main reference models. All experiments were implemented in PyTorch using the Transformers library, and code and data are publicly released.

Why This Matters

The paper's distinctive move is to explain a behavioral bias that had already been documented in LLMs (in the Wason selection task and syllogistic inference by Lampinen et al., 2024, and in multiple-choice syllogisms by Bertolazzi et al., 2024) by pointing to a specific geometric property of the model's representations, then using that property to build a fix. Prior mitigation work either fine-tuned on pseudo-word vocabularies (Bertolazzi et al., 2024) or steered toward behaviorally defined correct answers (Valentino et al., 2026); this work instead steers by disentangling validity from plausibility, so the intervention follows from the diagnosis rather than from an external definition of correctness.

Real-world applications:

• Verification and fact-checking pipelines: systems that must distinguish "this claim follows from the given evidence" from "this claim sounds true" are directly vulnerable to the conflation documented here.
• Legal and regulatory reasoning assistants: argument-structure evaluation is the core task, and a bias toward plausible-sounding conclusions is precisely the failure mode to avoid.
• Educational tools that teach logic: a system grading syllogism validity would give misleading feedback if it rewarded plausible conclusions.
• High-stakes decision support where conclusions should follow from premises rather than from prior belief.

Industry relevance: The debiasing intervention requires no parameter updates and operates through activation steering at inference time using a single vector, which makes it a comparatively cheap technique to layer onto deployed models. The finding that chain-of-thought prompting reduces content effects substantially (and nearly eliminates them for Qwen3-14B, CE = 0.014) is also actionable, though the authors note in their limitations that they cannot yet explain why CoT models show lower content effects despite exhibiting similar representational similarity and cross-influence patterns.

Future Directions

• Extend beyond the last token and beyond dense residual-stream vectors. The current analysis reads only the residual stream at the final token before the label; the authors propose tracking how validity and plausibility representations develop across the entire generated sequence and using methods that extract sparser features. They flag this as a particularly promising route to explaining the zero-shot versus CoT gap.

• Explain the origin of the entanglement. The paper documents that the concepts are entangled but not why. The authors conjecture that it stems from training data, since valid arguments in most texts are also sound (true premises), leaving models with limited exposure to valid arguments with false premises.

• Link representations to training-data statistics. The conjecture above suggests directly investigating the relationship between how these concepts are represented inside LLMs and the statistical properties of the corpora they were trained on.

• Test the framework on other biases. The authors suggest extending the approach to other cognitive biases and asking whether similar representational mechanisms underlie them, and exploring whether disentangled representations support more reliable reasoning more generally.

Target Audience

This paper is most useful to interpretability and LLM-reasoning researchers who want a concrete case study connecting a measurable representational property (cosine alignment between concept directions) to a measurable behavioral bias (content effects) and to a working intervention. It is also relevant to practitioners building reasoning, verification, or fact-checking systems who need to know how and why models substitute plausibility for logical validity, and to cognitive scientists interested in the analogy with dual-process accounts of human reasoning. Readers without prior exposure to activation steering or difference-in-means probing will need to consult the cited background work (Belrose, 2023; Rimsky et al., 2024; Marks and Tegmark, 2024; Liu et al., 2024) to follow the experimental design in detail.

Authors’ abstract

Both humans and large language models (LLMs) exhibit content effects: biases in which the plausibility of the semantic content of a reasoning problem influences judgments regarding its logical validity. While this phenomenon in humans is best explained by the dual-process theory of reasoning, the mechanisms behind content effects in LLMs remain unclear. In this work, we address this issue by investigating how LLMs encode the concepts of validity and plausibility within their internal representations. We show that both concepts are linearly represented and strongly aligned in representational geometry, leading models to conflate plausibility with validity. Using steering vectors, we demonstrate that plausibility vectors can causally bias validity judgements, and vice versa, and that the degree of alignment between these two concepts predicts the magnitude of behavioral content effects across models. Finally, we construct debiasing vectors that disentangle these concepts, reducing content effects and improving reasoning accuracy. Our findings advance understanding of how abstract logical concepts are represented in LLMs and highlight representational interventions as a path toward more logical systems.