Recognition-Refusal Misalignment in LLMs: Why Models Answer Structurally Unanswerable Questions

AI summary

Overview

Research area: Mechanistic interpretability and AI safety for large language models, specifically the internal representations that govern abstention and refusal behavior.

Technical level: Advanced. The paper assumes familiarity with residual-stream probing, linear directions, activation steering, and cosine geometry of hidden states.

Scope: The paper examines whether instruction-tuned language models from 1.7B to 70B parameters internally encode that a structurally impossible question has no answer, and whether that recognition is routed into the model's existing safety-refusal mechanism.

What This Paper Is About

Large language models often produce answer-like outputs to questions that have no admissible answer — for example returning cot(-540°) = 0 when the expression is undefined, or True for (1).startswith("1"), which in Python raises an attribute error — instead of abstaining. The authors ask whether this failure comes from the model not recognizing impossibility at all (an encoding failure) or from recognizing it but failing to route that signal into the mechanism that produces abstention (a routing failure). They test this by extracting a linear "impossibility" direction from hidden states before generation and measuring its geometric and causal relationship to the well-known safety-refusal direction.

Key Contributions

1. A one-dimensional, null-space impossibility probe. Using a class-stratified 50/50 held-out answerable/unanswerable split, the authors fit PCA with k=100 on answerable-class training states, project states out of that answerable subspace to a residual, and estimate a mean-difference direction scored by cosine similarity (the CosNSRT probe, an instance of their Generalized Subspace Residual Score template). This probe separates answerable from structurally impossible prompts with mean AUC 0.939 across a 22-cell, 11-model grid.

2. A geometric comparison between recognition and refusal. The paper measures the angle between the impossibility direction d_imp and the canonical safety-refusal direction d_ref,safety (constructed as μ_harmful − μ_harmless from 50 harmful and 50 harmless prompts, following Arditi et al., 2024), reporting mean cosine 0.087 across the 22-cell grid.

3. Causal intervention via generation-time steering. A forward hook adds α·d̂ to the residual stream during model.generate across α ∈ {5, 10, 20, 40}·σ in both signs, on a 4-anchor × {math800, code800, fact800} grid, with a random-direction control and a behavior classification rubric.

4. Base-versus-instruct geometry comparisons. Six paired base/instruct checkpoints on math800 test whether the misalignment is created by post-training, alongside a base-model probe showing impossibility detection with mean AUC ≈ 0.98 (range [0.94, 0.99]) across the six paired base checkpoints.

Main Findings

• Recognition exists before generation. A single one-dimensional A-null MeanDiff direction separates answerable from unanswerable prompts with mean AUC 0.939 (range [0.841, 0.993]) across the 22-cell, 11-model main grid. SmolLM2-1.7B reaches 0.880 on math800, Qwen3-32B reaches 0.993, and every 7B+ instruct model exceeds 0.90 on math800. The A-null projection is the load-bearing step: full residual stream scoring gives 0.907 and top-k A-PC scoring gives 0.890, with A-null exceeding Full in 19/22 cells and top-k A-PC in 21/22 cells.

• Recognition is nearly orthogonal to safety refusal. The mean cosine between d_imp and d_ref,safety is 0.087, with range [0.020, 0.130] across the 22 cells. The tightest bootstrap interval is Mistral-Small-24B on math800 (cos = 0.065, 95% CI [0.059, 0.071]). Observed cosines are 2–13 times the empirical random baseline.

• Subspace controls do not explain the angle. Projecting both directions into A-null before taking the cosine gives a range of [0.021, 0.154] and mean 0.097. The larger full-space cosine (range [0.057, 0.781], mean 0.240) is driven by shared answerable-structure variance: the A-PC component accounts for mean 0.813 of the magnitude of the full-space cosine (range [0.607, 0.983]).

• Behavior-defined invalidity awareness is only partially aligned. An in-domain direction d_struct,behav, contrasting unanswerable prompts where the model is invalidity-aware against those where it answers anyway, has cosine 0.40 (range [0.16, 0.59]) with d_imp across the six-cell primary non-Qwen3-8B subset, and cosine 0.08 with safety refusal. On that same six-cell subset, cos(d_imp, d_ref,safety) = 0.13, not the 22-cell 0.087.

• Strict refusal language is essentially absent. Strict refusal-only generations are absent at 0 of 50 unanswerable prompts per cell, so a strict-refusal direction is not constructible. Of 96 non-baseline rows, 48 are unmeasurable under a refusal-only criterion because clean baselines almost never produce explicit "I cannot" or "I do not know" phrasing; of the 48 measurable rows, 47 have refusal-only gated ΔG ≤ +5 pp. The lone exception, Mistral math A→U at α = 20, is +6.0 pp, versus a matched invalidity-aware ΔG of +32.7 pp.

• Steering changes invalidity-aware behavior, dose-responsively. Signal-minus-random gated flip rates reach +33 to +52 pp on anchor-quality structural cells, while random directions do not. Mistral-7B-Instruct is the keystone causal anchor and the only model exceeding +30 pp gated ΔG in both directions and both structural domains: math A→U +33 pp, math U→A +38 pp, code A→U +35 pp, code U→A +44 pp. The cleanest dose response is Mistral code A→U, rising 2.1 → 2.1 → 12.5 → 35.4 percent across increasing steering magnitude against a near-zero random baseline.

• Other anchors are asymmetric or domain-specific. Code remains anchor-quality for Gemma-3-4B (+35/+40 pp) and Qwen3-14B (+37/+52 pp); Qwen3-8B code is positive but sub-anchor (+22/+24 pp). Gemma-3-4B and Qwen3-8B reach anchor quality on math U→A only (+48 pp and +42 pp), while Qwen3-14B math fails in both directions. Of 24 anchor–dataset–direction slots, 10 are anchor-quality, 10 are positive but sub-anchor, and 4 are anecdotal (gateN ≤ 4, all U→A on fact).

• A post hoc two-factor account of effect variation. All 10 anchor-quality structural directions reach +30 pp with signal-branch degeneration no higher than 16 percent; five of the six sub-threshold directions reach at least 40 percent degeneration at a tested dose, while Qwen3-8B code A→U stays below 21 percent and peaks at +22 pp. At α = 20, mean structural-cell degeneration is 9.5 percent for Mistral-7B versus 31.5–45.5 percent for the other anchors. Recognition-to-behavior coupling correlates with the minimum U→A anchor dose (Spearman ρ = −0.86 across eight structural cells) but not with best effect size; A→U requires 2–4 times the U→A dose in the four cells reaching the anchor in both directions.

• The misalignment largely predates instruction tuning. Across 6 paired base/instruct checkpoints on math800, Δcos (instruct minus base) has range [−0.008, +0.110] and mean +0.037, small relative to the 22-cell instruct band [0.020, 0.130]. The largest shift is Qwen3-8B at +0.110; the vendor-confirmed post-training-only contrast, Llama-3.3-70B-Instruct versus Llama-3.1-70B-Base, has Δcos ≈ −0.0001.

• Boundary behavior for fact800 and FalseQA. On fact800, the U→A direction is not reliably measurable because clean baselines almost never abstain in passage-grounded language (gateN 0, 1, 4, 2 across the four anchors); A→U is well measured (gateN ≈ 49–50) but uniformly sub-anchor (+4 to +24 pp). FalseQA appears only as a zero-shot transfer boundary.

• Five alternative explanations ruled out. Prompt length (length-matched GSM8K length-only AUC → 0.500 while CosNSRT reaches 0.799), answerable-class difficulty (hard-vs-easy 0.61 versus impossibility 0.96, Δ = +0.35), category concentration (NS_SNR–AUC Spearman ρ 0.730, significant in 18/22 cells at p < 0.05), model scale (AUC 0.841–0.993), and a generic unanswerability axis (within-minus-cross drop 0.080; dot-product NSRT transfer 0.64–0.98 on GSM8K and 0.59–0.90 on FalseQA).

• What the paper does not support. The 48-cell deterministic breadth sweep across 16 models uses a non-comparable hallucination-reduction metric, ranks math versus code differently from the intervention grid, and supports no per-cell claim. The causal regularity is stated as a property of structural impossibility, not of unanswerability in general.

Methodology in Plain English

The authors treat three distinct kinds of unanswerable questions separately rather than merging them: structural impossibility (a formal rule makes no admissible answer exist, tested with math800 with 16 categories × 50 and code800 with 8 categories × 100), epistemic unanswerability (fact800, built from 800 paragraph-matched SQuAD 2.0 pairs whose unanswerable question's answer is absent from the shared passage), and false premises (FalseQA, used only as a zero-shot transfer boundary).

To find the recognition signal, they take hidden states at a fixed layer and split answerable versus unanswerable prompts in a class-stratified 50/50 held-out split. They fit PCA with 100 components on the answerable training states and subtract that subspace from every state, which removes what answerable prompts share (topic, wording, syntax) but not their answerability. In the remaining residual they compute the mean-difference direction between unanswerable and answerable states, then score held-out states by cosine similarity to it, averaging over 5 seeds per cell and using orientation-invariant AUC.

To build the refusal comparator, they follow Arditi et al. (2024): take 50 harmful and 50 harmless prompts at the same layer, compute the mean difference, and verify that the direction actually behaves like a refusal direction. Twenty of the 22 main-grid cells pass this verification; the two Llama-3.1-8B cells use a proxy and are flagged.

To test causality rather than mere correlation, they attach a forward hook that adds a scaled version of the impossibility direction to the residual stream at the last token of every forward pass during generation. They sweep four magnitudes in both signs and compare against a random direction of matched magnitude. The headline measure is the gated flip rate: the conditional probability that behavior changes on samples whose clean baseline already matched the pre-intervention class, judged by an invalidity-aware classifier with guards for mixed and degenerate outputs. Labels were assigned by an LLM-assisted batch review under a fixed written rubric with deterministic domain-specific labeling utilities, a second LLM-assisted audit pass, and first-author review of uncertain cases; the authors describe the effective labels as LLM-assisted rather than human-adjudicated and report the audit discrepancies in detail.

Why This Matters

Impact on research. The paper reframes a known output-level failure — models confidently answering unreasonable math problems, documented by Ma et al. (2026) — as a routing problem rather than a knowledge problem. It contributes a reusable diagnostic recipe: A-null probing, behavior-verified mean-difference refusal directions, matched-layer cosine measurement, gated generation-time steering, and base/instruct contrasts. It also argues that unanswerability regimes should not be collapsed into one generic category, since their ground truth, pair construction, intervention behavior, and even their geometry differ.

Real-world applications.

• Math and code assistants that currently return a confident numeric or boolean answer to an undefined expression could be made to flag the question as ill-posed rather than emit a plausible-looking wrong value.
• Retrieval-augmented systems that must distinguish "the evidence does not say" from "the answer does not exist" could use the separate structural and epistemic signals rather than a single abstention trigger.
• Safety and alignment pipelines could be tested for the specific gap the paper identifies, where a model that refuses harmful requests still answers impossible ones because the two abilities sit on nearly perpendicular internal axes.
• Selective-prediction or abstention scoring systems could be audited to confirm that their trigger direction is the one the model actually uses to modify behavior, not merely one that correlates with answerability.

Industry relevance. The base/instruct result implies that standard instruction tuning and preference optimization do not fix this geometry: the mean shift is +0.037 and the vendor-confirmed post-training-only contrast is Δcos ≈ −0.0001. If the misalignment survives current post-training recipes largely unchanged, then teams expecting alignment training to generalize from harmful-content refusal to structural-impossibility abstention may need domain-specific supervision instead.

Future Directions

• Closing the angle directly. The authors suggest training the safety-refusal pathway to read d_imp without collapsing it into d_ref,safety, and note that the Llama-70B contrast (Δcos ≈ −0.0001) implies direct structural-impossibility supervision may be required rather than incidental transfer.

• Testing whether the misalignment generalizes to other recognition-action gaps. The paper proposes that the same separation may recur when epistemic confidence, harmfulness, or instruction-compliance signals fail to drive hedging, refusal, or format adherence, and that the same probe-and-steer toolkit could test those cases.

• Extending beyond single directions and the tested architecture scope. The main analysis uses single directions; a four-cell 5–10-dimensional linear-subspace check preserves the low-overlap conclusion, but nonlinear or more distributed representations remain open. The authors also state that the assumption of single-direction readability is not yet established for vision-token conditioning, mixture-of-experts routing, hybrid attention, or explicit reasoning-mode training.

• Improving the measurement and causal coverage. The CosNSRT probe fixes PCA k=100 as a heuristic without tuning it per model or dataset; the pipeline requires labeled answerable-class prompts to fit the null-space basis, which may be costly in new domains; and fact800 remains a boundary case rather than a clean replication, with causal evidence cleanest for structural impossibility.

Target Audience

This paper is written for mechanistic interpretability and AI safety researchers, particularly those working on representation probing, refusal geometry, and activation steering. It will also interest alignment engineers who need empirical evidence about whether safety training transfers to new failure modes, and benchmark designers who care about keeping distinct unanswerability categories separate. Readers need working familiarity with residual-stream geometry and linear probing; the empirical results are otherwise described in explicit numeric terms.

Authors’ abstract

Large language models often answer structurally unanswerable questions, such as computing cot(-540°) or evaluating (1).startswith("1"), instead of abstaining. We ask whether this failure reflects missing recognition or failed routing from recognition to abstention. Across instruction-tuned models from 1.7B to 70B parameters, a single linear direction in the hidden state separates answerable from structurally impossible math and code prompts, showing that models represent impossibility before generation. Yet this recognition direction is nearly orthogonal to the canonical safety-refusal direction that mediates trained harmful-content refusal. An in-domain behavior-defined invalidity-aware direction is closer to recognition, but only partially aligned with it, and remains near-orthogonal to safety refusal. Generation-time steering along the recognition direction changes invalidity-aware behavior bidirectionally and dose-responsively on structural math and code cells, while random directions do not. Base/instruct comparisons further show that the low-cosine geometry is already present at the pretraining endpoint. The confident-on-impossible failure is therefore better explained as a routing failure than as an encoding failure: the model has a usable "no admissible answer" signal, but the safety-refusal pathway is not aligned to use it.