Semantic Gravity Wells: Why Negative Constraints Backfire

AI summary

Overview

Research area: Mechanistic interpretability of large language models, specifically instruction-following failure under negative constraints.

Technical level: Advanced. The paper assumes familiarity with transformer internals, attention patterns, the logit lens, and activation patching.

Scope: The paper presents a mechanistic and causal account of why instruction-tuned language models violate negative constraints (instructions of the form "do not use word X"), using a single open-weights model, Qwen2.5-7B-Instruct.

What This Paper Is About

Users frequently tell language models not to say a specific word, and the model often says it anyway. Rather than attributing this to a generic failure to understand negation, the paper argues that two competing computational pressures are at work: "semantic pressure," the model's baseline inclination toward a natural completion, and "constraint pressure," the suppression signal created by the negative instruction. The goal is to measure both pressures quantitatively, trace where in the network the conflict is resolved, and establish causally which layers drive the failure.

Key Contributions

1. A controlled dataset of 2,500 single-word prompts with validation, designed to isolate negative-constraint compliance, spanning five semantic categories of 500 prompts each (idioms, factual, common sense, creative, and out-of-distribution).
2. A definition of semantic pressure (P₀, the baseline probability of producing the forbidden word) and a demonstration of a tight logistic relationship between P₀ and violation probability across 40,000 stochastic generations (R² = 0.78).
3. Identification of a suppression asymmetry between successes and failures using layer-wise analysis: the instruction reduces target probability by 22.8 percentage points in successes versus 5.2 points in failures, a 4.4× difference.
4. A taxonomy of two mechanistically distinct failure modes — priming (87.5% of violations, n=441) and override (12.5%, n=63) — with the responsible layers localized causally through activation patching.

Main Findings

• Violation rate is predictable from baseline probability. Fitting p(violation) = σ(β₀ + β₁ · P₀) yields β₀ = −2.40 (95% CI [−2.44, −2.35]) and β₁ = +2.27 (95% CI [2.21, 2.33]), with the model explaining 78% of variance. At P₀ = 0.1, only 9% of samples violate the constraint; at P₀ = 0.9, violations exceed 46%.

• Suppression is real but asymmetric. Both successes and failures show positive suppression magnitude ΔP = P₀ − P₁. Successes: ΔP = 0.228 (95% CI [0.211, 0.245]). Failures: ΔP = 0.052 (95% CI [0.041, 0.063]). The intervals are non-overlapping, ruling out the hypothesis that the model simply ignores the constraint.

• Failures attend to the forbidden word rather than the negation. At high pressure (P₀ > 0.8), failures show elevated Target-Mention Focus (0.205 vs. 0.200), slightly lower Negation Focus (0.141 vs. 0.142), and higher overall Instruction Attention Ratio (0.282 vs. 0.272). The paper cautions that attention weights are a routing signal, not proof of information use.

• Target probability emerges explosively in late layers. Logit lens analysis reveals three regimes: early layers (0–20) where all conditions sit below P(X) < 10⁻⁴; critical layers (21–27) where curves diverge sharply; and the final layer, where the baseline/failure condition reaches 0.71 while the negative/success condition stays at 0.08. For failure-prone prompts, the constraint barely helps: baseline 0.71 versus negative instruction 0.66. For success-prone prompts, baseline 0.30 drops to 0.08.

• Attention suppresses while feed-forward networks promote, and the FFN wins in failures. At layer 27, successes show FFN = +0.10 and attention = −0.05; failures show FFN = +0.39 and attention = −0.13. The suppression signal is actually stronger in failures, but it is overwhelmed by an FFN push nearly 4× larger.

• Layers 23–27 are causally responsible. Activation patching on high-pressure prompts (P₀ ≥ 0.8, n = 769) shows that transplanting baseline activations into negative-instruction runs decreases target probability in layers 0–22 (negative ΔP from −0.10 to −0.21), crosses zero at layer 23 (+0.0004), and increases target probability in layers 24–27 (up to +0.07).

• Two failure modes account for all violations. Priming failure (87.5%, n=441) occurs when the instruction's explicit mention of the forbidden word activates rather than suppresses it; the illustrated example shows Target-Mention Focus of 0.30 versus Negation Focus of 0.11 for the target "book," meaning the model attended 3× more to the forbidden word than to "do not." Override failure (12.5%, n=63) occurs when suppression exists but late-layer FFN contributions overwhelm it; the illustrated example (target "Real") shows ΔP = 0.92 with a final probability of P(X) = 0.82 and an FFN contribution at layer 27 of +0.34.

Methodology in Plain English

The authors built a test set of 2,500 prompts that each have one obvious expected answer — idioms, facts, common sense, creative, and surreal prompts — and validated them using GPT-5.2 scoring against four criteria: unique best answer, low ambiguity, no answer leakage, and naturalness. Prompts were gated to a baseline probability of at least 0.20 and bin-balanced across the pressure range.

Each prompt was run under two conditions: a plain baseline and a negative instruction ("Do not use the word '[X]' in your answer"). Semantic pressure was computed by summing the probabilities of all valid token sequences that decode to the target word — capitalization, whitespace, and punctuation variants alike — measured under the baseline condition. For behavioral evaluation, the model generated 16 stochastic samples per prompt at temperature 1.0 and top-p 0.9, giving 40,000 samples, with violation detected by deterministic case-insensitive substring matching with word-boundary checking.

To find out why failures happen, the researchers projected hidden states through the unembedding matrix at every layer (the logit lens) to see when the target probability appears; split each layer's contribution into attention and feed-forward components; measured where attention flowed within the instruction; and then used activation patching — swapping residual-stream activations from baseline runs into negative-instruction runs — to test whether late layers merely correlate with failure or actually cause it. All experiments ran on a single NVIDIA A100-80GB GPU, on Qwen2.5-7B-Instruct, chosen because its open weights permit internal access and because it has 28 transformer layers, enough scale to show the effect while remaining tractable.

Why This Matters

Impact on research. The paper connects a widely observed behavioral quirk to specific computational mechanisms, from attention routing to component contributions to causal interventions. The authors position it as the first complete mechanistic account of instruction-following failure in large language models, and as a demonstration that interpretability techniques can move from describing behavior to explaining it. It also distinguishes itself from prior negation work, which studied comprehension in masked-language and inference settings rather than compliance during generation.

Real-world applications:

• Safer prompt design. Since priming failures dominate at 87.5% of violations, the paper argues that naming a forbidden word is itself the primary risk factor. Category-level constraints ("Do not mention any cities"), positive reframing ("Use only general geographic terms"), or circumlocution may work better — though the authors explicitly leave systematic evaluation of alternative phrasings to future work.
• Content moderation and output filtering. Because generation-time constraints may be insufficient for high-pressure cases, the paper recommends combining generation constraints with post-generation filtering in safety-critical applications.
• Difficulty estimation before deployment. Systems can measure semantic pressure up front and apply additional safeguards where the natural answer is strongest.
• Runtime diagnostics. Elevated Priming Index correlates with failure risk, suggesting that monitoring attention routing in real time could enable early detection of constraint failures.

Industry relevance. Any deployed system that relies on negative instructions — content restrictions, brand-safety rules, formatting prohibitions, refusal behaviors — faces the tension this paper describes. The finding that the model partially complies even when it fails, and that failure rates are predictable from a single measurable quantity, gives practitioners something actionable rather than purely behavioral.

Future Directions

• Cross-model validation. Whether the pressure–violation relationship and the two failure modes generalize to larger models (70B+), other architectures such as Mixture of Experts and state-space models, and other training procedures such as RLHF variants and Constitutional AI remains untested; all experiments here use a single model.
• Systematic comparison of alternative phrasings. The paper proposes category-level constraints, positive reframing, and circumlocution as likely improvements over explicit target naming, but does not evaluate them.
• Finer-grained causal localization. The work patches full residual streams; patching individual attention heads or FFN neurons could pinpoint which components process negation versus which activate the target, and targeted knockout experiments could establish the causal role of attention patterns.
• Safety-relevant constraints and naturalistic settings. Extending the analysis to safety-oriented negative constraints, multi-token targets (this study analyzes the first token of multi-token targets), longer outputs, and ambiguous constraints would test how far the findings travel beyond controlled one-word completions.

Target Audience

This paper is best suited to mechanistic interpretability researchers and AI safety engineers who want a causal, layer-level explanation of instruction-following failure rather than a behavioral one. It will also benefit prompt engineers, evaluation specialists, and product teams who deploy negative constraints in production and need to know when those constraints can be trusted. Readers without familiarity with logit lens probing, attention metrics, and activation patching will find the middle sections demanding, though the behavioral results and the failure taxonomy are accessible on their own.

Authors’ abstract

Negative constraints (instructions of the form "do not use word X") represent a fundamental test of instruction-following capability in large language models. Despite their apparent simplicity, these constraints fail with striking regularity, and the conditions governing failure have remained poorly understood. This paper presents the first comprehensive mechanistic investigation of negative instruction failure. We introduce semantic pressure, a quantitative measure of the model's intrinsic probability of generating the forbidden token, and demonstrate that violation probability follows a tight logistic relationship with pressure ($p=σ(-2.40+2.27\cdot P_0)$; $n=40{,}000$ samples; bootstrap $95%$ CI for slope: $[2.21,,2.33]$). Through layer-wise analysis using the logit lens technique, we establish that the suppression signal induced by negative instructions is present but systematically weaker in failures: the instruction reduces target probability by only 5.2 percentage points in failures versus 22.8 points in successes -- a $4.4\times$ asymmetry. We trace this asymmetry to two mechanistically distinct failure modes. In priming failure (87.5% of violations), the instruction's explicit mention of the forbidden word paradoxically activates rather than suppresses the target representation. In override failure (12.5%), late-layer feed-forward networks generate contributions of $+0.39$ toward the target probability -- nearly $4\times$ larger than in successes -- overwhelming earlier suppression signals. Activation patching confirms that layers 23--27 are causally responsible: replacing these layers' activations flips the sign of constraint effects. These findings reveal a fundamental tension in negative constraint design: the very act of naming a forbidden word primes the model to produce it.