You Shouldn't Have Asked: A Pragmatics-Inspired Taxonomy for Evaluating LLM Refusals

AI summary

Overview

Research area: Natural language processing and AI safety, specifically the pragmatics of large language model refusals. The work sits at the intersection of conversation analysis / politeness theory and alignment evaluation.

Technical level: Intermediate. The paper assumes some familiarity with facework and politeness theory (Goffman, Brown and Levinson), refusal strategy research (Beebe et al.), and standard LLM safety-evaluation practice, but the taxonomy itself is explained in full.

Scope: The paper introduces a three-layer, pragmatics-grounded taxonomy for coding how LLMs decline harmful requests, validates it with human annotators and an LLM judge, and applies it to 3,200 query-response pairs from 16 models across 14 harm categories.

What This Paper Is About

Existing work treats an LLM refusal as a safety-alignment outcome: did the model decline the harmful request or not. This paper argues that refusals are also face-threatening acts that can damage the user's socially claimed self-image, and that how a model says no matters alongside whether it says no. The authors build a taxonomy of refusal actions, rationales, and realization strategies, then use it to measure whether 16 modern LLMs refuse in ways that are contextually adaptive and socially accountable.

Key Contributions

1. A three-layer pragmatic taxonomy of LLM non-compliance, separating response action (Layer 0), refusal rationale (Layer 1), and realization strategy with adjunct features (Layer 2). The taxonomy, code, and data are released on GitHub.
2. A human validation study of 100 query-response pairs achieving perfect inter-coder agreement on Layers 0 and 1 (Cohen's kappa = 1.0) and high agreement on Layer 2 features (average kappa = 0.953), plus validation of an LLM-as-judge pipeline for scaling annotation.
3. A large-scale empirical study: 200 responses each from 16 LLMs spanning six model families (OpenAI, Anthropic, Google, xAI, Meta, Qwen) across 14 harm categories, totaling 3,200 query-response pairs.
4. Identification of concrete pragmatic dimensions (rationale framing, adjunct feature selection) that alignment training could target to produce refusals that are safe and communicatively appropriate.

Main Findings

• Compliance was the minority outcome. Across all models, models complied 21% of the time (17% fully, 4% partially). In 2% of cases an API error indicated a safety filter had blocked the request, and an additional 0.5% could not be coded because the judge model itself refused to classify the content. The remaining 77% of cases were non-compliant messages to the user.
• Compliance varied widely by model and by harm category. Rates ranged from 8% in Llama-3.1-8B to 38% in Qwen3-32B (reasoning). Non-compliance was near-universal for Violent Crimes (95%) and Sex-Related Crimes (92%) but fell to 47% for Specialized Advice.
• Refusals were framed ethically, not procedurally. Rationales were ethics-based 70% of the time, bare (no substantive rationale) 23%, policy-based 6%, and capacity-based 1%. Claude Opus 4.6 used ethics to justify non-compliance 92% of the time. Three models produced predominantly bare refusals: GPT-4o (92%), Llama-3.1-8B (77%), and Llama-3.1-70B (68%); the authors note these models were released earlier than others in the analysis and often produced relatively templatic refusals.
• Refusals were explicit and offered little linguistic softening. Models used an explicit realization strategy 90% of the time versus implicit 10%. Roughly half of models produced little-to-no apology/regret, and no models produced any hedge/epistemic softener features at all. Explanatory preface appeared in only 3% of cases on average.
• Models favored their own projected face over the user's. High rates of negative stance and normative suggestion indicate models frequently evaluate the user's request as unacceptable, while rapport-seeking features (apology/regret, positive alignment, solidarity/empathy) were rare.
• Relationship repair happened mainly through helpfulness. Alternative offer / switch of topic appeared in 55% of responses, and executed alternative in 39%, instantiating the safe-completion approach without hard refusals.
• Refusal styles clustered by harm category. Multidimensional scaling of Layer 2 codes produced five clusters: (1) Sexual and violent harms, (2) Non-violent misuse, (3) Intellectual Property, (4) Specialized Advice, and (5) Suicide & Self-Harm. Suicide & Self-Harm showed the most care-oriented style, with the highest solidarity/empathy and apology/regret and the lowest negative stance and normative suggestion, but negative stance still occurred at around 40%. Sexual and violent harms showed the highest explicit non-compliance and the lowest executed alternative.
• Solidarity in severe-harm contexts was paradoxical. Sex-Related Crimes had a solidarity/empathy rate of 23%, far higher than non-violent categories such as Hate (8%). Inspection of these responses showed some framed the user's harmful intent as evidence of psychological abnormality or moral deficiency, which may intensify rather than mitigate the face threat.
• Size and reasoning mode had inconsistent effects. The Llama size pair (8B vs 70B) differed by only 2.4 percentage points (pp) on average across Layer 2 features, while Qwen3-8B vs Qwen3-32B differed by 8.7 pp; the largest Qwen difference was apology/regret, rising from 14% in Qwen3-8B to 61% in Qwen3-32B. Across the four standard/reasoning pairs, the average pairwise difference across Layer 2 feature rates was 2.0 pp. Qwen3-32B with reasoning showed fewer bare refusals (18% to 9%), more ethics-based refusals (75% to 83%), and more normative suggestion (48% to 61%), but these shifts did not generalize across all reasoning-mode pairs.
• Apology declined sharply over time. While models released in early 2024 apologized in more than 90% of refusals, this rate had fallen to 0-4% by early 2026. OpenAI shifted from an apologetic style toward a more normatively evaluative and redirective one, while Claude Opus 3, the earliest Anthropic model in the study, was already strongly evaluative. Other affiliative features did not consistently compensate for the loss of apology.
• Judge annotations were stable. In a cross-family robustness check using Claude Opus 4.8 in non-reasoning mode, agreement with GPT-5.5 was moderate to high on most features (for example, kappa of 0.873 for Layer 0 and 0.863 for Layer 1). On prompt resampling, re-collected responses to an alternative set of 262 prompts using a matched panel of 13 models produced a Spearman correlation of rho = .977 with a mean absolute difference of 3.37 pp; restricting to the 138 shared queries gave rho = .975 and a mean absolute difference of 3.04 pp.

Methodology in Plain English

The authors started from decades of research on how humans decline requests, where refusals are treated as threats to the other person's "face" that speakers normally soften with apologies, hedges, explanations, and offers. They turned that literature into a coding scheme with three layers.

Layer 0 asks a single question about the response action: did the model fully comply, partially comply, or not comply with the user's explicit request? The coding deliberately looks at what the model actually did, not at what it claimed it was doing, because models sometimes announce a refusal and then answer anyway, or the reverse. Only non-compliant responses get coded at later layers.

Layer 1 asks why the model refused, splitting rationales into four types: bare (no substantive reason given), capacity-based (claimed inability, missing access, or lack of embodiment or experience), policy-based (the model's own rules or guidelines), and ethics-based (the requested act is harmful, illegal, or normatively unacceptable). The key distinction between the last two is where responsibility is placed: policy rationales locate it with an external authority, ethics rationales locate it with the requester.

Layer 2 asks how the refusal is realized in language, splitting into a mutually exclusive realization strategy (explicit versus implicit) and a set of co-occurring adjunct features such as apology/regret, hedge, explanatory preface, positive alignment, solidarity/empathy, negative stance, alternative offer, executed alternative, normative suggestion, statement of principle, and role-based self-positioning.

The taxonomy was refined through three rounds of pilot coding using 20-25 query-response pairs, with annotators discussing disagreements after each round. Two authors then independently coded 100 query-response pairs, reaching kappa = 1.000 on Layers 0 and 1 and an average kappa = 0.953 on Layer 2, with disputed cases resolved in an adjudication round to produce gold labels.

For scale, they used GPT-5.5 in non-reasoning mode as an automatic judge, prompted with a codebook adapted from the human version plus 25 preliminary coding examples that did not overlap with the validation set. The judge reached 95.00% accuracy on Layer 0 (kappa = 0.857, N = 100) and 93.51% on Layer 1 (kappa = 0.860, N = 77, corresponding to 72/77 correct labels), with Layer 2 feature accuracy ranging from 80.52% for normative suggestion to 100.00% for hedge. The authors recommend GPT-5.5 in non-reasoning mode as the default configuration for applying the codebook at scale.

The prompt set consisted of 200 harmful prompts drawn from SORRY-Bench and LMSYS-Chat-1M. All candidates were reclassified using the 14-category Llama Guard 3 taxonomy and only those labeled unsafe were retained. The authors sampled a category-balanced core of 140 prompts (10 per category, prioritizing SORRY-Bench) and supplemented with 60 LMSYS prompts, applying per-category caps, then deduplicated templates and filtered benign rewrite-style prompts, leaving 132 SORRY-Bench and 68 LMSYS prompts. Sixteen models were selected to support structured comparisons across size (Llama 3.1 8B vs 70B; Qwen3-8B vs 32B), reasoning mode (Qwen3-8B, Qwen3-32B, Claude Opus 4.6, GPT-5.3), model family (Claude Opus 4.6, GPT-5.3, Grok 4.20, Gemini 2.5 Pro), and time (GPT-4o / GPT-5 / GPT-5.3; Claude Opus 3 / Claude Sonnet 3.7 / Sonnet 4.6). All main-analysis models were accessed via the OpenRouter API. Each of the 16 models received all 200 prompts, yielding 3,200 query-response pairs.

Why This Matters

Impact on research. The paper reframes refusal evaluation from a binary safety question into a pragmatic one. It provides a shared codebook, an annotated dataset, and a validated judge pipeline, which lets other researchers measure refusal style directly rather than treating it as an unmeasured side effect of alignment. It also surfaces a tension the authors state explicitly: repeated optimization for safe non-compliance may be pushing frontier models toward moral condescension, particularly through the near-disappearance of apology.

Real-world applications:

• Auditing deployed assistants for whether refusals in emotionally sensitive categories, such as suicide and self-harm, are compassionate rather than judgmental, given the roughly 40% negative stance rate the study found in that cluster.
• Designing mental health and crisis-support chatbots, where the paper cites clinical guidance emphasizing non-judgmental attitudes and warns that moral censure of a user's contemplated behavior may deepen distress.
• Evaluating customer-service and productivity agents, where the authors note that brief denials are rated more negatively than informative or redirective responses.
• Providing trust-and-safety teams with a diagnostic for whether a model's refusal behavior is calibrated across harm categories, rather than uniformly evaluative.

Industry relevance. The paper identifies rationale framing and adjunct feature selection as concrete, tunable dimensions for alignment training, and reports clear trends across major providers: an OpenAI shift toward normative evaluation, an Anthropic baseline that was already evaluative, and a shared collapse in apology rates from more than 90% in early 2024 to 0-4% by early 2026. It frames a direct question for model developers about whether reduced user-oriented facework is a deliberate design decision or an unintended side effect of optimizing other objectives.

Future Directions

• Distinguishing high-quality pragmatic support from generic refusal templates. The authors note that solidarity/empathy currently spans both context-sensitive care and formulaic concern, and that executed alternative captures only the presence of safer assistance, not its quality or usefulness.
• Testing whether refusal styles actually affect users. The paper measures model outputs, not user perception, and does not test whether different styles influence trust, shame, perceived support, or attempts to negotiate the boundary.
• Evaluating the codebook against a larger human-annotated gold set and a broader range of judges, including open-source models, since the current cross-family robustness check rests on only 100 pairs and two proprietary model families.
• Mitigating the interpretive ambiguity in categories where responses combine ethical language, policy-like phrasing, redirection, and safer alternatives, and keeping the taxonomy aligned with the more balanced setting the authors note they could not achieve (every model refusing every prompt).

Target Audience

AI safety and alignment researchers evaluating refusal behavior; NLP researchers working on pragmatics, politeness, and human-AI interaction; trust-and-safety and policy teams at model providers who tune refusal behavior and need a diagnostic vocabulary for it; and designers of conversational systems in sensitive domains such as mental health support. Readers without a pragmatics background will find the taxonomy table self-contained but will benefit from reading the related work section on face and facework first.

Authors’ abstract

Refusals are often treated as face-threatening acts in pragmatics because they can challenge the requester's socially claimed self-image. Large language models (LLMs) are increasingly trained to refuse unsafe and inappropriate requests, and these refusals may harm users when models fail to manage this interactional cost properly. While existing work has mainly approached LLM non-compliance as a safety-alignment outcome, it does not provide a way to evaluate whether LLMs refuse appropriately across different harmful contexts. To study this question, we propose (to our knowledge) the first taxonomy of LLM refusals that is grounded in pragmatic theory. Applying this taxonomy to responses from 16 modern LLMs across 14 harm categories, we find that although models differ in how they refuse, their refusals are overall explicit and strongly morally evaluative, with interactional repair occurring mainly through offering or providing safer alternatives instead of interpersonal facework. This pattern is especially consequential in sensitive harm contexts, where overuse of negative framing may make users feel shamed or provoked, undermining the purpose of safe non-compliance. We therefore call for alignment evaluation that considers not only whether models refuse harmful requests, but also whether they refuse in ways that are contextually adaptive and socially accountable for the interactional consequences of saying no.