Safety for Whom? Boundary-Aware Self-Distillation for Controlled LLM Safety Refusal

AI summary

Overview

Research area: Safety alignment and controllability of large language models, specifically fine-grained refusal behaviour within a topic rather than topic-wide refusal.

Technical level: Advanced (assumes familiarity with supervised fine-tuning, LoRA adapters, KL regularisation, and safety benchmark terminology such as refusal rates and over-refusal).

Scope: The paper formulates "narrow-boundary safety," builds an offline self-generated data pipeline with coverage repair and boundary-pair data, and evaluates the resulting LoRA fine-tunes of Qwen3-8B on political persuasion as a test domain, reporting safety and over-refusal jointly.

What This Paper Is About

Most safety alignment treats harm as a topic-level property, asking only whether a subject is harmful. Real deployments need narrower rules: a civics tutor and a public-sector assistant may need to refuse manipulative political persuasion while still answering factual questions about the same election. The authors formalise this as learning a refusal boundary inside a topic — refusing a target harmful subset while preserving the benign complement — and build a self-generated data pipeline that repairs coverage gaps and supplies paired harmful/benign prompts to measure where that boundary actually falls.

Key Contributions

1. Narrow-boundary safety formulation. Safety is defined as refusing a target harmful subset H inside a broader topic universe Ω, rather than refusing Ω entirely. The setting is operationalised with 1,539 held-out harmful-benign boundary pairs per side, testing both the refusal-worthy and comply-worthy sides.

2. Coverage-oriented self-generation. The authors identify a coverage gap in single-shot self-generated refusal data that silently discards 8,009 prompts, or 19.88% of the audited prompt pool, and introduce coverage variants (Graft, Escalate, Escalate+Graft) that complete training data instead of dropping hard prompts, reducing residual failures to 79 prompts, or 0.20%.

3. Controlled topic-specific data construction. A hierarchical generation pipeline with persona, length, style, and paraphrasing controls yields 40,293 harmful training prompts after coverage repair. The same pipeline was used to build a religion-domain dataset (not used in any experiment reported here); datasets and generation code are stated for release on publication.

4. Compensation for downside reactions and boundary-aware evaluation. In-distribution compensation data includes 11,955 verified surface-dangerous benign prompts across 18 semantic types, and harmful-benign prompt pairs measure boundary precision directly.

5. Safety transfer and its confound. Refusal tuning on harmful political prompts alone lowers unsafe behaviour on broader benchmarks (from 0.2626 to 0.0014 in the strongest configuration), but this is not separable from a rise in false-positive refusal (from 0.0200 to 0.7400 on XSTest at the same checkpoint), so both axes are reported jointly.

Main Findings

• Coverage repair closes the drop set. Single-shot generation leaves 19.88% of prompts without accepted refusal traces, whereas escalating retries leave 0.20%. Graft and Escalate are interpreted as ways to retain supervision for difficult prompts, not as a final performance ordering.

• Strong target-domain refusal with a large over-refusal cost. On political persuasion with Qwen3-8B, training on refusal data completed through Escalate raises in-distribution political refusal from 0.0947 to 0.8475 and reduces harmful_unsafe_avg from 0.2626 to 0.0014 at epoch 4, while XSTest over-refusal rises from 0.0200 to 0.7400.

• Response source matters for over-refusal. Replacing externally adopted SafeChain responses with target-model-generated and verified responses reduces XSTest over-refusal at epoch 4 from 0.1520 to 0.0520 under Single-shot generation and from 0.2520 to 0.0440 under Graft, at a modest increase in harmful_unsafe_avg.

• Surface-dangerous benign data addresses a different failure mode. Relative to refusal-only training, adding FakeHarm reduces both XSTest over-refusal and harmful_unsafe_avg in the Escalate+Graft family, and performs better than externally adopted compliance data on both plotted metrics. Combining FakeHarm with general compliance data further lowers over-refusal but increases the unsafe-response rate relative to FakeHarm alone.

• Boundary pairs improve local precision at a small recall cost. Adding PB in the clean Single-shot ablation at epoch 4 reduces comply-side PB-OOD over-refusal from 0.3294 to 0.0416, while refusal-side PR-OOD falls more modestly, from 0.9188 to 0.8772.

• Compensation components are not simply additive. The two effects emerge at different training stages, motivating checkpoint selection using both harmful and benign evaluations.

• Length control affects which prompts receive reliable supervision. The length-controlled construction yields higher political refusal than the natural-length PR construction in every prompt-length bucket, with the largest gap for two-to-eight-word prompts.

• Cross-model and cross-topic generalisation are not established. Results are limited to political persuasion, mostly to Qwen3-8B and a single LoRA configuration, with selected coverage-repair experiments also using DeepSeek-R1-Distill-Qwen-7B.

Methodology in Plain English

The authors treat harm as a subset rather than a topic. For a chosen topic — politics — they define a target-harmful subset (prompts asking for manipulation, propaganda, radicalisation, or targeted persuasion) and a benign complement (factual civics questions, neutral summaries, non-persuasive explanations). The ideal behaviour is to refuse the first and answer the second.

To build training data, they generate prompts hierarchically: topic, then subtopic, then intent points, then persona-conditioned requests, with controls over persona, style, paraphrasing, and prompt length. For harmful prompts, the target model is steered to produce a refusal, and a guard model (WildGuard) verifies whether the trace really is a refusal. Prompts that fail to yield an accepted refusal are the "drop set." Three strategies recover them: Graft pairs a failed prompt with an accepted topic-neutral refusal, Escalate retries the prompt through up to four tiers of resampling and stronger steering, and Escalate+Graft applies the ladder first and then grafts whatever remains, using each neutral refusal at most five times.

Benign behaviour is protected in two ways. Benign examples are trained with forward-KL regularisation against the frozen reference model so the model does not drift on harmless prompts, and harmful examples use cross-entropy on the refusal target. Three kinds of benign data are used: externally adopted SafeChain responses (SC), target-model-generated and verified replacements for the same prompts (SC2), and locally constructed pairwise benign data (PB, PB2) plus surface-dangerous benign prompts (FH) that look harmful but are not.

Evaluation deliberately splits the question. Target-domain refusal is measured on political prompts, unsafe behaviour on HarmBench, StrongREJECT, and WildJailbreak with LlamaGuard-3-8B, over-refusal on XSTest and the benign side of held-out pairs, and boundary precision on 1,539 held-out harmful-benign pairs per side. Training uses LoRA adapters of rank 32, alpha 16, and dropout 0.05, with AdamW at a learning rate of 1×10⁻⁵, cosine scheduling, warm-up ratio 0.1, bf16, effective batch size 8, maximum sequence length 16,384, seed 42, on one H200 GPU for up to 12 epochs with per-epoch checkpoints.

Why This Matters

Impact on research. The paper argues that safety alignment should be evaluated on both sides of the intended refusal boundary. Standard practice reports harmful-compliance reduction alone, which can hide a model that has simply become a blunt refusal machine — the authors' own configuration with the lowest harmful-response rate also refuses 74% of safe XSTest prompts. Framing safety as a boundary rather than a topic also exposes a limitation of guard-based methods: a method that defines safety as agreement with a guard model cannot express a boundary the guard does not already encode, as the narrow LlamaGuard-3 election category illustrates.

Real-world applications:

• Educational products such as a civics tutor that must answer factual questions about an election while refusing requests to craft targeted political persuasion.
• Public-sector and enterprise assistants with deployment-specific policies that differ inside the same topic, sharing a base model but needing different boundaries.
• Safety-data engineering workflows where failed self-generation is currently discarded, causing the hardest prompts to be silently dropped from training.
• Refusal calibration for reasoning models, where post-training is expected to preserve deliberation and problem-solving while maintaining reliable refusal.

Industry relevance. The results are directly actionable for teams that fine-tune a single base model for multiple products: data composition — the mix of harmful refusal data, in-distribution compensation data, and boundary pairs — controls where a checkpoint lands in the safety-versus-usability space. The paper also notes that response source matters (externally adopted traces versus verified target-model traces), and that components are not simply additive, implying that checkpoint selection must use both harmful and benign evaluations rather than harmfulness alone.

Future Directions

• Test other topics and model families. The authors state that their evidence is limited to political persuasion; the religion-domain dataset was constructed with the same pipeline but has not been used in any experiment, and the corresponding religion in-distribution, held-out, PR-OOD, and PB-OOD files have not been built.
• Broaden model and configuration coverage. Most experiments use Qwen3-8B with a single LoRA configuration, with DeepSeek-R1-Distill-Qwen-7B reserved for cross-model diagnostics, so generalisation across topics, model families, and training configurations remains unestablished.
• Resolve the judge and metric confound. Unsafe responses across HarmBench, StrongREJECT, and WildJailbreak are judged with LlamaGuard-3, which measures general harmfulness rather than persuasion or deployment-specific neutrality; the complementary WildGuard refusal aggregate is reported but not analysed in the same detail, and is not yet backfilled for all matched checkpoints.
• Separate the causal factors in data composition. The builds used to compare response source and verification have high but incomplete prompt overlap (SC and SC2 share 4,953 exact prompt strings; PB and PB2 share 4,961), so the authors state these comparisons are consistent with distinct effects but do not isolate either factor causally.
• Document a stopping rule. The checkpoint trajectories show a recurring trade-off where later checkpoints can reduce over-refusal while worsening broader harmfulness, which points to a need for principled checkpoint selection rather than best-epoch-per-method reporting.

Target Audience

This paper is most useful to safety-alignment researchers and practitioners who fine-tune LLMs for specific deployments, particularly those working on refusal calibration, over-refusal, or synthetic safety-data pipelines. It also suits evaluation researchers interested in benchmarks that measure both sides of a refusal boundary, and product or policy teams at organisations deploying one base model across multiple use cases with differing safety requirements. Readers without a background in fine-tuning and safety benchmarks will find the formal notation in Sections 3 and 4 demanding, though the framing of the boundary problem is accessible.

Authors’ abstract

Safety alignment is usually posed as a topic-level question: is this subject harmful? Deployments ask a narrower one. A civics tutor and a public-sector assistant may share a base model yet need different boundaries inside the same topic, refusing targeted political manipulation while still answering factual questions about the same election. We formulate this as narrow-boundary safety and introduce an offline self-generated framework combining controlled topic generation, coverage repair, in-distribution compensation data, and harmful-benign pairs for training and evaluation. Single-shot generation leaves 19.88% of prompts without accepted refusal traces, whereas escalating retries leave 0.20%. On political persuasion with Qwen3-8B, training on refusal data completed through Escalate increases target-domain refusal from 9.47% to 84.75% and reduces the mean unsafe-response rate across three broader harmfulness benchmarks from 26.26% to 0.14%, but increases XSTest over-refusal from 2.00% to 74.00%. In a separate matched comparison, replacing external responses with verified target-model responses reduces over-refusal from 15.20% to 5.20%. Boundary-pair data reduces comply-side over-refusal on held-out pairs from 32.94% to 4.16%, while harmful-side refusal decreases only from 91.88% to 87.72%. These results show that data composition controls the safety and usability trade-off, and that safety alignment should be evaluated on both sides of the intended refusal boundary.