Reasoning Up the Instruction Ladder for Controllable Language Models

AI summary

Overview

Research area: Natural Language Processing — instruction following, instruction hierarchy, LLM controllability, and safety alignment.

Technical level: Intermediate. The paper's central idea (models should reason about which instruction wins before answering) is intuitive, but the experimental setup assumes familiarity with reinforcement learning with verifiable rewards (RLVR), supervised fine-tuning (SFT), LoRA, and benchmark terminology such as pass rate and attack success rate.

Scope: The paper reframes instruction hierarchy (deciding whether a system prompt or a user prompt takes precedence) as a reasoning task, builds a synthetic training dataset with verifiable answers called VerIH, and shows that lightweight RLVR on it improves instruction following, instruction hierarchy compliance, and out-of-distribution safety behavior across five reasoning-enabled models from two model families.

What This Paper Is About

Large language models increasingly receive directives from multiple sources inside one prompt — system designers, end users, and tools — and those directives sometimes conflict. Current models tend to treat all input text as equally weighted, which leaves them vulnerable to jailbreaks and prompt injection and makes their behavior hard to control by editing a system prompt. The authors' goal is to make models explicitly reason about the relationship between a user request and higher-priority instructions before generating a response, so that higher-priority instructions reliably override conflicting lower-priority ones.

Key Contributions

1. A reframing of instruction hierarchy as meta-reasoning. Rather than learning an implicit input-to-output mapping, the model is trained to first reason about "what task should be executed, who issued the instruction, and which instruction takes precedence if there is a conflict," then execute the task.

2. VerIH, a dataset of constraint-following tasks with verifiable answers. VerIH contains 7,192 samples drawn from 24 categories of the RLVR-IFEval dataset. Half of the pairs were rewritten by Claude-4-Sonnet so that the user prompt conflicts with the system prompt, while preserving the original intent and keeping the original verification functions applicable. The remaining half stay aligned.

3. An RLVR training recipe that transfers general reasoning to instruction prioritization. Models are trained with Group Relative Policy Optimization (GRPO) using a system-prompt hint called SysHint that asks the model to reason about the system–user instruction relationship, producing a Chain-of-Thought inside <think>...</think> tokens followed by an answer.

4. Evidence that instruction-hierarchy reasoning generalizes out of distribution to safety. No safety data appears in training, yet adding a higher-priority system prompt (GuardRules) at evaluation time reduces jailbreak and prompt-injection success, with the paper reporting up to a 20% absolute reduction in attack success rate. Code and dataset are released at https://github.com/skai-research/VerIH.

Main Findings

• Consistent gains on instruction following and instruction hierarchy. Training on VerIH improves every model tested. For Qwen3-4B, Qwen3-8B, and Qwen3-14B, the gains over the best baseline are +16.42, +7.17, and +7.47 points on IFBench, and +22.87, +17.00, and +18.21 points on IHEval-conflict. For Phi-4-mini-reasoning, improvements are larger on some benchmarks: +16.43 on IFEval, +14.03 on IFBench, +20.62 on IHEval-align, and +18.13 on IHEval-conflict.

• General reasoning is preserved. MMLU (5-shot) and MATH-500 (pass@1) scores stay similar or slightly improve after training, indicating the method does not trade away general capability. The authors report roughly 7K training examples suffice.

• A larger model still improves, even from a strong baseline. Qwen3.6-35B-A3B already reaches 90.96% on IHEval-conflict with CoT+SysHint, and VerIH still produces consistent gains across evaluated benchmarks.

• RLVR outperforms SFT. On Qwen3-8B, full SFT (filtered) reaches 65.28% on IHEval-conflict but drops MMLU to 67.88% and MATH-500 to 75.80%; unfiltered full SFT performs better than filtered (85.60% aligned, 62.35% conflict). LoRA reduces some degradation (78.34% MMLU, 85.80% MATH-500) but does not match base general performance. VerIH reaches 89.89% aligned, 63.48% conflict, 80.63% MMLU, and 94.20% MATH-500.

• Random rewards do not help. RLVR with a random reward produced no gains on IHEval (41.14% conflict), which the authors note contrasts with prior work reporting random rewards can improve Qwen reasoning.

• Safety improves without safety training data. Compared with the strongest baseline: Qwen3-4B improves 18.60 percentage points on WildJailbreak:harmful and 8.03 points on TensorTrust:inject; Qwen3-8B by 22.80 and 16.55 points; Qwen3-14B by 27.60 and 16.49 points; Qwen3.6-35B-A3B by 5.70 and 5.01 points; and Phi-4-mini-reasoning gains 15.31 points on Harmbench, 16.95 points on WildJailbreak:harmful, and 19.72 points on TensorTrust:helpful. The authors note a higher ASR on TensorTrust:inject for Phi-4-mini-reasoning, which they attribute to the trade-off between rejection and over-rejection.

• Ablations show both reasoning and conflicting samples are necessary. On Qwen3-8B, +VerIH achieves 38.21% IFBench, 89.89% IHEval-aligned, and 63.48% IHEval-conflict. Disabling CoT during training drops these to 31.34%, 56.95%, and 45.30%. Training only on aligned prompts (+VerIF) gives 35.22% IFBench, 88.53% aligned, and 54.03% conflict — comparable on aligned benchmarks but a 10–25 point drop on conflict benchmarks.

• Training increases explicit reasoning about instruction hierarchy. Judge analysis with Claude-4-Sonnet on Qwen3-8B traces shows the explicit IH reasoning ratio rising from 65.43% to 77.88% on IHEval:aligned and from 68.06% to 91.53% on IHEval:conflict.

• Gains transfer to multi-turn dialogue. On the IHEval multi-turn subset, Qwen3-8B goes from 40.63% (CoT+SysHint) to 84.53% conflict accuracy with VerIH, and from 87.90% to 92.25% on aligned.

• Real-world policy compliance improves. On OpenAI's Model Spec Evals, Qwen3-8B reaches 74.30% with CoT, 86.60% with CoT+SysHint+Policy, and 89.70% with VerIH — comparable to GPT-5 Thinking (89.00%) and GPT-5.4 Thinking (87.00%) as reported by Guo and Wolfe (2026).

• Test-time compute does not help here. Using budget forcing (replacing the </think> token with a "wait" token to extend CoT) produced no significant gains on IHEval for Qwen3-8B; the authors observe Qwen3 and Phi-4-mini-reasoning already emit "wait" tokens.

• Rewritten data is noisy but usable. An LLM-based check of 500 randomly sampled conflict examples found an 87.53% conflict rate (12.47% non-conflict). Pass rates under the original reward functions were 66.36% for aligned samples and 58.14% for rewritten conflict samples, a roughly 8% drop the authors attribute to conflict problems being harder.

Methodology in Plain English

The researchers started from an existing instruction-following dataset (RLVR-IFEval) where each example already pairs a system prompt that imposes constraints with a user prompt that gives a task, and where success can be checked automatically by simple functions (for example, "your entire response should be in lowercase letters"). These examples are naturally "aligned" cases.

To create conflict, they prompted Claude-4-Sonnet to subtly rewrite the user instructions so they contradict the system prompt while preserving the original intent — for instance, adding "avoid using any special formatting" to a user request when the system prompt demands at least 8 markdown-highlighted sections. Because only the user side changes, the original verifier still works. Half the 7,192 samples were rewritten this way.

Training then uses reinforcement learning with verifiable rewards. The system prompt includes a hint (SysHint) telling the model to think step by step about the relationship between system and user prompts and to let the system prompt take precedence in conflicts. The model produces a reasoning trace plus an answer, and a reward function scores the answer automatically; that score is the learning signal. This lets the model's existing reasoning ability be redirected toward deciding which instruction wins, rather than teaching a new input–output association.

For safety evaluation, the authors treat safety as a special case of conflict. They add a GuardRules system prompt (adopted from Brahman et al., 2024) at inference time only, with no safety examples in training, and test whether the model resists adversarial user prompts that clash with it. Training used GRPO with batch size 128, group size 4, 12 epochs / 600 steps, maximum response length 2048 tokens, on 4x H100 GPUs for 12–18 hours. All models were trained with full parameters except Qwen3.6-35B-A3B, which used LoRA with rank=64 on Tinker.

Why This Matters

Impact on research. The paper argues that instruction hierarchy is a context-dependent, compositional reasoning problem rather than a simple mapping that can be internalized, and it puts safety in the position of an emergent property of conflict resolution rather than a separate training objective. It also contrasts with prior work that requires retraining for new principles, proposing instead that behavior can be steered by prompt-based programming — editing higher-priority instructions at inference time.

Real-world applications:

• Deploying assistants with organizational policies in the system prompt that must hold even when users (or injected text) push against them.
• Defending against prompt injection and jailbreak attacks, which the paper measures directly on Harmbench, WildJailbreak, and TensorTrust.
• Multi-turn customer-facing agents where constraints issued earlier must survive later turns.
• Policy-driven deployments where operators need predictable control — for example, swapping policy text at inference time instead of rebuilding the model.

Industry relevance. The reported Qwen3-8B score of 89.70% on OpenAI's Model Spec Evals, trained with roughly 7K synthetic examples and a lightweight RLVR loop, suggests that small, task-specific training runs can move open models close to frontier systems on compliance-style evaluations — an appealing cost profile for organizations that need controllable behavior without full retraining pipelines.

Future Directions

• Broader evaluation. The authors state their methods need verification on extended benchmarks, especially multi-round dialogue and automated attacking methods, which "may reveal new challenges for instruction-hierarchy reasoning."
• Scaling and model coverage. Experiments were limited by compute to relatively small models (4B–35B), modest data (7,200 samples), and a fixed training schema (GRPO with group_size=4). The authors suggest extending to newer reasoning models such as Olmo3.
• Controlling token cost. Reasoning before acting increases response length; the authors point to work reducing CoT overhead via RLVR with GRPO and suggest integrating such approaches with their framework.
• Disentangling refusal behavior. The paper calls for further experiments to separate harmful-output suppression from over-refusals and to better quantify robustness in safety settings, and speculates that adding a small amount of safety-related data could further improve results.

Target Audience

This paper is most useful to researchers and engineers working on LLM alignment, instruction following, and safety — particularly those interested in reinforcement learning with verifiable rewards, prompt-injection defenses, or controlling model behavior through system prompts rather than weight updates. Practitioners building policy-driven agents will find the Model Spec Evals results and the prompt-based control framing directly applicable. Readers new to RL or to benchmark methodology will need background reading, but the core idea and the worked conflict examples make the motivation accessible.

Authors’ abstract

As large language model (LLM) based systems take on high-stakes roles in real-world decision-making, they must reconcile competing instructions from multiple sources within a single prompt context. Enforcing an instruction hierarchy, where higher-level directives override lower-priority requests, is critical to the reliability and control of LLMs. In this work, we reframe instruction hierarchy resolution as a reasoning task. The model must first "think" about the relationship between a given user prompt and higher-priority instructions before generating a response. To enable this capability, we construct VerIH, a training dataset of constraint-following tasks with verifiable answers, comprising aligned and conflicting system-user instructions. We show that lightweight reinforcement learning with VerIH effectively transfers general reasoning capabilities of models to instruction prioritization. Our method leads to consistent improvements across multiple model families on both instruction following and instruction hierarchy benchmarks, achieving ~20% absolute improvement in conflict setups. Our method also leads to improved alignment to safety-critical scenarios beyond the training distribution, exhibiting increased robustness against jailbreak and prompt injection, reducing absolute attack success rates by up to 20%. Our results establish reasoning over instruction hierarchies as a practical mechanism for improving AI reliability, where targeted updates to system prompts produce predictable, controllable, and robust changes in model behavior.