Health-ORSC-Bench: A Benchmark for Measuring Over-Refusal and Safety Completion in Health Context

AI summary

Overview

Research area: AI safety alignment and evaluation, specifically applied to healthcare/medical LLM deployments.

Technical level: Intermediate — accessible to readers familiar with basic LLM alignment concepts, but the evaluation protocol and benchmark construction pipeline assume some background in NLP benchmarking.

Scope: This paper introduces Health-ORSC-Bench, the first large-scale healthcare benchmark for measuring both over-refusal of benign queries and the quality of "safe completion" (helpful-but-cautious answers) across 30 state-of-the-art language models.

What This Paper Is About

Safety-aligned language models tend to treat refusal as a binary decision — either answer or refuse. In healthcare, this breaks down: models refuse many benign questions that a clinician would answer (over-refusal), while sometimes complying with genuinely dangerous ones (unsafe compliance). The paper's goal is to build a rigorous benchmark that measures not just whether a model refuses, but whether it can deliver safe, useful guidance on borderline health questions without crossing into actionable harm.

Key Contributions

1. Health-ORSC-Bench, the first over-refusal and safety-completion (ORSC) evaluation framework for the health domain, containing 31,920 benign boundary prompts across 7 categories, generated via an automated pipeline with human validation.
2. A tiered difficulty methodology splitting the benchmark into Easy-5K, Medium-5K, and Hard-1K subsets based on how many models refuse each prompt, enabling separate measurement of Over-Refusal Rate (ORR) and Safe Completion Rate (SCR).
3. A dual-metric evaluation of 30 LLMs spanning 8 model families (proprietary, open-source, and medical-specialised), establishing baseline ORR and SCR scores and quantifying the inverse relationship between safety guardrails and helpfulness.
4. A validated evaluation protocol, including a 7-model moderation ensemble for prompt filtering, human category annotation (Fleiss' κ = 0.745), and a three-judge cross-validation study confirming the stability of SCR measurements.

Main Findings

• The "ideal" region is empty. No model achieves both strong rejection of toxic prompts and low refusal of benign prompts. The top-left corner of the safety-vs-over-refusal plot — where models should sit — remains largely unoccupied.

• Safety optimisation causes severe over-refusal. Safety-tuned frontier models (GPT-5, GPT-OSS-120B) refuse up to 80% of the "Hard" benign prompts. GPT-OSS-120B tops the refusal table at 81.1% overall on Health-ORSC-Hard-1K.

• Model family and size matter more than scale alone. Larger frontier models (GPT-5, Llama-4) exhibit what the authors call "safety-pessimism," refusing more than smaller or Mixture-of-Experts counterparts. Qwen-Max, Qwen-Plus, and Qwen3-Next-80B-A3B show near-zero over-refusal (0.1–0.2%) but weaker rejection of genuinely toxic prompts.

• Domain-specialised models trade safety for utility. Medical models (Meditron-7B, BioMistral-7B, AlpaCare-7B, Clinical-Camel-7B) cluster in the bottom-left region with under-15% over-refusal, but they also provide much weaker safeguards. MedGemma-4B is the outlier at 47.6%.

• Category-specific sensitivities vary sharply by family. Claude is most sensitive to Biological/Chemical Harm, Gemini to Mental Abuse, Mistral to Mental Abuse and Unqualified Medical Advice, and Llama to Health Privacy.

• Safe completion trends mirror over-refusal. Qwen-Max performs best overall on SCR — near-zero over-refusal with roughly 70% safe completion — while Meditron-7B ranks lowest, below 10%.

• Cross-judge validation holds up. Using Grok-4, DeepSeek-3.2, and GLM-5 as independent judges, model rankings and behavioural trends stay stable, with standard deviations typically 1–15% (higher only in subjective boundary cases).

• Safety filters misfire on legitimate academic framing. Case studies show 15 models refused a benign dermoscopy query and 21 refused a sociology-of-science question about respiratory medicine, because jargon triggered disinformation heuristics.

Methodology in Plain English

The researchers built the benchmark in five steps:

1. Collect toxic seeds. They pulled harmful health-related prompts from seven existing datasets (AdvBench, HarmBench, DoNotAnswer, MedSafetyBench, etc.) using keyword search plus an LLM classifier, yielding 2,306 toxic seeds split into seven health categories.

2. Validate with humans. Sixteen annotators sorted sampled seeds into categories. Inter-annotator agreement was substantial (Fleiss' κ = 0.745; 92% majority agreement), confirming both the quality of the seeds and the distinctness of the taxonomy.

3. Rewrite toxicity into benign boundary prompts. They fed each toxic seed to Kimi-K2 (an open-weight, lightly censored model) with rewrite rules to produce a parallel prompt that looks similar but is actually benign — the kind that trips safety filters. This generated 39,886 candidate prompts.

4. Filter for residual toxicity. Seven independent safety moderator models (Granite-Guardian, Llama-Guard-3, NemoGuard, Qwen3Guard, GPT-OSS-Safeguard, ShieldGemma-2, WildGuard) voted on each prompt. Only prompts where at least 4 of 7 models judged them non-harmful survived, leaving 31,920.

5. Stratify by difficulty. Based on how many of the 30 evaluated models refused each prompt, the data was split into Easy-5K (fewer than 4 refusals), Medium-5K (5–9 refusals), and Hard-1K (10+ refusals).

For evaluation, models were run with temperature 0 and no system prompt. Refusal was detected via keyword matching. Safe completion was scored by an LLM-as-judge framework (Grok-4, with two other judges for validation), which labels each response's intent (Benign / Dual-use / Malicious) and helpfulness level (No Value / Safety Education / Partial Answer / Full Answer).

Why This Matters

Impact on research. The paper reframes medical AI safety evaluation from "does it refuse?" to "does it refuse appropriately, and does it help when it should?" It provides a shared, reproducible baseline that other researchers can build on, and it supplies concrete evidence that current alignment methods create a genuine tension between safety and utility rather than solving both.

Real-world applications:

• Clinical decision-support tools — developers can use ORR/SCR scores to tune models so they answer patient questions about, say, medication interactions, without either clamming up or giving dangerous dosing advice.
• Consumer health chatbots — near-zero over-refusal on benign queries directly reduces the risk of users abandoning a reliable tool for unmoderated sources after being refused repeatedly.
• Regulatory and compliance review — provides measurable criteria for auditing medical AI products before deployment, moving beyond vague "safety" claims.
• Mental-health triage systems — since roughly 30% of the benchmark covers Self Harm and Mental Abuse, it offers targeted diagnostics for crisis-response assistants.

Industry relevance. The finding that MoE models (like Qwen3-Next) and certain smaller models calibrate better than large frontier models is directly actionable for companies choosing a base model for a health product. It also suggests that "bigger and more safety-tuned" is not automatically the right trade-off.

Future Directions

• Multilingual extension. The benchmark is English-only, yet safety guardrails are known to be weaker in low-resource languages — a critical gap for globally deployed health AI.

• Expanding the risk taxonomy. Seven categories exclude areas like insurance fraud, hospital cybersecurity, and subtle treatment-recommendation bias, all of which matter clinically.

• Moving beyond synthetic prompts. The boundary prompts are LLM-generated, so they may lack the natural phrasing and messy context of real patient–AI conversations. Collecting genuine user queries would strengthen ecological validity.

• Developing training methods that improve ORR and SCR simultaneously. The paper identifies the problem clearly but offers no training recipe; context-aware confidence estimation and joint safety-helpfulness optimisation are flagged as promising directions.

Target Audience

This paper is most valuable to AI safety and alignment researchers working on refusal calibration, medical NLP practitioners building or evaluating clinical assistants, and ML engineers and product teams selecting or fine-tuning models for health applications. Regulators and clinical informatics specialists assessing the risks of deployed medical AI will also find the benchmark construction methodology and the ORR/SCR framing directly useful.

Authors’ abstract

Safety alignment in Large Language Models is critical for healthcare; however, reliance on binary refusal boundaries often results in over-refusal of benign queries or unsafe compliance with harmful ones. While existing benchmarks measure these extremes, they fail to evaluate Safe Completion: the model's ability to maximise helpfulness on dual-use or borderline queries by providing safe, high-level guidance without crossing into actionable harm. We introduce Health-ORSC-Bench, the first large-scale benchmark designed to systematically measure Over-Refusal and Safe Completion quality in healthcare. Comprising 31,920 benign boundary prompts across seven health categories (e.g., self-harm, medical misinformation), our framework uses an automated pipeline with human validation to test models at varying levels of intent ambiguity. We evaluate 30 state-of-the-art LLMs, including GPT-5 and Claude-4, revealing a significant tension: safety-optimised models frequently refuse up to 80% of "Hard" benign prompts, while domain-specific models often sacrifice safety for utility. Our findings demonstrate that model family and size significantly influence calibration: larger frontier models (e.g., GPT-5, Llama-4) exhibit "safety-pessimism" and higher over-refusal than smaller or MoE-based counterparts (e.g., Qwen-3-Next), highlighting that current LLMs struggle to balance refusal and compliance. Health-ORSC-Bench provides a rigorous standard for calibrating the next generation of medical AI assistants toward nuanced, safe, and helpful completions. Furthermore, our benchmark facilitates reproducible evaluation, encourages safety calibration, and supports development of clinically reliable, context-aware, human-aligned medical AI systems. Our code and data are available at: https://github.com/ZhihaoZhang97/Health-ORSC-Bench. Warning: Some contents may include toxic or undesired contents.