HealthBench-Psych: A Mental Health Subset of OpenAI's HealthBench

AI summary

Overview

Research area: Evaluation of large language models on mental-health conversations; benchmark construction and specialty-resolved model evaluation.

Technical level: Intermediate. The paper is readable for someone familiar with LLM evaluation basics (rubrics, judges, bootstrap confidence intervals), though it assumes comfort with statistics terms like Gwet's AC1, Kendall τ, and Holm–Bonferroni correction.

Scope: The paper carves a clinician-validated mental-health subset of 610 conversations out of OpenAI's 5,000-conversation HealthBench, adds a 119-conversation hard subset, and uses it to rank 20 models under three LLM judges.

What This Paper Is About

General-purpose health benchmarks such as HealthBench are increasingly cited by AI companies as evidence of medical competence, but they carry no clinical-specialty metadata, so a strong overall score cannot be broken down into psychiatry or clinical psychology performance. At the same time, most existing mental-health evaluations are bespoke academic benchmarks focused on suicide-risk detection and diagnostic classification, which are poor proxies for real open-ended help-seeking and are hard to plug into developer workflows.

The goal is to produce a mental-health slice of HealthBench that is transparently constructed, clinician-validated, and reusable at essentially no marginal cost — so that any lab already running full HealthBench can report specialty-level numbers by filtering results it already has.

Key Contributions

1. A transparent, reproducible pipeline for carving expert-validated specialty subsets out of unlabeled health benchmarks, including the first empirical characterization of HealthBench's mental-health content.
2. The openly released HealthBench-Psych subset (610 conversations, 12.2% of the corpus) and HealthBench-Psych-Hard (119 conversations), released as content-hashed lists of HealthBench conversation identifiers.
3. A specialty-resolved leaderboard over 20 frontier and open models evaluated by a cross-vendor panel of three LLM judges, revealing a statistically tied frontier cluster and surfacing model refusal behavior as a measurable, safety-relevant property.
4. Evidence on judge reliability at specialty scale, including cross-judge rank agreement, severity offsets, and self-preference behavior after severity correction.

Main Findings

• Subset size and composition: The screen labeled the 5,000-conversation corpus relevant 378 (7.6%), borderline 263 (5.3%), and not_relevant 4,359 (87.2%). The released subset is 610 conversations (12.2%). By modal clinician category, perinatal mental health is largest (112; 18.4%), followed by anxiety (84), psychiatric medication (69), mood disorders (57), sleep (43), cognitive–neurocognitive (41), ADHD (36), substance use (31), and suicidality/self-harm (19), with seven further categories at n ≤ 18 and 65 conversations lacking a category majority.

• Two-round construction loop terminated by rule: Round 1 covered 791 conversations (641 screened-in plus 150 concealed controls) with relevance AC1 of 0.79; the majority included 587 screened-in conversations and 9 of 150 controls, a 6.0% control-inclusion rate that exceeded the 5% decision threshold and triggered a recall round. The recall rubric re-examined 4,209 excluded conversations not used as controls, surfacing 81 candidates (9 clear, 72 possible) reviewed with 25 fresh concealed controls; the majority included 14 candidates (6/9 clear, 8/72 possible) and 1 of 25 controls, a 4.0% rate below the threshold, terminating the loop. Round-2 agreement was lower (AC1 0.49; remaining pair 0.84).

• Harness validated against the reference: Replicating OpenAI's GPT-4.1 evaluation on the complete 1,000-conversation HealthBench-Hard subset using OpenAI's judging temperature of 0.5 and 2,048-token generation cap, this pipeline scores 0.157 against the published 0.16 (Δ = −0.003), with 0 failed gradings and 1,000/1,000 conversations scored.

• Statistically tied frontier cluster: The top five — kimi-k2.6 (0.627), gpt-5.5 (0.624), claude-opus-5 (0.620), grok-4.5 (0.612), and gpt-5.6-sol (0.610) — form a statistically tied cluster. Across all ten paired comparisons, only kimi-k2.6 and gpt-5.5 separate from gpt-5.6-sol at uncorrected 95%, and neither separation survives Holm–Bonferroni correction. kimi-k2.6 holds the highest point estimate under each judge, including the reference grader.

• Score spread below the frontier: Scores step down through a mid-tier (0.51–0.59) to gemini-2.5-flash, qwen3-8b, and claude-haiku-4.5 (0.44–0.46), mistral-small (0.36), and gpt-3.5-turbo (0.176). On the hard subset (n = 119), claude-opus-5 leads at 0.415 and gpt-3.5-turbo scores 0.000.

• Stability of the ranking: The ranking on the 596-conversation round-1 subset is identical to the final n = 610 ranking.

• Judges agree on order but differ on severity: The three judges ranked the 20 candidates near-identically (Kendall τ = 0.926–0.947) but differed in severity: against a grand mean of 0.524, gemini-2.5-flash graded +0.077 leniently while claude-haiku-4.5 and gpt-4.1 graded −0.045 and −0.032 strictly. Raw self-preference was large and inconsistent in sign; after severity correction it collapsed to within noise of zero for all three judges.

• Measurable refusal behavior in two models: claude-opus-5 returned empty refusals on 10/610 conversations (1.6%) and claude-fable-5 on 3/610 (0.5%); no other model refused. The two models refused disjoint conversation sets: claude-opus-5's refusals concentrated on psychiatric-medication questions and clinician-voiced requests about patient care, while all three claude-fable-5 refusals concerned neurodegeneration-related content (Alzheimer's biomarkers, mechanisms, and prevalence). Excluding these conversations raises claude-opus-5 to 0.631 and claude-fable-5 to 0.594, both within the primary estimate's confidence interval and leaving the frontier cluster's composition unchanged.

• Newest frontier releases score below their immediate predecessors in three lineages: claude-fable-5 0.591 vs. claude-opus-5 0.620 (paired Δ = −0.029 [−0.045, −0.013]), kimi-k3 0.568 vs. kimi-k2.6 0.627 (−0.059 [−0.077, −0.041]), and gpt-5.6-sol 0.610 vs. gpt-5.5 0.624 (−0.014 [−0.026, −0.002]). The first two gaps are described as decisive and the third as marginal, but the direction is uniform across three lineages that post gains on general benchmarks.

• Alignment with published full-HealthBench scores: gpt-4.1 scores 0.512 (0.483 with the identical GPT-4.1 judge) here against a published 0.48, gemini-2.5-pro 0.527 (0.499) against 0.52, and gpt-3.5-turbo floors at 0.176 (0.125) against 0.16, while kimi-k2.6's lead of 0.627 (0.600) matches the 0.58 Moonshot reports for its reasoning predecessor.

Methodology in Plain English

The researchers started from the public HealthBench open-source release and, since it has no specialty labels, wrote a screening rubric that classifies each conversation from the user's turns alone into relevant, borderline, or not_relevant. The rubric was drafted with LLM assistance and finalized by the authors. Ambiguous cases route to human review, and ties resolve to the less-inclusive label, making the screen deliberately conservative.

They applied the rubric with an LLM (Claude Opus 4.8) in independent batches of 50 conversations, each producing a constrained record with a label, an 18-term category, a confidence, and a short rationale. Three licensed clinicians (R1–R3; 1 MD, 1 LICSW, 1 LPC) then reviewed a blinded set containing all relevant and borderline conversations plus 150 not_relevant conversations interleaved as concealed controls, in one seeded random order with screen labels withheld. Agreement was measured with Gwet's AC1 rather than kappa, because the screened-in set's high prevalence deflates kappa. Inclusion required a reviewer majority (≥2/3), with per-item votes and a consensus tier retained in the release.

Because the controls came from the screen's excluded pool, the rate at which reviewers included them estimates the screen's miss rate. A control-inclusion rate above 5% triggered a recall round, which used the included controls and reviewer notes to codify the miss types into a recovery rubric, run with Claude Sonnet 4.6 over the remaining excluded conversations. Recovery candidates entered a fresh blinded round with the same clinicians, same majority rule, fresh concealed controls, and independent randomization.

For evaluation, each of 20 candidate models from eight providers received each HealthBench conversation verbatim, with no added system message. Responses were generated at temperature 0 with an 8,192-token cap (4,096 for gpt-3.5-turbo), with provider defaults recorded for models whose APIs rejected or fixed the temperature. Grading reused the HealthBench grader verbatim — same template, system message, and 2,048-token cap — with one change: judges graded at temperature 0 instead of the default 0.5. Every conversation–candidate pair was graded independently by GPT-4.1, Claude Haiku 4.5, and Gemini 2.5 Flash, drawn from three vendors so judge-family effects were measurable. Scores follow HealthBench's convention, and uncertainty was estimated by conversation-level bootstrap with 1,000 resamples.

Why This Matters

Impact on research. The paper shows a reusable way to convert a widely used general health benchmark into a specialty-resolved instrument, and demonstrates that frontier mental-health conversational quality is not automatically improving alongside general benchmark gains. It also treats refusal behavior as a measurable, safety-relevant property rather than an incidental artifact, and provides evidence that judge reliability holds at specialty scale even where judge severity differs.

Real-world applications:

• Filtering existing per-conversation HealthBench results against the released identifier lists to report mental-health scores retroactively, with no additional generation or grading.
• Populating model cards and technical reports with specialty-resolved numbers where mental-health competence is currently implied by a single aggregate score.
• Re-grading a released leaderboard under new judges, or extending it to new models, without regenerating any candidate responses.
• Transferring the same construction loop to other clinical specialties carved out of HealthBench and beyond.

Industry relevance. HealthBench has begun to appear in AI company model cards and technical reports as the core of health testing, making it one of developers' most widely used mental-health evaluation instruments. A lab reporting a strong HealthBench number may therefore appear competent in psychiatry and clinical psychology with no clear way to isolate that domain performance. The authors state that the specialty-resolved numbers currently missing from model cards are available from runs developers already perform. They also state that benchmark scores measure rubric adherence on fixed conversations and should not be read as evidence that any model is safe or effective for mental-health support, crisis response, or clinical use.

Future Directions

• Extend the pipeline beyond mental health: The authors state the pipeline transfers to other specialties within HealthBench and beyond, making specialty resolution a general capability rather than a one-off.
• Re-grade and extend without regeneration: Because responses, grades, and analysis code are released, the leaderboard can be re-graded under new judges and extended to new models over the same stored outputs.
• Investigate the frontier regression pattern: The uniform direction of newer models scoring below their immediate predecessors across three leading lineages raises the question of what in post-training or deployment shifts degrades mental-health conversational quality.
• Strengthen construction and evaluation limits: The authors flag that the LLM screen ran inside an agentic harness with a proprietary system-prompt layer and is treated strictly as a pre-filter; that concealed controls estimate roughly 6% residual mental-health content in the excluded pool by liberal clinician standards, so some conversations may remain unrecovered; that reviewers were English-speaking and rated 123 non-English conversations using machine translations as reference; that candidates were sampled once at deployed defaults rather than at matched inference compute; that judge-reliability evidence derives from three judges in one specialty and should be treated as suggestive; and that the hard-subset comparison (n = 119) has wide intervals.

Target Audience

This paper is most useful to AI developers and evaluation engineers who run HealthBench and need specialty-level reporting; to clinical and health-AI researchers studying how LLMs behave in mental-health conversations; to clinicians and informaticists involved in benchmark validation and rubric design; and to policy and model-card reviewers who assess claims of medical competence. It is written for readers comfortable with standard LLM evaluation methodology — rubric grading, judge panels, bootstrap intervals, and rank-agreement statistics — rather than for a general audience.

Authors’ abstract

General-purpose health benchmarks increasingly anchor claims about LLM medical performance, but they are not always resolved by clinical specialty, making domain-specific performance hard to isolate. Mental health is of acute public-health concern as millions of people turn to LLMs for psychological support, and most existing evaluations are bespoke academic benchmarks that are difficult to integrate into developer workflows. We introduce HealthBench-Psych and HealthBench-Psych-Hard. We screened HealthBench's 5,000 physician-rubric conversations for mental-health relevance with a transparent LLM-applied rubric, then validated the subset through two rounds of blinded clinician review with concealed known-exclude controls, yielding 610 conversations (12.2% of the corpus). Evaluating 20 frontier and open models under a cross-vendor panel of three LLM judges, we find a statistically tied frontier cluster, measurable refusal behavior in two models, and near-identical rankings across judges ($τ\ge 0.92$). We release the subset, pipeline, model responses, grades, and analysis code as a reusable resource.