PatientHub: A Unified Framework for Patient Simulation

AI summary

Overview

Research area: Natural Language Processing, specifically LLM-based mental health simulation and clinical training tools.

Technical level: Intermediate. The paper is readable without deep ML background, but it assumes familiarity with LLM prompting, fine-tuning terminology (SFT, DPO), and LLM-as-a-judge evaluation.

Scope: The paper introduces PatientHub, an open-source, modular framework that unifies 16 existing LLM-based patient simulators under shared abstractions for profile creation, multi-turn/multi-session simulation, and rubric-driven evaluation.

What This Paper Is About

Therapists-in-training and LLM-based mental health supporters both need realistic practice and testing partners, and LLM-simulated patients are a scalable, low-cost way to provide them. The problem is that existing patient simulators are built in isolation, using incompatible profile formats, prompts, orchestration code, and evaluation metrics, which makes it hard to reproduce results, compare methods fairly, or reuse components. PatientHub addresses this by putting many existing simulators behind one common interface and pipeline.

Key Contributions

1. Unification of 16 patient simulators across a shared abstraction of agents, events, generators, and evaluators, enabling cross-method and cross-model comparison that incompatible codebases previously prevented.
2. A configurable LLM-as-a-judge evaluator supporting four rubric paradigms (Binary, Scalar, Categorical, Extraction) applied at both the turn and session level, with separate judges for generated profiles and full conversations.
3. A graph-based orchestrator representing interactions as configurable events (turn-taking, termination criteria, session structure) and logging all turns to a unified JSON schema.
4. Demonstrated extensibility: prototyping a new simulator variant, Ψ-Doh, in only 72 lines of code on top of PATIENT-Ψ (45 for Pydantic schemas, 27 for trust-estimation and rewriting logic).

Main Findings

• Consistency scores look similar under a cooperative therapist. Under the CBT therapist condition, all methods scored highly on consistency dimensions, with factual consistency ranging from 3.72 (MindVoyager) to 3.94 (PATIENT-Ψ, CARS, Ψ-Doh), so the methods appear broadly comparable.
• An adversarial therapist separates otherwise similar methods. Under the "bad" therapist condition, MindVoyager's factual consistency dropped to 2.72, versus 3.86 for PATIENT-Ψ and Ψ-Doh.
• CARS disengages early. CARS, the only method that can end a session itself, averaged just 5.1 of the 15 allotted turns, and had the highest appropriate resistance scores under both therapists (3.70 under CBT, 3.74 under bad).
• Resistance rises under the bad therapist for every method, which the authors treat as a sanity check that the judge tracks interaction quality rather than surface form.
• Profile adaptation was largely faithful. Adapting PATIENT-Ψ CCDs into the Eeyore schema introduced no contradictions with source facts in 43 of 50 cases; the flagged discrepancies were minor re-framings or differences in inferred severity.
• Profile generation surfaced real gaps. Applying the extraction-based profile judge to CARS-generated profiles consistently flagged completeness gaps and coherence contradictions, such as Grace's coping strategy of "seeks social support and spends time with friends" conflicting with a background of isolation and a belief that she "must handle everything alone."
• Cost tracks architecture, not response length. The multi-agent and reasoning-augmented methods incurred the highest per-session API cost despite not producing the longest replies, while Eeyore ran locally at no API cost.
• Ψ-Doh behaved differently from its base. The prototype showed primarily higher appropriate resistance and feedback quality across both therapists, at shorter response lengths but higher per-session cost.
• The authors caution against over-reading the realism scores. They note prior work finds LLM judges can diverge from or invert expert ratings on fidelity dimensions, so they treat naturalness, emotional depth, and similar dimensions as illustrative rather than as evidence of clinical realism.

Methodology in Plain English

The researchers built a Python framework that breaks patient simulation into four reusable pieces: clients (the simulated patients), therapists (the counterpart agent), events (the rules for how a conversation unfolds, represented as graphs), and evaluators (LLM judges that score outputs). Each simulated patient is defined by a persona profile, a response-generation specification, and optionally an internal latent state that changes over the dialogue.

Existing simulators were re-implemented against these abstractions, reusing original prompts and profiles verbatim and only rewriting the surrounding orchestration code. A command-line interface exposes commands to generate profiles, adapt a persona to another method's schema, simulate interactions, evaluate results, and create new agents. Configuration is managed with Hydra, events are executed with Burr, LLM calls go through LiteLLM, outputs are validated with Pydantic schemas, and prompts are Jinja-rendered YAML templates over JSON character records.

For the demonstration evaluation, the team used 50 human-validated cognitive conceptualization diagrams (CCDs) from PATIENT-Ψ as the shared persona seed and ran four simulators (PATIENT-Ψ, CARS, MindVoyager, Eeyore) against two therapists (a CBT therapist and a deliberately bad, dismissive therapist). Prompted agents used GPT-4o as the backbone with Temperature = 0.7; Eeyore used the original authors' fine-tuned Llama-3.1-8B model and was instantiated via the adapt command. Each simulator was paired with each therapist using the Therapy Session event of 15 turns, with a moderator prompting closure at turn 13, and one session per persona (50 per therapist). To reduce self-preference bias, the conversation judge was an independent LLM, Sonnet-4.6, rather than the GPT-4o backbone.

Why This Matters

Impact on research. Patient simulation research is currently fragmented across papers and codebases with different profile formats, protocols, and metrics. By consolidating 16 simulators into one reproducible pipeline, PatientHub lowers the infrastructure overhead for benchmarking and makes it feasible to identify which design choices actually matter instead of re-building scaffolding per project.

Real-world applications:

• Training counselors and peer supporters through low-risk role-play, as a scalable alternative to costly standardized patients (trained actors).
• Benchmarking LLM-based supporters in a safe simulated environment, avoiding the risk of harmful advice reaching real users during evaluation.
• Generating synthetic therapy conversation datasets for training and analysis, where privacy and expert-cost constraints limit real data.
• Cross-method persona reuse, since the adapt command maps a persona into another method's schema, letting one persona drive otherwise-incompatible simulators.

Industry relevance. Teams building mental health chatbots or clinical training products need repeatable, configurable testbeds. PatientHub's standardized evaluator, cost and length tracking, and MIT-licensed installable package provide an off-the-shelf comparison harness, and its exposure of per-session API cost trade-offs is directly useful for choosing a simulator to run at scale.

Future Directions

• Expert validation of realism metrics. The paper states that LLM-judge fidelity scores can align weakly with or even invert expert ratings, and that validating the Likert realism dimensions against expert clinicians remains important future work.
• Automating faithful cross-schema adaptation. The authors report that LLM-judge fact-matching was too unreliable for a full preservation score, so they verified adapted profiles manually; automating that process remains an open problem.
• Broadening clinical coverage. The current demonstration covers 50 CCDs in a single domain (CBT counseling); the authors plan to expand coverage of simulation methods and clinical domains.
• Extending beyond single-session counseling. PatientHub does not yet target multi-party care teams, long-horizon follow-up, crisis intervention, or tool-augmented clinicians.

Target Audience

Researchers and practitioners working on LLM-based mental health agents, patient simulation, or clinical training tools; NLP researchers who need a reproducible harness for comparing simulated-patient methods; and developers building counseling training or therapy-support applications who need a configurable, open-source pipeline for profile generation, simulation, and evaluation.

Authors’ abstract

As Large Language Models increasingly power role-playing applications, simulating patients has become a valuable tool for training counselors and scaling therapeutic assessment. However, prior work remains fragmented: existing approaches rely on incompatible, non-standardized profiles, prompts, and evaluation metrics, hindering reproducibility, fair comparison, and reuse. We introduce PatientHub, a unified and modular framework that standardizes the creation, simulation, and evaluation of LLM-based patients. Our framework provides 16 patient simulators, a graph-based orchestrator for multi-turn, multi-session interactions, and a configurable LLM-as-a-judge evaluator that supports multiple rubric types. Via our command-line interface, users can generate patient profiles, run simulations, and apply rubric-driven evaluation at the turn and session level. To demonstrate PatientHub's utility, we compare several supported simulators under a shared interaction protocol and showcase its extensibility by prototyping a new simulator variant with minimal additional code. By consolidating existing work into a single reproducible pipeline, our framework eliminates much of the infrastructure overhead that currently fragments this research and accelerates the development of new methods. Our code and data are publicly available via https://github.com/Sahandfer/PatientHub.