PatientAct: Theory-Grounded Mental Health Client Simulation

AI summary

Overview

Research area: Natural Language Processing, specifically LLM-based role-play and computational mental health (simulated therapy clients for counselor training, therapist evaluation, and synthetic data generation).

Technical level: Intermediate. The framework is conceptually accessible, but understanding it requires familiarity with LLM prompting pipelines and basic clinical concepts (case formulation, resistance, therapeutic alliance).

Scope: The paper presents PatientAct, a theory-grounded framework for simulating mental health clients whose profile structure, disclosure behavior, resistance, and trust dynamics are all grounded in established clinical theory, evaluated against three representative baselines on 40 clinical situations.

What This Paper Is About

LLM-based simulated clients are widely used to train novice counselors, test LLM therapists, and generate synthetic therapy data, but existing simulators produce overly cooperative clients: they disclose too readily, accept therapeutic reframes without resistance, and resolve core issues within a single session. The authors trace this to two design flaws: profiles that describe what a client thinks and feels without explaining why, and behavioral mechanisms that treat all profile content as equally accessible regardless of how much trust has been established. PatientAct addresses both by grounding profile construction in the 5Ps clinical case formulation and by gating disclosure through a dynamic memory layer with trust thresholds.

Key Contributions

1. A modality-agnostic profile schema that integrates clinical case formulation (the 5Ps framework) with a cognitive layer and an interpersonal relational layer, providing the causal depth missing from prior work and remaining applicable across therapeutic modalities such as CBT and MI.

2. A theory-grounded simulation framework that processes each therapist utterance through explicit stages: emotional reaction, behavior selection, and a dynamic retrieval pipeline. The client's evolving trust gates disclosure of profile content, and resistance spans multiple clinical dimensions (quantity, content, style) rather than a single label.

3. A trust-gated dynamic memory mechanism grounded in Mental-Trust, an annotation study of 212 real counseling sessions that identified seven expert-verified ordinal trust levels, mapped by the authors onto a numeric scale from 1.0 (least trust) to 4.0 (achieved trust) in steps of 0.5.

4. Comprehensive evaluation across expert human annotators and an LLM judge, demonstrating more realistic client simulations than three representative baselines, plus an ablation study isolating each component.

Main Findings

• Profile quality was rated high by expert annotators. On 5-point scales, Clinical Plausibility received 4.43 (± 0.59, α = 0.41), Internal Consistency 4.38 (± 0.58, α = 0.43), Case Specificity 4.32 (± 0.52, α = 0.43), and Clinical Depth 4.28 (± 0.55, α = 0.45). Diversity was rated 4.2 ± 0.42.

• Annotators recovered the intended psychological seeds. Attachment styles were predicted with 77.5% accuracy (κ = 0.61) and core belief themes with 64.2% accuracy (κ = 0.65). Core belief themes were sampled uniformly, giving a 33.3% chance level; under the clinical attachment prior, always predicting disorganized would yield 54%.

• PatientAct achieved the highest human ratings across all five simulation dimensions. Against the best baseline, the largest human-rated gains were in Resistance Quality (+0.67) and Behavioral Realism (+0.63), followed by Disclosure Pacing (+0.50), with smaller gains in Coherence (+0.39) and Emotional Authenticity (+0.32).

• Profile depth mattered more than dynamic mechanisms alone. Patient-ψ ranked second across most dimensions despite using only a static profile, outperforming both AnnaAgent and ConsistentMI, which employ dynamic mechanisms.

• The LLM judge produced a different ranking than humans. PatientAct was rated highest by the LLM judge on Emotional Authenticity and Realism (the dimensions with the highest human agreement), but Patient-ψ ranked highest on Coherence (4.40) and Disclosure Pacing (4.60), and ConsistentMI significantly outperformed baselines in LLM-rated resistance quality (3.83). The authors suggest the GPT-5.4 judge may favor outputs closer to its own generation style, and note the dimensions where PatientAct was outperformed had the lowest human agreement (α = 0.41–0.43) and weakest human–LLM correlation (ρ = 0.09–0.29).

• Ablation: every component contributed. Removing any of the three components reduced human ratings across all dimensions. The drop was largest for the removal of dynamic memory on four of five dimensions, particularly Resistance Quality (1.04 in human and 0.58 in LLM ratings, versus 0.94 and 0.30 for the removal of trust-gating). Trust-gating contributed most to Disclosure Pacing (0.93 vs. 0.75).

• Automatic evaluation alone favored an ablated system. The variant without the reaction-behavior-resistance pipeline was the only one the LLM judge rated above the full framework across all dimensions, while human annotators rated it lower on all five.

• Case study revealed distinct failure modes in baselines. Patient-ψ disclosed the core relational dynamic by turn 4 and began generating its own solutions by mid-session; AnnaAgent never pushed back and asked to pause from emotional exhaustion rather than resistance; ConsistentMI, despite being profiled in precontemplation, cooperated with every reflection. PatientAct deflected the first emotionally charged topic (turn 4), shared deeper feelings but immediately retreated (turn 7), and initiated its own therapeutic work only after the therapist repeatedly demonstrated respect for boundaries (turn 10), without resolving the pattern.

Methodology in Plain English

The authors build client profiles from three layers. Demographics ground the client's identity. A problem formulation follows the 5Ps framework: the presenting problem, precipitating factors (what triggered the current episode), predisposing factors (why this person became vulnerable), perpetuating factors (the cycles that keep the problem going), and protective factors (strengths and supports). A psychological formulation captures the client's in-session psychology: intermediate beliefs, automatic thoughts, triggers, coping patterns, emotional range, and interpersonal patterns described through Core Conflictual Relationship Themes.

Profiles are generated by a multi-step pipeline. A clinical situation, a demographic scaffold, and a psychological seed (a core belief theme — unlovable, worthless, or helpless — and an attachment style — anxious, avoidant, or disorganized) are passed to an LLM, which produces the problem formulation under instructions enforcing causal chains. A rule-based conflict checker flags incompatible combinations, the LLM revises, then generates the psychological formulation, and an LLM judge validates the whole profile for coherence against the original situation, feeding revisions back until validation passes.

Before simulation, each profile is split into two layers. A static layer (demographics, current concern, emotional range, the therapist-directed interpersonal pattern) always sits in the system prompt. A dynamic layer contains the remaining content converted into individual memory items, each with a disclosure level (minimum trust required), activation tags (when it becomes relevant), and a discomfort flag (whether approaching it too early should produce visible discomfort).

At each turn, the therapist's utterance is matched against activation tags. Items whose trust threshold is met can be disclosed; items with a discomfort flag that fail the gate are placed on a blocked list, creating pressure to deflect without revealing why. The client then determines an emotional reaction from seven reactions adapted from Hill's taxonomy, each with an intensity level (low, moderate, or high), and selects from eight behaviors adapted from Hill's categorization of client actions. If resistance is selected, one of seven resistance patterns is chosen across three dimensions: quantity, content, and style. After the exchange, trust is updated in steps of ± 0.25 (slight) or ± 0.5 (significant), bounded between 1.0 and 4.0 and starting at 2.5, with attachment style shaping the dynamics (anxious clients lose trust readily, avoidant clients build trust slowly and penalize pushiness, disorganized clients may lose trust even after positive exchanges).

Evaluation used 40 hand-crafted clinical situations (20 each for depression and anxiety), with GPT-5.4 as the backbone LLM in the profile pipeline. Ten psychology-background annotators rated profiles, with each profile seen by three annotators. For simulation, GPT-4o was the backbone across all methods with temperature 0.7, all clients talked to the same modality-agnostic therapist agent for 15 turns, producing 4 × 40 = 160 conversations. Human evaluation sampled 10 conversations per disorder per system (4 × 20 = 80 conversations), rated by three of the ten annotators per conversation (24 conversations per annotator), blind to method identity; an LLM judge (GPT-5.4) rated all 160 conversations.

Why This Matters

Impact on research. The paper argues that profile depth and quality yield larger improvements in simulation realism than dynamic behavioral mechanisms alone — a finding that redirects effort in patient-simulation research from state-tracking machinery toward richer, causally structured profiles. It also provides evidence that LLM judges are insufficient for evaluating therapy simulations on dimensions involving clinical judgment, since the judge favored an ablated variant that human annotators rated worse on every dimension, and human–LLM correlations were as low as ρ = 0.09.

Real-world applications.

• Training novice counselors in scalable practice environments where realistic resistance and paced disclosure actually occur.
• Standardized evaluation of LLM-based therapeutic systems, so evaluations are not inflated by compliant simulated clients.
• Generation of synthetic therapy dialogue for psychology research that includes the friction characteristic of real therapeutic interaction.
• Development of more realistic research infrastructure for computational mental health.

Industry relevance. Any organization building or benchmarking conversational mental health tools needs test clients that behave like real ones; PatientAct's trust-gated disclosure and multi-dimensional resistance offer a more credible evaluation substrate. The code and data are publicly available via github.com/Sahandfer/PatientHub, and the experiments were run on PatientHub, a unified framework for developing and benchmarking patient simulation methods.

Future Directions

• Extending beyond depression and anxiety. The authors evaluated only these two conditions because they are the most prevalent worldwide and the focus of existing baselines, and caution that conditions such as PTSD may involve qualitatively different therapeutic dynamics. They note the framework is disorder-agnostic by design and suitable for extension.

• Multi-session simulation. Experiments were limited to single 15-turn sessions, whereas real therapy unfolds over multiple sessions; the authors identify multi-session simulation, where trust carries over and the client's presentation evolves between sessions, as an important direction that the trust mechanism is positioned to support.

• Cross-linguistic and cross-cultural generalization. The study was English-only and relied on GPT-4o for all methods, and LLM-generated profiles may reflect training-data biases, including under-representation of non-Western presentations of mental disorders. Demographic scaffold sampling does not yet address deeper cultural differences in how distress is experienced and expressed.

• Linking realism to downstream outcomes. The paper evaluates realism through expert ratings but does not test whether more realistic simulations improve therapist training or LLM therapist evaluation, which the authors call an important direction their data and framework are designed to facilitate.

Target Audience

Researchers and practitioners working on LLM-based mental health simulation, computational mental health, and dialogue systems evaluation; clinical psychologists and counseling educators interested in scalable training and standardized testing tools; and NLP researchers studying role-play agents, persona design, and the reliability of LLM-as-judge evaluation, since the paper's clearest caution concerns the divergence between automated and expert assessment.

Authors’ abstract

LLM-based simulated clients are increasingly used to train novice counselors, evaluate LLM therapists, and generate synthetic data. However, current simulators produce overly cooperative clients that disclose too readily, accept therapeutic reframes without resistance, and resolve core issues within a single session. We trace these issues to profiles that lack causal depth and behavioral mechanisms that treat all content as equally accessible. We present PatientAct, a framework for client simulation grounded in established clinical theories. Our profiles integrate the 5Ps clinical case formulation, providing causal depth without tying the design to any single therapeutic modality. During simulation, profiles include a dynamic memory layer in which items carry trust thresholds (e.g., symptoms are available early, whereas formative memories require a sustained therapeutic alliance). At each turn, the client's emotional reaction and behavior are modeled before generating a response. If the therapist approaches gated content, PatientAct expresses resistance in terms of quantity, content, and style rather than defaulting to cooperation or a single resistance pattern. We evaluate our framework on 40 clinical situations and demonstrate that it generates diverse profiles with high clinical plausibility. Moreover, PatientAct significantly outperforms the baselines, yielding substantial gains in resistance quality and behavioral realism. Our code and data will be publicly available via github.com/Sahandfer/PatientHub.