Roleplaying with Structure: Synthetic Therapist-Client Conversation Generation from Questionnaires

AI summary

Overview

• Research area: Natural Language Processing for mental health; synthetic dialogue generation; LLM roleplaying and fine-tuning.
• Technical level: Intermediate (requires familiarity with LLM prompting, fine-tuning, benchmark evaluation, and human annotation study design).
• Scope: The paper introduces SQPsych, a pipeline that turns structured clinical questionnaires and client metadata into synthetic therapist-client conversations (SQPsychConv) using locally hosted open-weight models, then fine-tunes smaller models (SQPsychLLM) on that corpus and evaluates them with automatic benchmarks and trained psychotherapists.

What This Paper Is About

Privacy laws such as HIPAA, GDPR, and PIPL, plus ethical concerns about client confidentiality, make real therapist-client dialogues hard to collect, so prior synthetic counseling datasets (CACTUS, SMILE) relied on generic information and proprietary API models. The authors ask whether grounding generation in real structured client profiles and standardized psychological questionnaires, using only locally hosted open-weight models, produces more therapeutically valid synthetic conversations. They generate a corpus, fine-tune smaller models on it, and test whether those models actually roleplay therapists better, judged both by benchmarks and by licensed psychotherapists.

Key Contributions

1. SQPsych, a pipeline that converts structured client metadata and questionnaire results into natural language and then runs turn-based roleplaying between separate therapist and client personas.
2. SQPsychConv, synthetic conversational corpora generated by seven open models (the paper describes them as 23B–123B; Table 1 lists checkpoints from 27B to 123B), used to fine-tune seven smaller SQPsychLLM models built on Llama3-8B-Instruct.
3. A holistic evaluation combining automatic benchmarks (CounselingBench, CBT Bench, an LLM Panel), pairwise preference evaluation on CounselBench-Adv, and human expert evaluation by three psychologists.
4. An ablation study generating an additional corpus conditioned on client metadata only (no clinical questionnaires), plus release of both corpora, fine-tuned model weights, code, and a dummy fully synthetic dataset for others to run the pipeline without the restricted data.

Main Findings

• Experts prefer the fine-tuned models. In a blind pairwise preference evaluation, SQPsychLLM gemma was consistently preferred by experts over the synthetic-data baseline (CAMEL), the real-world-data baseline (Psych8K), and vanilla LLaMA-3-8B-Instruct. A notable tie rate against Psych8K suggests real-world training data yields a naturalistic style humans appreciate, while the fine-tuning approach delivers more preferred interventions.
• Benchmarks understate the gains. Standard benchmarks do not adequately capture the dataset's strengths, but expert judgment shows SQPsych makes LLMs significantly better at therapist roleplaying. LLM panels assigned near-ceiling scores while clinical experts gave more conservative, nuanced ratings.
• CounselingBench (1,612 scenario-based NCMHCE mock-exam questions). SQPsychLLM achieves a new state of the art among mental-health-specific models in the zero-shot setting, frequently exceeding baselines by more than 10 points. Vanilla Llama3-8B-Instruct scores higher, consistent with prior observations about general-purpose models. In zero-shot chain-of-thought, the models remain competitive, often trailing only CAMEL, which uses data from the much larger proprietary gpt-4o.
• CBT Bench. The models set a new state of the art in two of three categories (CBT-CD cognitive distortion classification, 146 examples; CBT-PC primary belief identification, 184 examples; CBT-FC fine-grained belief subtypes, 112 examples), with robust gains in distortion detection and primary belief identification. The lower CBT-FC scores are described as a deliberate trade-off, since the models prioritize conversational flow and therapeutic alliance over surfacing deep beliefs early.
• Human expert scores (two psychotherapists, single-blind). On therapist skills (maximum 18 points), SQPsychConv qwen2.5, gemma, and mistral exceeded 15 points, while SQPsychConv qwq scored lowest at 11.3, roughly a 4-point difference. An Intraclass Correlation Coefficient of 0.45 indicates moderate annotation reliability. Client response scores (maximum 8 points) were consistent, with only llama3.3 scoring above 7.
• Questionnaires matter for therapist modeling. Providing questionnaires improved therapist skill scores in six of seven models, with llama3.3 as the notable exception. Client response quality stayed comparatively stable in all conditions, fluctuating within about one standard deviation, suggesting questionnaire grounding mainly improves therapist-specific competencies.
• LLM panel ranking. command scored highest on therapist skill in SQPsychConv, followed by nemotron and gemma; mistral and llama3.3 showed wider score distributions; qwq and qwen-2.5 ranked lowest.
• LLM judges are imperfect proxies. Human expert ratings showed moderate positive correlation with gpt-4o and gemini-2.0-flash, while inter-model correlations among the LLMs were significantly higher, particularly between larger models.

Methodology in Plain English

The pipeline starts with structured data: general demographics (age, gender, education, employment), family background (genetic risk, parental education, number of children), clinical characteristics such as age of onset and episode duration, and questionnaire results from the Hamilton Depression Rating Scale (HAM-D), Hamilton Anxiety Rating Scale (HAM-A), and Beck Depression Inventory (BDI). Each variable is serialized as a JSON object mapping a key to its textual description, turning numbers into a narrative. The source corpus contains 2,090 anonymized clients split into a control group and an MDD group.

Generation uses a dual-agent setup rather than one model playing both sides. Separate models act as therapist and client, each with distinct reasoning and information access. The therapist agent is prompted with CBT techniques such as case conceptualization, paraphrasing, empathic validation, identifying core thoughts and beliefs, guided discovery, Socratic questioning, cognitive restructuring, strategy offering, encouragement, behavioral action planning, and eliciting session feedback. The client agent expresses emotions and hesitation grounded in the questionnaires. The therapist speaks first, both models see the full conversation history, and the dialogue ends when the therapist emits an [/END] token, with a minimum of 15 turns required before termination. Outputs are post-processed with regular expressions to strip artifacts such as "thinking" tokens.

Generation used vLLM to host seven open models locally on four A100 GPUs with 80GB VRAM each in BF16 precision, because the clinical data's access terms prohibit sending it to third-party services. A typical 15-turn conversation took about three minutes, and 2,090 conversations were generated per model in approximately 4.5 days. The resulting corpora total 64,238 to 101,694 utterances depending on the model, with 15.534 to 24.599 average turns and 26.291 to 51.790 tokens per utterance. For comparison, the paper reports CACTUS at 995,512 utterances, 15.263 average turns, and 27.051 tokens per utterance, and Psych8k at 16,374 utterances, 1 turn, and 54.685 tokens per utterance.

Llama3-8B-Instruct was fine-tuned on each corpus portion with the same hyperparameters as CAMEL (5 epochs, learning rate 2e-4). Baselines were MentaLLaMA (built on Llama-2-7b-chat-hf, optimized for detecting and explaining mental health signals in social network text), CAMEL (Llama3-8B-Instruct trained on CACTUS), and Psych8k fine-tuned under the same settings. Automatic evaluation used CounselingBench in zero-shot and zero-shot chain-of-thought, CBT Bench in zero-shot, and an LLM Panel of gemini-2.0-flash, Deepseek-v3, gpt-4o, and gpt-4o-mini with temperature 0.0 judging 35 dialogues (five per model, three MDD and two control cases, with no access to source questionnaires or metadata to prevent leakage). Preference evaluation used the 120 prompts of CounselBench-Adv with majority vote and draws when no majority was reached. Three therapists holding graduate degrees in psychology participated: two for quantitative scoring on the same criteria and Likert-2 scale (0 = No, 1 = Somewhat, 2 = Yes), and one for qualitative analysis; therapist criteria totaled 18 points across nine items and client criteria 8 points across four items.

Why This Matters

• Impact on research: The work shows that structured clinical grounding can be used for synthetic data generation without exporting sensitive data to proprietary APIs, and it exposes a gap between automatic mental health benchmarks and expert clinical judgment. It releases both the questionnaire-conditioned and metadata-only corpora so the community can study the effect of clinical grounding.
• Real-world applications:
  • Training and simulation environments for psychotherapy trainees practicing CBT techniques.
  • Data augmentation for mental health NLP tasks in privacy-constrained jurisdictions (Europe, US, China).
  • Research infrastructure for evaluating therapist-facing LLM behavior against expert rubrics.
  • Foundation for future multilingual and audio-modality therapy simulation, as proposed by the authors.
• Industry relevance: Organizations that cannot legally send clinical data to third-party LLM vendors can replicate the local, open-weight generation approach. The results also caution against relying on LLM-as-a-judge for high-stakes domains, given the observed positivity bias and only moderate correlation with expert ratings. The authors explicitly state the models are in a research phase and not ready for deployment in clinical or public-facing scenarios.

Future Directions

1. Hybrid multi-agent configurations to further refine therapist-client interactions beyond the current dual-agent setup.
2. Extension to audio modality, capturing non-verbal cues such as voice tone or pacing that the paper notes are central to real counseling and absent from the current text pipeline.
3. Multilingual expansion to support mental health research across diverse cultural contexts, addressing the current geographic homogeneity of data collected in Marburg and Münster, Germany.
4. Broader psychiatric coverage and robust multi-turn evaluation. The corpora currently focus on MDD and control groups because schizophrenia and bipolar samples are highly limited; the authors also call for a multi-turn evaluation framework and a larger human study with higher inter-rater agreement before any deployment consideration.

Target Audience

Researchers and practitioners in clinical NLP and mental health AI, clinical psychologists and psychotherapists involved in evaluating AI-generated therapy dialogue, LLM engineers building privacy-preserving synthetic data pipelines, and benchmark designers interested in the limits of automatic and LLM-as-a-judge evaluation in high-stakes domains.

Authors’ abstract

Large Language Models (LLMs) are promising tools for synthetic data generation in mental health. However, privacy policies and restrictions forced previous work to rely mainly on generic information. We present a comprehensive corpus of synthetic therapist-client conversations generated through LLMs. We construct our generation pipeline, SQPsych (Structured Questionnaire-based Psychotherapy), which uses real structured client profiles and psychological questionnaires without leaking any sensitive data. We fine-tune various open-weight LLMs on our generated corpus, SQPsychConv , and test them through both automatic benchmarks and human evaluation with trained psychotherapists. We find that standard benchmarks do not adequately capture the strengths of our dataset, but expert judgment shows that SQPsych makes LLMs significantly better at therapist roleplaying. Experts also consistently prefer therapy sessions generated by our models compared to other mental-health-oriented LLMs. We release our code, fine-tuned models SQPsychLLM, and corpora at https://ai-mh.github.io/SQPsych.html.