SynthAgent: A Multi-Agent LLM Framework for Realistic Patient Simulation -- A Case Study in Obesity with Mental Health Comorbidities

AI summary

Overview

Research area: Clinical AI / synthetic patient generation, combining multi-agent large language model (LLM) systems, behavioral and psychological modeling, and obesity medicine with psychiatric comorbidity.

Technical level: Advanced. The paper assumes familiarity with multi-agent architectures, LLM-as-a-Judge evaluation, embedding-based diversity analysis, and clinical data types (claims, surveys, case reports).

Scope (one sentence): The paper introduces SynthAgent, a five-agent LLM pipeline that fuses medical claims, NHANES surveys, epidemiological priors, and PubMed case reports into synthetic obesity patients with mental health comorbidities, and benchmarks four LLMs as the core simulation engine over 120 generated patients.

What This Paper Is About

Real-world data on obesity patients who also have conditions such as depression, anxiety, social phobia, or binge eating disorder is fragmented, sparse, and privacy-restricted, which makes it hard to study how metabolic and psychological factors interact. The authors build a multi-agent system of specialized LLM agents that together construct complete virtual patient records — demographics, symptoms, labs, treatments, personality traits, and a multi-year disease timeline — grounded in empirical data sources rather than invented from scratch. The goal is a scalable, privacy-preserving way to explore patient journeys and treatment adherence, and to determine which LLM works best as the simulation engine.

Key Contributions

1. A five-stage multi-agent framework. SynthAgent coordinates a summarizer, generator, augmenter, evaluator, and refiner agent, each handling a distinct stage: condensing empirical data into a blueprint, drafting the patient, enriching it with literature evidence, auditing for plausibility, and repairing detected issues.

2. Layered multi-source data integration. The framework combines 12 NHANES survey cycles (1999–2023) spanning 21 health domains, 70,000 de-identified patients from the PurpleLab medical claims database (10-year lookback), epidemiological probabilities from BRFSS, the World Obesity Federation, and the National Comorbidity Survey Replication, plus case reports from PubMed.

3. Personality-grounded behavioral modeling. Each simulated patient receives personality dimensions across HEXACO, reinforcement sensitivity theory (RST), and the Temperament and Character Inventory (TCI), which modulate adherence, emotion regulation, and lifestyle behavior.

4. A controlled four-engine benchmark. The authors hold the summarizer agent (Gemini 2.0 Flash) constant and vary only the core engine across GPT-5, Gemini 2.5 Pro, Claude 4.5 Sonnet, and DeepSeek-R1, producing 30 standardized foundational profiles evaluated on 120 simulated patients.

Main Findings

• GPT-5 and Claude 4.5 Sonnet lead on overall quality. GPT-5 achieved the highest mean quality score (76.27 ± 2.84), followed closely by Claude 4.5 Sonnet (76.17 ± 3.90). Claude 4.5 Sonnet produced the single highest-rated patient (score = 82).

• The two top models are statistically equivalent. Claude vs GPT-5 yielded p = 0.91. Gemini 2.5 Pro (71.17 ± 2.96) and DeepSeek-R1 (67.83 ± 3.26) performed significantly lower, with p < 0.001 against both leaders, and Gemini vs DeepSeek at p = 0.0001.

• The models have complementary strengths. GPT-5 scored highest on structure and time: Medical History (100.0), Treatments (80.0), and Timeline (82.3). Claude 4.5 Sonnet led on Symptoms (79.1), Habits (61.7), Labs (61.1), and Psychological Scales (71.3). DeepSeek-R1 uniquely led on Current Conditions (86.2).

• Claude 4.5 Sonnet produces the most diverse cohort. Its embedding diversity score was div = 0.353, versus div = 0.275 for both DeepSeek-R1 and GPT-5. The t-SNE projection showed dense clustering for DeepSeek-R1 and Gemini 2.5 Pro (especially obesity-only patients) and broad dispersion for GPT-5 and, most notably, Claude 4.5 Sonnet.

• Habits is the weakest dimension across all engines. Scores ranged from 43.3 (DeepSeek-R1) to 61.7 (Claude 4.5 Sonnet).

• Case study demonstrates end-to-end plausibility. A simulated 51-year-old married male in North Carolina with Class I obesity (BMI 33), hypertension, hyperlipidemia, obstructive sleep apnea, GERD, and osteoarthritis scored 81, with labs including HbA1c 5.3%, fasting glucose 95 mg/dL, total cholesterol 185 mg/dL, LDL 105 mg/dL, HDL 44 mg/dL, and triglycerides 160 mg/dL.

Methodology in Plain English

The authors first standardize four data sources so they can be matched to one another: national survey records, insurance claims, published prevalence statistics, and published patient case reports.

Then five agents work in sequence. The summarizer rolls a demographic and comorbidity profile probabilistically, then matches it against 10 representative NHANES samples and 20 longitudinal claims records, picking the closest NHANES match and the top three matching clinical trajectories to build a blueprint — including a BMI class (I, II, or III) and the assigned comorbidity. The generator turns that blueprint into a full patient record. The augmenter extracts disease keywords from the draft, pulls 10 relevant PubMed case reports per keyword, filters them for age and gender match, and weaves the findings into symptoms, psychological scales, and the role-play profile. The evaluator audits demographics, medical logic (including temporal errors like a treatment appearing before its diagnosis), psychological coherence, and lifestyle realism, and rates each issue as major, moderate, or minor. The refiner then makes targeted edits rather than regenerating the record.

For evaluation, the summarizer agent was locked to Gemini 2.0 Flash across all four configurations so that only the core engine varied. GPT-4o served as an independent LLM-as-a-Judge, scoring all 120 profiles on 10 dimensions using a deductive scheme that starts at 100 and subtracts points for problems in Information Sufficiency, Logical Consistency, and Medical Plausibility. Comparisons used Student's t-tests with a significance threshold of p < 0.05. Separately, Jina Embeddings v4 converted records into 768-dimensional vectors (with demographic information excluded) to measure within-cohort diversity by mean Euclidean distance to centroid, between-cohort similarity by cosine similarity, and visualize the results in 2D via t-SNE.

Why This Matters

Research impact: The paper argues that multi-agent architectures can fill data gaps for conditions where real datasets under-represent severe psychiatric phenotypes, while preserving privacy and enabling stress testing under rare but clinically critical scenarios. It also contributes a reproducible comparison protocol for choosing an LLM backbone on clinical simulation tasks, showing that quality and cohort diversity are separate axes that can point to different models.

Real-world applications:

• Simulating treatment adherence and lapses driven by emotional or cognitive stressors that rarely appear in clinical datasets.
• Prototyping and validating digital therapeutics or precision behavioral interventions against a diverse virtual cohort.
• Generating training and test scenarios for clinical AI systems and for clinician education on obesity plus mental health comorbidity.
• Supporting hypothesis generation and cross-site validation when real multi-site data cannot be shared.

Industry relevance: The work is authored by researchers at Klick Health, a health marketing and commercialization company, which points to applications in patient-journey modeling and behavioral intervention design. Because GPT-5 (structural fidelity) and Claude 4.5 Sonnet (psychological realism and diversity) have complementary strengths, the authors suggest hybrid systems may be the practical path forward.

Future Directions

• Clinician-in-the-loop validation. The current evaluation measures internal quality and semantic diversity only; external clinical validation is explicitly named as missing.

• Less rule-based inter-agent reasoning. Conflict resolution between agents is currently partially rule-based, which the authors say limits emergent behavioral complexity.

• Reinforcement-driven agent adaptation and hybrid engines. Combining the structural strengths of GPT-5 with the diverse, psychologically rich output of Claude 4.5 Sonnet is raised as a promising design.

• Better handling of input bias and rare phenotypes. The framework inherits bias from claims data and population surveys, which may underrepresent rare psychiatric phenotypes, and relies on proxy variables like personality traits that cannot fully capture sociocultural nuance.

Target Audience

Researchers and practitioners in clinical AI and synthetic data generation; computational psychiatry and behavioral health researchers studying obesity with comorbid mental disorders; teams building multi-agent LLM pipelines who need guidance on model selection tradeoffs between quality scores and cohort diversity; and health technology or pharmaceutical groups exploring privacy-preserving simulation for patient-journey and adherence modeling. Readers without background in LLM agent design or clinical data structures will find the evaluation sections dense.

Authors’ abstract

Simulating high-fidelity patients offers a powerful avenue for studying complex diseases while addressing the challenges of fragmented, biased, and privacy-restricted real-world data. In this study, we introduce SynthAgent, a novel Multi-Agent System (MAS) framework designed to model obesity patients with comorbid mental disorders, including depression, anxiety, social phobia, and binge eating disorder. SynthAgent integrates clinical and medical evidence from claims data, population surveys, and patient-centered literature to construct personalized virtual patients enriched with personality traits that influence adherence, emotion regulation, and lifestyle behaviors. Through autonomous agent interactions, the system simulates disease progression, treatment response, and life management across diverse psychosocial contexts. Evaluation of more than 100 generated patients demonstrated that GPT-5 and Claude 4.5 Sonnet achieved the highest fidelity as the core engine in the proposed MAS framework, outperforming Gemini 2.5 Pro and DeepSeek-R1. SynthAgent thus provides a scalable and privacy-preserving framework for exploring patient journeys, behavioral dynamics, and decision-making processes in both medical and psychological domains.