PanicToCalm: A Proactive Counseling Agent for Panic Attacks

AI summary

Overview

• Research area: AI counseling and mental health NLP, specifically synthetic dialogue dataset construction, LLM fine-tuning and preference alignment for crisis intervention, and evaluation frameworks for therapeutic models.
• Technical level: Intermediate. The paper is readable without deep NLP background, but assumes familiarity with supervised fine-tuning, LoRA adapters, and Direct Preference Optimization (DPO).
• Scope: The paper introduces a Psychological First Aid (PFA)-grounded dataset of panic-attack counseling dialogues (Pace), a counseling model trained on it (Pacer), and a multi-dimensional evaluation framework (PanicEval), showing that Pacer is preferred over general, CBT-based, and GPT-4o-powered models in panic scenarios.

What This Paper Is About

Panic attacks are sudden episodes of intense fear with physical symptoms such as rapid heartbeat, shortness of breath, and dizziness, and timely grounding or breathing-based intervention can help people regain stability. Because authentic in-the-moment counseling sessions are nearly impossible to collect for ethical and logistical reasons, the authors build a synthetic dataset from first-person panic narratives and PFA guidelines, train a model to deliver both empathetic and directive support, and test whether it stabilizes distressed clients better than existing counseling baselines.

Key Contributions

1. Pace, described as the first counseling dataset designed for acute panic episodes and grounded in Psychological First Aid. It contains 1,226 verified panic profiles, expanded to 9,997 panic profiles via persona-based augmentation, and filtered into 8,263 dialogues and 106,433 turns.
2. Pacer, a PFA-based panic counseling model built on LLaMA 3-8B with two LoRA adapters (one for stage-specific planning, one for strategy and utterance generation), trained with supervised fine-tuning and then aligned with DPO using simulated client preference feedback.
3. PanicEval, a new evaluation framework that scores models on both general counseling skills (understanding, empathy) and first-aid-specific skills (clarity, directive support, stabilization, closure) on a 1–5 Likert scale, plus two auxiliary indicators: First Sign of Stabilization (turn level) and Intervention Turn Ratio.
4. Human validation with three clinical experts and three individuals with lived experience of panic or anxiety disorders, running head-to-head comparisons between Pacer and three baselines, and an expert CTRS review of 100 sampled dialogues.

Main Findings

• Pacer SFT+DPO leads on first-aid metrics: It scored 4.890 for clarity, 4.890 for directive support, 3.990 for stabilization, and 4.225 for closure, outperforming all baselines on first-aid-related metrics, including the closed-source GPT-4o-Incontext (4.880 clarity, 4.725 directive, 3.895 stabilization, 3.780 closure).
• General skills are competitive but not uniformly best: Pacer SFT+DPO scored 4.875 for understanding and 4.925 for empathy, versus GPT-4o-Incontext at 4.855 understanding and 4.960 empathy, so Pacer trails slightly on empathy while surpassing GPT-4o on understanding and clarity.
• Faster stabilization: Pacer SFT+DPO reached the first sign of stabilization at 7.17 turns on average, compared with 8.14 for Pacer SFT, 10.81 for GPT-4o-Incontext, 14.04 for Camel and Llama-Psych8k, 16.07 for Smilechat, and 16.71 for CBT-LLM.
• DPO improves over SFT: Pacer SFT+DPO gained +0.19 in empathy, +0.27 in stabilization, and +0.3 in closure over Pacer SFT, and stabilized users 0.97 turns earlier on average.
• Intervention ratios are high: Pacer SFT+DPO shows an intervention turn ratio of 88.24%, versus 53.16% for Smilechat, 52.28% for Camel, 47.16% for Llama-Psych8k, 43.70% for CBT-LLM, and 93.36% for GPT-4o-Incontext.
• Strategy patterns differ: GPT-4o relies heavily on breathing as a single technique (33.80% of its turns), while Pacer SFT+DPO more frequently uses physically grounded interventions and follows a staged PFA approach, guiding the client to a safe space before introducing techniques such as breathing.
• Client affect improves on PANAS: On overall PANAS scores, Pacer SFT+DPO produced a +1.20 positive-affect change and a -1.94 negative-affect change, compared with GPT-4o-Incontext (+1.18, -1.86), Camel (+0.98, -0.97), Llama-Psych8k (+1.07, -1.54), Smilechat (+1.08, -1.60), and CBT-LLM (+0.80, -1.21).
• Emotional changes are specific: Gains were most pronounced in self-related emotions such as "strong" and "proud," while "excited" and "alert" showed little to no improvement; on the negative side, "distressed" and "scared" decreased sharply while "hostile" remained largely unchanged.
• Results are not tied to GPT-4o: In a follow-up experiment using Gemini 2.0 as both client simulator and evaluator, Pacer SFT+DPO was still preferred over Pacer SFT, Camel, and GPT-4o.
• Human evaluators preferred Pacer: Against Smilechat, clinical experts chose Pacer in 80.7% of comparisons (1.0% tie, 18.3% lose) and lived-experience users in 86.7% (3.3% tie, 10.0% lose). Against Camel, experts preferred Pacer 74.3% of the time (0.3% tie, 25.3% lose) and lived-experience users 70.7% (12.3% tie, 17.0% lose). Against GPT-4o, experts preferred Pacer 66.0% of the time (3.7% tie, 30.3% lose), while lived-experience users preferred Pacer 41.7% of the time (16.0% tie, 42.3% lose).
• Expert CTRS review supported data quality: Three mental health professionals rated 100 randomly sampled training dialogues, with average scores of 4.04 for empathy, 3.88 for clarity, 3.75 for emotional appropriateness, 4.17 for directive support, and 4.05 for encouragement.
• Evaluator priorities diverge: Pearson correlations show clinical experts weight first-aid-specific skills more heavily, while lived-experience evaluators value empathy and understanding more equally, which the authors use to explain different preferences between the two groups.

Methodology in Plain English

The authors first collected first-person panic narratives from publicly available social media posts and used an LLM to extract a structured "panic episode profile" containing the Environment, Trigger Type, and Vicious Cycle (the interlinked physical, emotional, and cognitive responses that reinforce one another). After a two-stage personally identifiable information removal process (automated detection plus manual review), 1,226 profiles were retained. To diversify scenarios, they randomly paired a trigger type and a persona from an external persona dataset, then had an LLM generate a plausible environment, producing 9,997 panic profiles total.

They then restructured PFA into dialogue-feasible goals across three stages: LOOK (identify the panic profile and guide the client to a safe environment), LISTEN (stabilize through breathing, grounding, and reassurance), and LINK (encourage follow-up with a human counselor and end positively). Using GPT-4o-mini as the primary generation model, each stage was generated separately with a plan plus a script-style dialogue in a single inference pass, where each turn includes a strategy label of "keep" or "next" to control the transition to the next stage. The three stages were concatenated into a full session.

Filtering happened in two steps: dialogues with abnormal formatting or utterances over 100 words were removed (5.92% of the data), then GPT-4o rated five CTRS dimensions (Empathy, Clarity, Emotional Alignment, Directive Support, Encouragement) on a 1–5 scale and any dialogue scoring 3 or below on any dimension was removed (2.42%), leaving 8,263 dialogues and 106,433 turns.

Pacer was trained on LLaMA 3-8B with two LoRA adapters (rank 8, scaling factor 32) using 8,163 dialogue examples for SFT. For alignment, they instantiated a GPT-4o client simulator conditioned on a panic profile, had it interact with the SFT model, and generated 10 response candidates per turn via stochastic search. Preference pairs came from three tasks: strategy preference feedback (matching the simulator's keep/next decision), response preference feedback (ranking utterances by averaged empathy and directiveness scores on a 1–5 scale), and next client utterance generation. DPO was run on 19,103 preference pairs. Evaluation used 200 held-out profiles with GPT-4o as the client simulator, and PanicEval and PANAS scores were computed before and after each simulated session.

Why This Matters

Impact on research: The paper argues it is the only counseling dataset that explicitly models clients in active crisis, whereas prior synthetic counseling datasets like PsyCon, Psych8k, SmileChat, HealMe, CBT-LLM, and CACTUS target general mental health concerns and assume relatively stable, reflective clients. It also demonstrates a pipeline for combining theory-grounded data synthesis with simulation-based preference alignment, and releases code at the linked GitHub repository.

Real-world applications:

• Round-the-clock crisis support

Authors’ abstract

Panic attacks are acute episodes of fear and distress, in which timely, appropriate intervention can significantly help individuals regain stability. However, suitable datasets for training such models remain scarce due to ethical and logistical issues. To address this, we introduce PACE, which is a dataset that includes high-distress episodes constructed from first-person narratives, and structured around the principles of Psychological First Aid (PFA). Using this data, we train PACER, a counseling model designed to provide both empathetic and directive support, which is optimized through supervised learning and simulated preference alignment. To assess its effectiveness, we propose PanicEval, a multi-dimensional framework covering general counseling quality and crisis-specific strategies. Experimental results show that PACER outperforms strong baselines in both counselor-side metrics and client affect improvement. Human evaluations further confirm its practical value, with PACER consistently preferred over general, CBT-based, and GPT-4-powered models in panic scenarios (Code is available at https://github.com/JihyunLee1/PanicToCalm ).