Stress-Testing Emotional Support Models: Moving from Homogeneous to Diverse Help Seekers

AI summary

Overview

• Research area: Natural Language Processing, specifically evaluation methodology for LLM-based emotional support (mental health) chatbots and help-seeker simulation.
• Technical level: Intermediate. The paper is readable without deep ML background, but understanding the contributions requires familiarity with supervised fine-tuning, LoRA adapters, and Mixture-of-Experts routing.
• Scope: The paper builds and validates a controllable, population-diverse "help-seeker simulator" trained on Reddit emotional support dialogues, then uses it to stress-test seven supporter (chatbot) models.

What This Paper Is About

Evaluating emotional support chatbots usually works by having a simulated "help-seeker" talk to the chatbot and then scoring the resulting dialogue. The authors argue that existing seeker simulators are both too homogeneous — they portray seekers as overly cooperative and articulate — and not controllable, so researchers cannot target a specific seeker population. The goal is a simulator whose behavior can be dialed to match nine psychological and linguistic features of real seekers, so that supporter models can be tested against diverse, harder, more realistic populations rather than a single "easy" seeker type.

Key Contributions

1. A nine-feature seeker profile taxonomy and a large-scale annotated dataset. The authors define nine psychological and linguistic features, annotate 11,066 Reddit emotional support dialogues with them using a hybrid LLM-tagging and rule-based pipeline, and split them into 8,868 training, 1,094 validation, and 1,104 test profiles.
2. A Mixture-of-Experts seeker simulator with explicit behavioral routing. A frozen SFT backbone (Llama-3-8B-Instruct, LoRA r=16, merged) is augmented with four low-rank expert adapters (r=4) per linear layer, selected by a dialogue-level routing network driven by a rule-based 14-dimensional feature vector, trained with a language modeling loss plus a Task-wise Decorrelation loss.
3. Validation across three dimensions — profile adherence, fidelity, and diversity. The simulator achieves the highest profile adherence Macro F1 (0.549) among all baselines, wins an expert-rated fidelity evaluation with an average win rate of 69.5%, and shows the broadest UMAP coverage of seeker behaviors among compared simulators.
4. A population-diverse evaluation of seven supporter models, plus a human-validated check on the automated judge. The framework reveals performance degradations that cooperative simulators hide, and a human evaluation confirms the drops are genuine rather than artifacts of automated scoring.

Main Findings

• Profile adherence favors the MoE model. Across all simulators tested, the proposed model scored the highest Macro F1 on profile adherence (0.549), ahead of SFT (0.515), Contrastive Learning (0.484), DeepSeek-V3.2 (0.431), GPT-5 (0.319), GPT-4.1-mini (0.301), Qwen-2.5-14B-Instruct (0.284), and Llama-3-8B-Instruct (0.259). It also had the second-lowest standard deviation (0.125) after Qwen-2.5-14B-Instruct (0.095).
• Linguistic features are easy to control; psychological features are hard. Verbosity and dialogue length are learned well by all training-based methods, whereas resistance and self-disclosure remain difficult: zero-shot models score around 0.2 Macro F1, standard training reaches only around 0.3, and only the MoE model exceeds 0.4.
• Experts specialize into interpretable behavioral roles without being assigned roles in advance. Expert 0 is Emotion-Oriented (emotional distress, emotion-processing coping, high resistance, upset/verbose style); Expert 1 is Collaborative & Open (high engagement, low resistance, high self-disclosure, with engagement at 1.0, resistance at 0.76, and self-disclosure level 4 at 0.53 in the analysis heatmap); Expert 2 is Pragmatic & General with weaker specialization; Expert 3 is Reclusive (avoidant or maladaptive strategies, low engagement, minimal disclosure).
• Expert-rated fidelity favors the simulator. Three clinical psychology graduate students compared dialogues along linguistic naturalness, role authenticity, and psychological plausibility over 90 unseen Reddit summaries with sessions fixed at 10 turns. The simulator achieved an average win rate of 69.5%, beating Eeyore (68.9/66.7/71.1), ESC-Judge (68.9/72.2/62.2), and ESC-Role (80.0/67.8/67.8).
• The simulator produces broader behavioral diversity. Using 300 held-out profiles per simulator with GPT-5-mini as the fixed supporter, UMAP projections of dialogue-level seeker embeddings (all-MiniLM-L6-v2) showed the largest coverage area for the MoE simulator. Non-fine-tuned simulators (ESC-Judge, ClientCAST) occupied the most constrained regions, while SFT-based simulators (ESC-Role, Eeyore) covered moderately broader areas.
• Supporter rankings and scores shift depending on the seeker population. Under cooperative simulators such as ESC-Judge and ESC-Role, supporter models scored uniformly high on emotional support skills with small gaps; under the proposed simulator, ES scores dropped sharply and rankings changed noticeably, indicating that narrow seeker populations can overestimate supporter robustness.
• Emotional support skills are more sensitive than general conversational skills. General fluency stayed relatively stable across simulators, while suggestions and informativeness showed larger drops and higher variance.
• Two recurring failure patterns appear in low-scoring dialogues. With resistant seekers, models fell back on repetitive apologies instead of substantive empathy or constructive suggestions; with low-engagement seekers, they persisted in monotonous probing instead of adapting strategies like validation or reflection.
• Human evaluation confirms the automated judge. Two clinical psychology graduate students rated 60 dialogues on the same 1–5 rubric. Score differences (Ours minus ESC-Judge) were negative for both human and automated evaluation across all five ES skills — Identification (-1.167 human, -0.733 LLM), Comforting (-0.900, -0.833), Suggestions (-1.583, -1.067), Experience Sharing (-0.417, -1.067), and Informativeness (-1.917, -1.467). Human-perceived gaps were larger in magnitude than LLM-judged gaps for every metric except Experience Sharing. Rank correlations were 0.454, 0.377, 0.744, 0.402, and 0.673 respectively, all significant at p < 0.05.
• Seeker populations differ between settings. Applying the same annotation pipeline to a real-world spoken counseling dialogue dataset and comparing with the Reddit-based training data showed population-level differences that can guide profile configuration.

Methodology in Plain English

The authors begin by collecting real emotional support conversations from large Reddit communities (subreddits with more than 500,000 members, such as r/offmychest and r/mentalhealth). Unlike prior work that filters heavily for clean, cooperative text, they deliberately keep informal and raw language, including wide variation in utterance length and aggressiveness, while masking all personally identifiable information and discarding low-quality threads by conversation length, topic, and upvote counts.

They then describe each seeker with a profile made of nine features: six psychological ones (main coping strategy, utterance style, resistance level, engagement level, self-disclosure level, seeker reaction proportions) and three linguistic ones (verbosity level, profanity flag, total dialogue turns level). Psychological features are tagged by an LLM and checked against human annotations on a 60-dialogue sample, where inter-annotator agreement averaged 0.57 and human–LLM alignment reached 0.84 accuracy; linguistic features are extracted by simple rules. Each profile is paired with a natural-language summary of the seeker's main problem, generated by GPT-4o-mini.

Training happens in two stages. First, Llama-3-8B-Instruct is fine-tuned with LoRA (r=16) on all linear layers to predict the next seeker utterance given a profile and dialogue history; the adapters are then merged and the model frozen. Second, four low-rank expert adapters (r=4) are attached to each linear layer in the attention and feed-forward blocks. A routing network — three residual MLP blocks plus a softmax gating layer — reads a rule-based feature vector (categorical features one-hot encoded, level-based features zero-centered and normalized) and outputs a single routing distribution that is computed once per dialogue and shared across all layers. The model is trained with next-token prediction loss plus a Task-wise Decorrelation contrastive loss that pushes samples with different feature labels toward distinct routing vectors in a 64-dimensional latent space.

For evaluation, the authors check profile adherence by generating dialogues with GPT-5-mini as the fixed supporter, re-extracting features from the generated seeker utterances, and comparing them to the input features with Macro F1. Fidelity is measured by expert pairwise comparison, and diversity by UMAP visualization plus lexical, semantic, and sentiment metrics. Finally, they run seven supporter models against 300 held-out test profiles each in multi-turn dialogues of up to 20 turns, scored by an expert-validated framework with ten metrics using GPT-4o-mini as the judge.

Why This Matters

Impact on research. The paper reframes evaluation of emotional support models as a population-coverage problem rather than a single-dialogue benchmark. It shows that a simulator biased toward cooperative seekers inflates scores and hides weaknesses, which means prior comparisons of supporter models may not transfer to real users. It also provides a reusable artifact — a feature taxonomy, an annotated corpus, and a controllable simulator — that makes targeted, diagnostic evaluation possible.

Real-world applications:

• Chatbot developers can configure seeker profiles to match their actual target population and detect which features cause failures before deployment.
• Clinical and mental health technology teams can stress-test systems against resistant or low-engagement users, the populations most likely to disengage from a chatbot.
• Researchers building new emotional support models can use the framework as a fairer benchmark that surfaces differences existing evaluations miss.
• Organizations evaluating third-party mental health chatbots can use the profile-based pipeline to compare vendors under consistent, documented conditions.

Industry relevance. The finding that rankings and scores change under different seeker populations directly affects procurement and model-selection decisions. The two identified failure modes — repetitive apology loops with resistant users and monotonous probing with disengaged users — are concrete, fixable product bugs that standard evaluations did not surface.

Future Directions

• Modeling change over time: the current framework treats seeker features as fixed throughout a dialogue, so dynamic or evolving psychological traits remain unaddressed.
• Measuring longer-term outcomes: the evaluation covers ten conversational metrics but not sustained emotional recovery, well-being, or behavioral change, which unfold across repeated interactions.
• Extending beyond the single Reddit-derived data source: the paper illustrates profile configuration using a spoken counseling dataset but does not train or fully validate the simulator on such populations.
• Improving the hardest features: resistance and self-disclosure remain far below linguistic features in adherence, and the routing analysis shows one expert (Pragmatic & General) has relatively weak specialization.

Target Audience

This paper is most useful for NLP and HCI researchers working on conversational agents for mental health, evaluation-methodology researchers interested in simulator-based benchmarking, and applied teams building or procuring emotional support chatbots. Clinicians and clinical psychology researchers collaborating on chatbot validation will also find the fidelity protocol and the human-versus-LLM judge comparison relevant.

Authors’ abstract

As emotional support chatbots have recently gained significant traction across both research and industry, a common evaluation strategy has emerged: use help-seeker simulators to interact with supporter chatbots. However, current simulators suffer from two critical limitations: (1) they fail to capture the behavioral diversity of real-world seekers, often portraying them as overly cooperative, and (2) they lack the controllability required to simulate specific seeker profiles. To address these challenges, we present a controllable seeker simulator driven by nine psychological and linguistic features that underpin seeker behavior. Using authentic Reddit conversations, we train our model via a Mixture-of-Experts (MoE) architecture, which effectively differentiates diverse seeker behaviors into specialized parameter subspaces, thereby enhancing fine-grained controllability. Our simulator achieves superior profile adherence and behavioral diversity compared to existing approaches. Furthermore, evaluating 7 prominent supporter models with our system uncovers previously obscured performance degradations. These findings underscore the utility of our framework in providing a more faithful and stress-tested evaluation for emotional support chatbots.