Tell Me: An LLM-powered Mental Well-being Assistant with RAG, Synthetic Dialogue Generation, and Agentic Planning

AI summary

Overview

• Research area: Natural Language Processing, specifically LLM-based conversational AI for mental well-being support.
• Technical level: Intermediate. The paper assumes familiarity with LLM concepts (RAG, embeddings, agentic orchestration) but explains each component clearly enough for a technically literate non-specialist.
• Scope: The paper introduces and evaluates Tell Me, an open-source prototype that unifies a retrieval-augmented conversational assistant, a synthetic client–therapist dialogue generator, and an agentic self-care planner into a single mental well-being system.

What This Paper Is About

Existing LLM-based mental health tools tend to produce generic, context-poor responses, rely on confidential therapeutic data that is hard to access, and remain static rather than adapting to a user's needs over time. The authors build Tell Me to address all three gaps at once: a knowledge-grounded reflective chatbot, a profile-conditioned synthetic dialogue generator for safe data augmentation, and a multi-agent planner that turns conversations into weekly self-care routines and guided meditations. The stated goal is a reflective support space that complements (not replaces) professional therapy.

Key Contributions

1. A context-sensitive RAG assistant for reflective, knowledge-grounded dialogue. It retrieves therapist-style responses from curated counseling datasets using a lightweight embedding model and is benchmarked across nine LLMs with an LLM-as-a-judge framework, then validated via a blind RAG-vs-non-RAG human study.
2. A synthetic client–therapist dialogue generator conditioned on user-defined client profiles (demographics, concerns, history), enabling data augmentation and research without exposing real patient records.
3. An agentic well-being planner built with CrewAI that coordinates multiple agents via GPT-4o to analyze a transcript, produce a structured seven-day routine, and synthesize a guided meditation audio file through Text-to-Speech integration.
4. An open, unified demo (Hugging Face Spaces and GitHub) that lets end-users switch between cloud and local LLMs, and lets researchers generate safe synthetic transcripts within the same interface.

Main Findings

• Retrieval reliably surfaces relevant context: On 30 sampled counseling interactions evaluated with synthetic queries, the pipeline achieved a Hit Rate@5 of 1.0000 and MRR@5 of 0.9222 — though the authors caution that the perfect hit rate is likely inflated by the small search space (N=30) and synthetic query construction.
• Top LLMs differ by evaluator emphasis: Under GPT-5 judging, Claude 3.7 Sonnet scored highest (9.5), followed by GPT-4o (8.8) and LLaMA-3 (8.6). Under GPT-4o judging, GPT-4o ranked itself first (8.9), with Claude 3.7 Sonnet and Gemma-3 tied at 8.7. GPT-5 favored relational depth and empathy; GPT-4o favored practical, actionable scaffolding.
• High performers share three traits: Empathy validation (acknowledging distress without judgment), invitational dialogue (encouraging reflection over prescription), and cultural sensitivity. Low performers (phi-4, mental_llama2) pathologized users, suggested unsafe disclosures, or produced verbose checklist-style replies.
• RAG improves clarity and overall preference: In a within-subject study of 10 participants rating 50 blinded prompt pairs, the RAG assistant scored higher on Clarity (4.20 vs. 3.50), Helpfulness (4.00 vs. 3.90), Supportiveness (3.90 vs. 3.80), and Overall (3.80 vs. 3.60). The non-RAG baseline was marginally higher on Groundedness (4.00 vs. 3.80).
• Trade-off between speed and engagement: Participants noted the RAG assistant was slower and occasionally blunt, while the non-RAG system was faster but less engaging and sometimes unclear.
• Synthetic dialogues show promise and a "fix-it" bias: The generator produced empathic, framework-informed exchanges, but prematurely pivoted to problem-solving when clients disclosed deep distress, and overused repetitive phrasing such as "It sounds like."
• The planner produces longitudinal structure: The CrewAI pipeline translated unstructured dialogue into a temporal plan (morning/midday/evening blocks), preserved client-specific coping mechanisms (e.g., hiking, non-academic socializing), and generated a downloadable meditation audio artifact.

Methodology in Plain English

The system is organized into three modules behind a Streamlit front end.

The RAG assistant converts counseling conversations from two open datasets (CounselChat and Mental Health Counseling Conversations) into a context–response schema. Documents are split into 128-token chunks with 20-token overlap, embedded using the lightweight BAAI/bge-small-en-v1.5 model, and stored in a LlamaIndex vector store. At inference, the top-3 most similar documents are retrieved and passed, along with a 3,000-token chat memory buffer, to the LLM to keep responses grounded and compassionate. A pre-processing guardrail (a LangChain sentiment chain) screens for self-harm signals and triggers deterministic crisis routing instead of free-form generation.

The synthetic dialogue generator runs two separate LangChain LLM chains — one playing the client (seeded with a persona) and one playing the therapist (instructed to use reflective listening) — passing conversation history back and forth in a closed loop to produce a downloadable transcript.

The planner uses CrewAI to coordinate three GPT-4o agents (temperature 0.7) with strict sequential dependencies: an analyzer extracts emotions and triggers from the transcript, a research/plan stage maps those to evidence-based exercises, and a planner compiles a seven-day routine and scripts a meditation. The meditation text is piped asynchronously through a TTS wrapper (Edge Neural TTS, ElevenLabs, or Coqui) to produce an MP3.

Evaluation combined a retrieval-only sanity check (Hit Rate@5 and MRR@5 on synthetic queries), an automated LLM-as-a-judge benchmark (180 scored outputs: 9 models × 10 prompts × 2 judges), and a small blind human study comparing RAG vs. non-RAG outputs on five dimensions. The synthetic generator and planner were presented as qualitative case studies rather than quantitatively evaluated.

Why This Matters

The work pushes research on responsible LLM deployment in emotionally sensitive domains. It offers an open, reproducible testbed where the RAG component, synthetic dialogue pipeline, and agentic planner can be studied independently — a useful complement to closed commercial systems like Woebot, Wysa, and Replika that are not available for research.

Real-world applications:

• Accessible reflective support: A low-barrier daily self-reflection tool for users who are not currently in therapy, delivered through a public web demo with crisis safeguards.
• Research/testbed use: A safe environment for NLP researchers to study therapeutic language, benchmark models, and generate synthetic cohorts without breaching patient confidentiality.
• Training and pedagogy: A platform for mental health communication training, where students or practitioners rehearse client interactions against profile-conditioned simulations.
• Multimodal digital health: The planner's TTS-driven meditation generation illustrates how conversational agents can extend into actionable, audio-based interventions.

Industry relevance:

The paper speaks directly to companies building digital health, employee well-being, and conversational AI products. Its architecture — lightweight embeddings, ephemeral API-key handling, modular open components — is a practical template for cost-conscious, privacy-respecting deployment. The finding that retrieval grounding improves clarity and perceived helpfulness, and that GPT-5 and GPT-4o judges reward different qualities, is also relevant to teams designing evaluation pipelines for sensitive-domain LLMs.

Future Directions

• Broader empirical validation: Expand to larger participant pools and diverse demographics; move beyond the current 10 prompts, 9 models, and 10-person study.
• Clinical alignment: Integrate domain-specific evaluation frameworks tied to therapeutic modalities such as CBT, and have licensed clinicians review outputs.
• Dynamic personalization: Enable the planner to adapt weekly plans based on longitudinal interaction data rather than one-off transcripts.
• Bias and fidelity auditing: Investigate cultural and demographic biases embedded in the underlying LLMs and datasets, and improve the synthetic generator's pacing and linguistic diversity to counter its "fix-it" bias and repetitive phrasing.
• Larger-scale expert studies: Transition the RAG assistant into user studies involving domain experts to validate real-world utility.

Target Audience

NLP and AI researchers working on conversational agents, retrieval-augmented generation, or agentic workflows; digital mental health product teams and clinical informatics practitioners evaluating AI-assisted support tools; responsible AI and HCI researchers interested in safety guardrails and evaluation frameworks for sensitive domains; and mental health professionals curious about what LLM-based tools can and cannot currently deliver. The paper is also accessible to graduate students seeking a clear, open-source reference architecture that spans RAG, synthetic data generation, and multi-agent orchestration.

Authors’ abstract

We present Tell Me, a mental well-being system that leverages advances in large language models to provide accessible, context-aware support for users and researchers. The system integrates three components: (i) a retrieval-augmented generation (RAG) assistant for personalized, knowledge-grounded dialogue; (ii) a synthetic client-therapist dialogue generator conditioned on client profiles to facilitate research on therapeutic language and data augmentation; and (iii) a Well-being AI crew, implemented with CrewAI, that produces weekly self-care plans and guided meditation audio. The system is designed as a reflective space for emotional processing rather than a substitute for professional therapy. It illustrates how conversational assistants can lower barriers to support, complement existing care, and broaden access to mental health resources. To address the shortage of confidential therapeutic data, we introduce synthetic client-therapist dialogue generation conditioned on client profiles. Finally, the planner demonstrates an innovative agentic workflow for dynamically adaptive, personalized self-care, bridging the limitations of static well-being tools. We describe the architecture, demonstrate its functionalities, and report evaluation of the RAG assistant in curated well-being scenarios using both automatic LLM-based judgments and a human-user study. This work highlights opportunities for interdisciplinary collaboration between NLP researchers and mental health professionals to advance responsible innovation in human-AI interaction for well-being.