EmoMed: An Emotionally-Aware Agent for Multimodal Medical Support with Real-Time Information Retrieval

AI summary

Overview

• Research area: Multimodal medical AI, affective computing, retrieval-augmented generation, and clinical safety evaluation.
• Technical level: Intermediate. The architecture is conceptually simple (prompting plus tool calls on top of a vision-language model), but the evaluation framework and medical-safety framing assume some familiarity with LLM agents, RAG, and medical QA benchmarks.
• Scope: The paper introduces EmoMed, an emotionally adaptive medical consultation agent, and demonstrates across seven vision-capable LLMs that adjusting response tone to a user's detected emotional state improves empathy and clarity without materially harming factual accuracy or safety.

What This Paper Is About

Medical chatbots powered by large language models are increasingly used for health questions, but they typically respond with static, emotion-blind language that can feel cold, overly alarming, or poorly matched to a worried user. EmoMed addresses this by first detecting the user's emotional state (for example, anxiety, confusion, or urgency) from text and optional medical images, then conditioning tone, structure, and level of detail on that signal. To prevent stylistic adaptation from degrading medical reliability, the agent simultaneously grounds its answers through two retrieval channels: live web search and an API-connected medical knowledge base.

Key Contributions

1. An emotion-conditioned medical agent architecture. A single tool-augmented vision-language model pipeline that performs emotion detection, tool selection, and response generation in one loop, with retrieval calls capped at five per query and an explicit content plan (acknowledge concern, restate, give clinical context, offer guidance, ask a clarifying follow-up).
2. Dual grounding for factual reliability. The agent can invoke Tavily web search for time-sensitive information and the MediSearch medical knowledge base for evidence-based clinical content, reducing hallucination and knowledge drift relative to a bare model.
3. A broad seven-model evaluation on text and image QA. Emotion-aware versus neutral-baseline prompts were compared across GPT-4o Mini, GPT-5 Nano, Qwen3 VL, Llama-4 Scout, Gemini 2.5 Flash, Claude 3 Haiku, and Grok-4 Fast using factual accuracy, LLM-as-judge dimensions, BERTScore, MedQA-style accuracy, and Med-VQA benchmarks.
4. A blinded human user study on perceived empathy and trust. Participants compared emotion-aware and neutral responses side by side, providing evidence that affective gains translate to real user perception rather than only to automated judge scores.

Main Findings

• Empathy and clarity improve consistently. Emotion-aware prompting raised average empathy scores across every model on both text QA (for example, Grok-4 Fast from 0.835 to 0.963; GPT-4o Mini from 0.628 to 0.919) and Med-VQA, with clarity gains of roughly +0.01 to +0.03 on text and larger gains on VQA.
• Factual accuracy is mostly preserved or improved. On text QA, four models matched or exceeded baseline accuracy, with notable gains for GPT-4o Mini (+0.139) and Grok-4 Fast (+0.087). The main exception was Gemini 2.5 Flash, which dropped 0.080 in accuracy while gaining 0.220 in empathy — a visible trade-off between affect and factuality.
• Safety and helpfulness stay stable or trend upward. Safety scores improved on average in both modalities, suggesting emotional warmth did not come at the cost of risk-appropriate language, and the agent's prompt explicitly forbids diagnostic claims and requires escalation recommendations for serious concerns.
• No detectable content drift. BERTScore and relevance metrics remained close to baseline, indicating that style changes did not substantively alter the underlying clinical content.
• VQA is harder and less stable. Emotional consistency sometimes declined in image-grounded tasks (for example, GPT-4o Mini from 0.818 to 0.732 and Llama-4 Scout from 0.813 to 0.709), which the authors attribute to noisier affect cues in image prompts. Accuracy differences were small and mixed.
• Human preference favors the emotion-aware version. In blinded pairwise comparisons, 34 of 44 participants (77%) rated the emotion-aware response as better, citing clearer structure for confusion, reassurance for anxiety, and calibrated hedging for urgent cases, with no perceived loss of accuracy or safety.
• Failure modes are limited and manageable. Rare over-hedging, occasional emotion misclassification on short or ambiguous inputs, and infrequent stale-retrieval misses are mitigated through a fixed response structure, detail caps, retries, transparent sourcing, and fallback to a neutral-warm tone.

Methodology in Plain English

The system wraps a single vision-language model in a decision loop. When a user submits text and an optional medical image, the model is first asked to name the user's dominant emotion in one word; if the output cannot be parsed as a valid emotion, the system defaults to neutral. That emotion label is then injected into a response-generation prompt that also specifies hard clinical rules: no diagnostic claims, explicit acknowledgment of uncertainty, and a recommendation to seek professional care when the situation warrants it.

Alongside generation, the model can call two retrieval tools — a general web search for time-sensitive health information and a specialized medical knowledge base for evidence-based content — with a hard limit of five tool calls per query to keep the pipeline efficient and prevent runaway reasoning.

To measure the effect of emotion conditioning, the authors ran every model twice: once with the full emotion-aware prompt and once with an identical prompt that strips out all emotion-related instructions. The two versions were compared on a text-only medical QA dataset (Medical Meadow and ChatDoctor-HealthcareMagic questions) and a multimodal pathology VQA dataset (PathVQA). Evaluation combined GPT-4o as an automated judge across empathy, clarity, consistency, helpfulness, safety, and relevance, plus binary factual accuracy and Sentence-BERT semantic similarity. A blinded pairwise user study served as a human check on the automated results.

Why This Matters

Research impact. The paper contributes a concrete demonstration that affective personalization and clinical factuality need not be in tension — a question that has been mostly discussed theoretically in medical AI safety literature. It also provides a reusable evaluation protocol (seven models, dual datasets, automated judges plus a human study) that other groups can adopt or extend.

Real-world applications.

• Patient-facing triage and pre-consultation chatbots that need to calm anxious users while still steering them toward appropriate care.
• Telemedicine intake tools that adapt explanation depth for confused or low-health-literacy users.
• Mental-health and chronic-care check-in assistants where perceived empathy strongly affects adherence and trust.
• Clinical decision support front-ends that must present urgent findings without inducing panic.

Industry relevance. Any organization deploying medical LLMs — health systems, telehealth platforms, insurers, and medical device vendors — faces the twin pressures of regulatory scrutiny over safety and user expectations of humane interaction. EmoMed offers a low-overhead design pattern: keep the underlying model, add emotion detection and dual retrieval as prompting and tooling layers, and measure both affect and accuracy rather than trading one for the other.

Future Directions

• Richer affect modeling. Replacing the single-label emotion classifier with a multi-dimensional or probabilistic representation of emotional state, and capturing how emotion shifts across a conversation rather than at a single turn.
• Stronger safety and provenance guarantees. Extending the current guardrails with formal privacy protections, explicit source attribution for every retrieved claim, and more systematic red-teaming for adversarial or poisoned inputs.
• Specialty-specific retrieval. Moving beyond a general medical knowledge base toward domain-tuned retrieval for dermatology, radiology, oncology, and other image-heavy specialties where VQA performance is currently weakest.
• Larger clinical validation. Running prospective studies with clinicians in the loop and larger, more diverse patient populations, since the present study relies on curated datasets and a relatively small user sample.

Target Audience

Researchers and graduate students working on medical NLP, multimodal agents, or affective computing will find the core contribution most relevant, particularly the evaluation design and the emotion-versus-accuracy trade-off analysis. Practitioners building healthcare chatbots or telehealth products will benefit from the architecture and the concrete prompting rules (no diagnoses, mandatory uncertainty acknowledgment, escalation language). Clinical informatics teams and AI safety reviewers evaluating medical LLM deployments should read the limitations and error-analysis sections closely, since those frame what the current evidence does and does not support for real-world use.

Authors’ abstract

We present EmoMed - a multimodal medical consultation agent that adapts its responses based on users' emotional states while maintaining clinical accuracy. The system processes text and medical images, detects affect indicators (anxiety, confusion, urgency) from user input, and adjusts response tone, structure, and detail level accordingly. To ensure factual reliability, the agent grounds clinical information through a dual retrieval mechanism: web-based fact-checking and an API-connected, continuously updated medical knowledge base. We evaluate our approach across seven state-of-the-art language models (GPT-4/5, Qwen3, Llama 4, Gemini 2.5, Grok4, Claude3) using comprehensive metrics including LLM-as-judge assessments, MedQA style accuracy tests, BERT Score, safety/helpfulness ratings, and multimodal medical benchmarks. The results demonstrate that emotionally adaptive responses consistently outperform neutral baseline across evaluation dimensions, without compromising clinical accuracy. A controlled user study validated these findings, with participants reporting improved perceived empathy and communication clarity, while maintaining trust in factual accuracy. Source code: https://github.com/NasonovIvan/EmoMed-Agent