Do LLMs Feel? Teaching Emotion Recognition with Prompts, Retrieval, and Curriculum Learning

AI summary

Overview

Research area: Affective computing and natural language processing — specifically Emotion Recognition in Conversation (ERC) using large language models.

Technical level: Intermediate. The paper assumes familiarity with LLM prompting, retrieval-augmented generation, parameter-efficient fine-tuning (LoRA), and standard ERC benchmark conventions, though the ideas themselves are explained clearly enough for readers with general machine learning background.

Scope: This paper introduces PRC-Emo, a training framework that combines emotion-aware prompt engineering, a purpose-built demonstration retrieval repository, and a difficulty-ordered curriculum learning schedule to improve LLM-based emotion recognition in multi-turn dialogue.

What This Paper Is About

Emotion Recognition in Conversation asks a model to label each utterance in a dialogue with the emotion behind it — but people often say one thing while feeling another. The paper targets this gap between explicit emotion (what is openly expressed) and implicit emotion (what the speaker actually feels underneath), which existing LLM-based systems tend to conflate or ignore. The authors ask whether LLMs can be taught to perceive both layers by supplying them with structured emotional interpretations, similar dialogue examples, and a training schedule that moves from easy conversations to hard ones.

Key Contributions

1. Explicit/implicit emotion prompting. The method generates two separate interpretations per utterance — one for the directly expressed emotion and one for the underlying, unspoken emotion — using an external LLM, and injects both, along with speaker characteristics, into the prompt as "external knowledge."

2. The first ERC-specific demonstration retrieval repository. A corpus of 36,712 utterances combining the training sets of IEMOCAP, MELD, and EmoryNLP with 14,009 newly generated dialogues. The new data was produced by GPT-4o across six real-world domains and seven scenarios, then filtered through a two-annotator human verification process repeated over three generation rounds.

3. A revised curriculum learning difficulty metric. The difficulty function extends prior work on weighted emotional shifts by counting shifts both within the same speaker and between different speakers — the latter being something earlier curriculum learning methods for ERC overlooked — using an arousal-valence emotion wheel to measure how similar any two emotions are.

4. State-of-the-art results on two benchmarks. Weighted F1 of 71.95 on IEMOCAP and 70.44 on MELD, with code and data released publicly.

Main Findings

• Both benchmarks improved. PRC-Emo reaches 71.95 weighted F1 on IEMOCAP (a 0.76 point gain over the previous best, InstructERC at 71.39) and 70.44 on MELD (0.61 points over BiosERC at 69.83). It also achieves the best accuracy on both datasets, at 71.03 and 71.50 respectively.

• Prompt design is the single largest contributor. When prompt, retrieval, and curriculum learning are all removed, IEMOCAP drops by 3.41 points and MELD by 1.72 points. Within the prompt itself, the explicit/implicit emotion interpretations matter more than demonstration retrieval or speaker characteristics.

• Every prompt component adds value. Removing demonstration retrieval costs 0.46/0.21 points; removing the emotion interpretations on top of that costs 1.68/0.78; removing speaker characteristics as well costs 2.05/1.10.

• The cross-speaker difficulty metric beats prior curriculum strategies. Compared against emotional shift (ES) and weighted emotional shift (WES) curricula on identical base models and prompts, the proposed metric wins by 1.52 and 0.47 points on IEMOCAP, and 0.57 and 0.30 points on MELD.

• Model choice interacts strongly with dataset. Qwen2.5-7B outperforms Qwen3-8B by over a point on IEMOCAP (71.95 vs. 70.86), while the reverse holds on MELD (69.73 vs. 70.44). The authors speculate this may reflect data leakage, since MELD derives from the widely available Friends scripts.

• Class imbalance is addressed at the data level. The generated corpus is deliberately balanced across five emotions (happiness, neutral, fear, sadness, anger), with each category falling in a narrow band between roughly 2,500 and 3,300 sentences.

Methodology in Plain English

The framework runs in two stages. In the first, a separate large language model reads the dialogue history and produces three kinds of commentary: what emotion the speaker is openly showing, what emotion the speaker is probably hiding, and a short description of the speaker's personality. These act as expert notes handed to the model being trained.

In the second stage, those notes are assembled into a single structured prompt together with the instruction, the last few utterances of context, and three retrieved examples of similar utterances with their correct emotion labels. The retrieved examples come from the authors' new repository and are matched using sentence embeddings and cosine similarity. The model — a Qwen variant fine-tuned with LoRA — then predicts the emotion label.

Training follows a curriculum. Every conversation is scored for difficulty using the new metric, which sums up how much the emotions shift across the dialogue, weighting shifts by how dissimilar the two emotions are on an arousal-valence wheel and counting both same-speaker and cross-speaker transitions. Conversations are sorted by this score and split into two buckets. The model trains on the easier bucket first, then the harder one is added, and finally the whole dataset is used for a couple of epochs of consolidation. The whole training run spans four epochs, with the curriculum active for the first two.

The authors validate the design through five sets of experiments: comparison against prior methods, module ablation, prompt component ablation, curriculum strategy comparison, and a base-model swap. All results are averaged over five random seeds.

Why This Matters

ERC sits at the intersection of language understanding and affective computing, and the paper's argument — that separating expressed emotion from felt emotion improves recognition — offers a reusable design principle for other dialogue-level affective tasks, including sentiment analysis, empathetic response generation, and mental health dialogue systems. The released retrieval repository is arguably the most durable contribution, since it gives the field a shared, structured, multi-source resource where previously each paper built its own private pool.

Real-world applications include:

• Customer service chatbots that detect frustration a customer is downplaying and escalate accordingly.
• Mental health and telehealth tools that flag unspoken distress in patient language, where what is not said matters most.
• Meeting and collaboration analytics that surface team morale problems from workplace conversation.
• Educational tutoring systems that read confusion or anxiety in a student's dialogue and adapt pacing or difficulty.

For industry, the appeal is practical: the method improves results by changing prompts, retrieval data, and training order rather than by scaling model size, which keeps inference costs flat. The LoRA-based training runs on a single consumer-grade GPU, putting it within reach of teams without large compute budgets.

Future Directions

• Stronger and more diverse backbones. The paper only tests two open-source Qwen models due to resource limits, and the anomalous IEMOCAP/MELD split raises unresolved questions about how model choice and benchmark contamination interact. Testing across more model families would clarify the generalizability claim.

• More efficient prompting paradigms. The current pipeline requires a separate LLM call to generate external knowledge for every dialogue, plus an embedding lookup per utterance. Reducing that overhead without losing the explicit/implicit signal is an obvious engineering target.

• Finer-grained curriculum designs. The current approach uses only two difficulty buckets and a single difficulty score per conversation. Utterance-level difficulty, adaptive bucket boundaries, or dynamically adjusted schedules could extract more from the same data.

• Extending the retrieval repository. The repository currently covers three benchmarks plus the authors' synthetic data. Adding multimodal signals (audio, facial cues), additional languages, and more emotion categories would test how far the diversity argument really stretches.

Target Audience

Researchers and graduate students working on affective computing, dialogue systems, or LLM fine-tuning will find the strongest material here, particularly the curriculum difficulty metric and the ablation evidence for explicit/implicit emotion prompting. Practitioners building conversational products — customer support, EdTech, telehealth, HR analytics — can use the framework as a practical recipe, and the released 36,712-utterance retrieval repository is directly usable by anyone doing few-shot or retrieval-augmented emotion classification. Readers without prior exposure to ERC benchmarks or LoRA will need to pick up some background first, as the paper assumes that context.

Authors’ abstract

Emotion Recognition in Conversation (ERC) is a crucial task for understanding human emotions and enabling natural human-computer interaction. Although Large Language Models (LLMs) have recently shown great potential in this field, their ability to capture the intrinsic connections between explicit and implicit emotions remains limited. We propose a novel ERC training framework, PRC-Emo, which integrates Prompt engineering, demonstration Retrieval, and Curriculum learning, with the goal of exploring whether LLMs can effectively perceive emotions in conversational contexts. Specifically, we design emotion-sensitive prompt templates based on both explicit and implicit emotional cues to better guide the model in understanding the speaker's psychological states. We construct the first dedicated demonstration retrieval repository for ERC, which includes training samples from widely used datasets, as well as high-quality dialogue examples generated by LLMs and manually verified. Moreover, we introduce a curriculum learning strategy into the LoRA fine-tuning process, incorporating weighted emotional shifts between same-speaker and different-speaker utterances to assign difficulty levels to dialogue samples, which are then organized in an easy-to-hard training sequence. Experimental results on two benchmark datasets -- IEMOCAP and MELD -- show that our method achieves new state-of-the-art (SOTA) performance, demonstrating the effectiveness and generalizability of our approach in improving LLM-based emotional understanding.