RESCUE-BENCH: Towards Relation-Aware Multi-Party Emotional Support Conversation Systems

AI summary

Overview

Research area: Natural language processing / affective computing — emotional support conversation systems, specifically extending them from one-on-one settings to multi-party scenarios where participants have relationships with each other.

Technical level: Intermediate. Readers should be comfortable with LLM evaluation terminology (precision/recall/F1, MRR, LLM-as-judge scoring, BERTScore), but the paper's own framing is conceptual rather than mathematically heavy.

Scope: The paper defines a new task ("relation-aware emotional support conversation"), builds a benchmark called RESCUE-Bench from real couple and family interview videos, and evaluates ten LLMs across six tasks.

What This Paper Is About

Existing emotional support chatbots assume a one-on-one exchange between a help-seeker and a supporter, and focus on that one person's emotional state. The authors argue this misses situations where several people who are related to each other — romantic partners, parents and children — need support at the same time, and where what one person feels and does directly affects the others. The paper's goal is to define this harder setting as a formal task, build a real-world benchmark for it, and measure how far current LLMs fall short.

Key Contributions

1. A new task definition: The authors introduce "relation-aware emotional support conversation," which extends emotional support from individual-centered interactions to relation-centered multi-party scenarios. Unlike relation-agnostic settings that target one seeker's emotional improvement, this task aims at group-level outcomes by addressing vulnerable members' distress while accounting for interpersonal tensions and dependencies.

2. A new benchmark: RESCUE-Bench is constructed from real multi-party interview conversations in two relational scenarios, couples and families, with 191 samples, 7,079 annotated turns, and 1,064.8 minutes of video.

3. Six relation-related tasks: The benchmark organizes evaluation into two capabilities. Relational Understanding covers Emotion Recognition (ER), Viewpoint Prediction (VP), and Relation Pattern Prediction (RPP). Relation-Sensitive Support covers Intervention Time Prediction (ITP), Support Target Prediction (STP), and Support Strategy Prediction (SSP). The paper notes traditional ESC involves mainly ER and SSP, while relation-aware ESC additionally requires VP, RPP, ITP, and STP.

4. A benchmark of ten LLMs: The authors evaluate ten representative LLMs zero-shot with identical task definitions and prompt formats, revealing systematic weaknesses on relation-intensive tasks.

Main Findings

• Local cue tasks are handled reasonably well: Average ITP F1 reaches 82.62% across the ten models, and average ER LLM-as-judge score reaches 4.05/5. The authors read this as models being able to identify emotionally salient moments and infer individual affective states from dialogue context.

• Relation-intensive tasks collapse: RPP is the hardest task, with the best accuracy only 45.60% and a model average of 40.45%.

• Viewpoint inference is harder than emotion recognition: VP achieves a relatively high average BERTScore of 0.8611, but its average LLM-as-judge score is 3.58/5 versus 4.05/5 for ER, suggesting that inferring a viewpoint toward another person is harder than recognizing one's own emotion.

• Support decisions degrade down the pipeline: STP performance is moderate (64.34% recall, 78.35% MRR), while SSP drops further (31.88% recall, 44.96% MRR), indicating that choosing a relation-sensitive support strategy is particularly difficult.

• The strongest general model is not the strongest relational reasoner: Qwen3.5-Plus attains 94.60% ITP F1 and a 4.22 ER score, but its RPP accuracy is only 40.27%, below DeepSeek-V4-Pro, Kimi K2.5, and Qwen3-Max (43.22%), and it does not achieve the best results on STP, SSP, or VP.

• Different models lead on different tasks: DeepSeek-V4-Pro performs best on RPP, STP, and SSP, while DeepSeek-V4-Flash obtains the highest VP LLM-as-judge score (4.08).

• RPP difficulty is about tracking change, not just long context: Accuracy across ten normalized dialogue position bins fluctuates rather than declining monotonically, and the gold relation-pattern distribution shifts across those stages, indicating the challenge is tracking dynamic transitions of relation patterns over time.

• Viewpoint failures stem from indirect expression: Participants do not always address each other directly and may convey attitudes through the therapist; vulnerable feelings such as hurt or grievance may be disguised as anger or dominance, and models capture only coarse-grained viewpoints rather than fine-grained target-specific reasoning.

• Strategy failures are precision rather than ranking failures: SSP's low recall alongside relatively higher MRR suggests models can narrow the candidate strategy set but struggle with fine-grained selection.

• Relational dynamics are nonlinear: A row-normalized transition matrix over consecutive relation-pattern labels shows that negative cycles such as pursue-withdraw and attack-attack do not usually move directly to stable coordination; repair softening often acts as an intermediate state before constructive alignment, and pursue-withdraw frequently reappears after withdraw-withdraw, repair softening, and mixed transition, functioning as a recurring attractor.

• Dataset composition: 174 couple clips and 17 family clips. Family samples are longer and involve more speakers on average (14.11 vs. 4.74 minutes per sample; 4.06 vs. 3.00 average speakers per sample), and therapist turn share is higher in family sessions (47.1%) than in couple sessions (35.4%), with 37.4% overall.

Methodology in Plain English

The researchers started from real documentary-style interview videos in two relational settings: couples and families. They manually identified independent interview segments, split them into self-contained conversation clips, and discarded clips shorter than two minutes because those usually lack enough relational context.

To make the data usable, they aligned subtitle, audio, and video streams along a shared timeline into multimodal interaction segments, and described each segment with six structured dimensions: timing and entity information; verbal content; individual cues (tone of voice, body posture, facial expressions, self-directed behavior, inferred internal emotion); relational stance (interaction behavior and viewpoints or attitudes toward others); therapist strategy (support strategy and intention); and relation pattern (higher-level relation-cycle states, reasons, and supporting evidence).

Annotation was done in two stages. First, Gemini-3.1-Pro pre-annotated each video segment according to that framework, using both video content and aligned subtitles. Second, the team built an online verification system and invited three PhD-level annotators to check faithfulness and consistency against the original videos, revising incorrect annotations and discarding segments with severe recognition errors, speaker mismatches, or substantial inconsistency with the video evidence.

Each of the six benchmark tasks was then framed as a prediction problem over a conversation context, with future turns hidden. ITP is binary classification (reported with precision, recall, F1). RPP is multiclass classification (accuracy). STP and SSP are top-3 ranking tasks (Recall@1 and MRR). ER and VP are generative tasks scored by GPT-5.4 acting as an LLM-as-judge on a 5-point Likert scale, together with BERTScore for semantic similarity. The authors also report conducting human evaluation to validate the correlation between human and LLM judgments, though the specific results of that validation are not included in the available text. All ten models were run zero-shot with identical task definitions and prompt formats.

Why This Matters

Impact on research. The paper shifts emotional support conversation research away from the assumption of a single help-seeker and a single supporter. It provides the first benchmark (per the paper's own comparison table) that centers interpersonal relations in emotional support, requiring models to reason about directed stance and dynamic relation patterns and to output timing, target, and strategy signals rather than just a response. It also connects AI research to a long-standing psychology and psychotherapy literature on relational dynamics that the authors say remains underexplored in the AI community; existing AI attempts on subtopics such as couple therapy have been preliminary case studies rather than systematic examinations.

Real-world applications:

• Couples counseling support tools that help a therapist decide when to step in, whom to address, and which strategy to use.
• Family therapy or family mediation settings, where more speakers and longer sessions (family samples average 14.11 minutes and 70.82 turns versus 4.74 minutes and 33.76 turns for couples) make timing and target selection harder.
• Training or supervision tools that help human practitioners notice recurring negative interaction cycles such as pursue-withdraw before they escalate.
• Group or team wellbeing settings, which the authors cite alongside family and couple contexts as real-life scenarios where relational support matters (citing Cheng and Chau, 2022, for team settings).

Industry relevance. Any organization building conversational agents for counseling, coaching, customer de-escalation, or community moderation faces the same core problem: deciding when to speak, whom to address, and how to phrase support when several related people are in the conversation. The paper's finding that strong general-purpose LLMs still fail at relational reasoning, with RPP accuracy topping out at 45.60%, is a direct caution against deploying such systems as clinical or consequential decision-makers. The authors explicitly state the benchmark is not designed for clinical deployment and should not substitute for professional therapy.

Future Directions

• Broaden demographic and cultural diversity. The authors note the source documentary videos carry selection biases from media production, editing, participant demographics, and cultural context, and suggest incorporating broader sources of naturally occurring relational interactions.

• Improve annotation reliability for subjective labels. Relation patterns, directed viewpoints, support targets, and support strategies involve inherently subjective judgments; the authors propose larger annotator pools and more fine-grained annotation guidelines to strengthen reliability.

• Handle long-tailed label distributions. Rare relation patterns and support strategies may affect training and evaluation; proposed directions include data augmentation, rebalancing strategies, or rare-label evaluation protocols, while acknowledging that the imbalance reflects real-world behavior.

• Improve reproducibility under privacy constraints. Because raw videos, audio, and visual content are not redistributed, the authors suggest privacy-preserving data-sharing mechanisms or controlled-access protocols that permit multimodal reproduction while maintaining ethical safeguards.

Target Audience

Researchers in affective computing and conversational AI who work on emotional support, empathetic dialogue, or multi-party dialogue; benchmark and evaluation researchers interested in LLM-as-judge methodology and relation-aware reasoning; and computational social scientists or clinical/NLP collaborators studying couples and family interaction. It is also relevant to practitioners building counseling or support agents who need a clear picture of where current LLMs fail before deploying in multi-party settings. The paper is written at an intermediate level: the six task definitions and metric choices are approachable for readers with basic NLP evaluation background, while the theoretical grounding in therapy process and alliance research adds context for those from psychology.

Authors’ abstract

Existing emotional support conversation systems mainly focus on one-on-one seeker-supporter interactions and individual emotional states, leaving interpersonal relations in multi-party scenarios underexplored. In this work, we introduce relation-aware emotional support conversation, a new task that evaluates whether LLMs can capture and utilize the evolving dynamics of relationships to offer more effective emotional support. We construct RESCUE (Relation-aware Emotional Support Conversation Understanding and Evaluation Benchmark) from real couple and family interview conversations, containing 191 samples, 7,079 annotated turns, and 1,064.8 minutes of video. Based on rich annotations of socio-emotional and support-related dynamics, RESCUE defines six tasks that evaluate two core capabilities required for relation-aware emotional support: Relational Understanding and Relation-Sensitive Support. Experiments with ten LLMs show that current models perform relatively well on tasks relying on local emotional or intervention cues, but struggle with relation-intensive tasks such as relation pattern prediction, viewpoint prediction, and support strategy prediction. These findings reveal the limitations of current LLMs in modeling interpersonal relations and making relation-sensitive support decisions.