Who Are They to Each Other? Multi-Agent Reasoning for Speaker Relationship Inference

AI summary

Overview

Research area: Multi-agent LLM reasoning applied to spoken conversation understanding and social signal processing.

Technical level: Intermediate. The paper assumes familiarity with LLM prompting paradigms (zero-shot, debate, judge-based aggregation) and basic speech/NLP evaluation conventions, but the core ideas are conceptual rather than mathematically heavy.

Scope: The paper introduces and evaluates a training-free multi-agent framework for inferring the social relationship between two speakers from their conversation, tested across text, audio, and audio+text modalities.

What This Paper Is About

When two people talk, the way they talk reveals whether they are close friends, siblings, coworkers, or total strangers. This paper asks whether an LLM-based system can figure out that relationship automatically from a recorded conversation, and whether it can do so without any task-specific training. The central challenge is that relational cues are subtle, spread across many turns, and often ambiguous, so the authors argue that a single model giving one answer is the wrong shape for the problem.

Key Contributions

1. Multi-Role Multi-Agent Debate (MRMAD): A variation of standard multi-agent debate where each agent is assigned a complementary, theoretically grounded perspective rather than duplicating the same viewpoint. Two instantiations are proposed: one with a Linguist, Sociologist, and Psychologist role (LSP), and one grounded in social psychology theories — Relational Models Theory, Communal/Exchange Theory, and Interpersonal Circumplex Theory (RCI) — where agents debate by exchanging quantified dimension scores rather than vague impressions.

2. Multi-Agent Compete (MAC): A tournament-style protocol in which three agents submit independent judgments, an LLM judge conducts pairwise comparisons, the weakest agent is eliminated, the surviving two revise after seeing the bracket results, and a final head-to-head adjudication picks the winner. This deliberately delays consensus to preserve competing hypotheses.

3. A multimodal benchmark construction: The authors filter the Seamless Interaction dataset's naturalistic subset into a 607-dialogue evaluation set (441 familiar, 166 stranger) with four fine-grained familiar subtypes, and evaluate six methods on identical splits across three modality settings.

4. A human evaluation study: 27 native English speakers annotate 50 conversations across three modalities, providing inter-annotator agreement measurements and a comparison point for model performance.

Main Findings

• Structured interaction helps, especially on imbalanced metrics: All three proposed methods beat every baseline on multi-class macro F1 and accuracy in the text setting, with gains concentrated in the Friends and Coworkers categories.

• Multi-Agent Compete is the strongest proposal in text: It improves macro F1 by 0.5 points over the best baseline (CortexDebate) on binary classification and 0.8 points on fine-grained classification, with accuracy gains of 3.2 and 3.4 points respectively.

• Audio+Text gives the largest and most consistent gains: MAC leads binary macro F1 on GPT-audio-1.5 by 3.3 points over the best baseline, while Multi-Role (LSP) leads multi-class macro F1 by 2.4 points. On Qwen2.5-Omni-7B, Multi-Role (RCI) improves binary macro F1 by 2.9 points over the best baseline.

• Audio-only is the weakest setting for the framework: Results are mixed, with some metrics improving substantially (Multi-Role RCI improves binary macro F1 by 7.8 points under Qwen) and others falling slightly behind baselines. The authors attribute this to debate methods relying on quotable textual evidence that is hard to extract from audio alone.

• More reasoning depth does not help: GPT-5 mini outperforms all GPT-5.4 variants on both binary and fine-grained accuracy, suggesting the task rewards integrating distributed evidence over long-chain deduction.

• Romantic/partner relationships resist the approach: All baselines perform identically on this category while every multi-agent method underperforms them, indicating the framework's benefits are not uniform.

• Humans find the task hard too: Inter-annotator agreement is low across the board (Krippendorff's alpha between 0.17 and 0.34), with the lowest agreement in the text modality and the highest in audio. Models can exceed averaged individual human votes in text-included settings, but humans make more balanced binary decisions in the audio-only setting.

• Adding text to audio does not simply add information: Human agreement in audio+text is lower than in audio alone, suggesting that conflicts between transcript cues and acoustic impressions can increase uncertainty.

Methodology in Plain English

The researchers framed relationship inference as a two-stage task: first decide familiar versus stranger, then, if familiar, pick the specific relationship type.

Rather than training a model, they orchestrated several LLM calls at inference time. In the debate variants, multiple copies of the same model answer independently, then read each other's answers and revise over additional rounds. The key twist is that agents are not identical — they are told to look at the conversation through different lenses, such as a linguist analyzing word choice, a psychologist tracking emotional dynamics, and a sociologist reading status and social norms. In the RCI variant, agents score the conversation along theoretical dimensions (like communal sharing versus authority ranking) and argue over those numbers instead of general impressions.

The competition variant removes collaboration entirely. Three agents submit answers as if entering an elimination tournament. A judge model compares them two at a time, throws one out, tells the survivors what happened, lets them revise, and then picks a final winner. The intuition is that keeping rival hypotheses alive longer helps when the evidence is genuinely ambiguous.

They tested all of this on the same filtered set of conversations, using GPT-5-mini for text and GPT-audio-1.5 and Qwen2.5-Omni-7B for audio and audio+text, and compared against zero-shot prompting, standard multi-agent debate, and CortexDebate. They also ran a human study on 50 of those conversations to calibrate how difficult the task really is.

Why This Matters

Impact on research: The paper carves out speaker relationship inference as a distinct social reasoning benchmark for multimodal LLMs, and shows that how you structure inference-time interaction matters more than simply adding agents or increasing reasoning depth. It also provides evidence that current multimodal models do not extract acoustic relational cues as effectively as humans do, which is a concrete gap for speech-and-language research.

Real-world applications:

• Meeting and call analytics that detect team dynamics, familiarity, or hierarchy for organizational insight.
• Mental health and social work tools that flag shifts in relationship closeness from recorded conversations.
• Media production and content moderation, where knowing whether speakers are intimates or strangers changes how content should be handled or tagged.
• Assistive and socially aware dialogue systems that adjust tone and register based on who is talking to whom.

Industry relevance: The training-free nature of the framework is directly relevant to teams that want to deploy social inference without collecting expensive labeled social data. It also matters for cost-sensitive deployments, since the paper's own limitations section flags that multi-agent protocols multiply inference calls — an accuracy-versus-cost trade-off that production teams will weigh carefully.

Future Directions

• Cross-dataset generalization: All experiments use one benchmark, so it remains unknown whether these protocols transfer to other conversation corpora, languages, or recording conditions.

• Efficiency and cost: Multi-agent debate and competition require many LLM calls per inference. Studying accuracy-per-token or accuracy-per-latency trade-offs, and pruning agent interactions, is a natural next step.

• Better use of acoustic evidence: The audio-only results and the human audio advantage both point to prosody, laughter, pacing, and turn-taking as underexploited signals. Specially designed acoustic agents or audio-native reasoning protocols are an open direction.

• Deeper treatment of ambiguity: Since even humans disagree substantially, future work could move beyond single-label ground truth toward modeling a distribution over plausible relationships, or evaluating systems on calibrated uncertainty rather than accuracy alone.

Target Audience

This paper suits researchers and graduate students working on multimodal LLM reasoning, speech understanding, and social signal processing, particularly those interested in inference-time orchestration techniques as an alternative to fine-tuning. It is also useful for applied engineers considering multi-agent LLM pipelines for conversation analysis, and for social scientists interested in how machine judgments of interpersonal relationships compare with human perception.

Authors’ abstract

Inferring speaker relationships from spoken conversations is an important step towards socially aware speech understanding. However, this task remains underexplored, and supervised modeling is costly to train and scale. At the same time, existing inference-time LLM approaches provide limited structure for handling subtle, distributed, and multimodal relational cues that may support multiple plausible interpretations. To address these limitations, we introduce a training-free multi-agent reasoning framework that organizes inference through structured interaction among LLM agents, allowing relationship judgments to be proposed, challenged, and adjudicated without task-specific training. We instantiate this framework with two complementary designs. We propose Multi-Role Multi-Agent Debate as a task-specific adaptation of standard multi-agent debate for speaker relationship inference, assigning agents complementary roles or social-theory-grounded perspectives rather than a single undifferentiated viewpoint. In contrast, we introduce Multi-Agent Compete, a competition-based protocol that compares agent judgments through pairwise adjudication, eliminates weaker candidates, and retains the most defensible one. We evaluate these methods on the Seamless Interaction dataset across different modality settings, covering both binary classification and fine-grained relationship-detail prediction. Results suggest that they improve over zero-shot and existing multi-agent baselines in most cases. Human evaluation further suggests that this task is challenging even for people. LLM methods can sometimes outperform human annotators in text-included settings but are less competitive in the audio setting. Together, these findings suggest that relationship inference benefits from structured inference-time interaction among agents, while acoustic cues are not yet fully captured by current models.