MASCOT: Multi-Agent Socio-Collaborative Companion Systems

AI summary

Overview

• Research area: Natural Language Processing, specifically multi-agent large language model systems for emotional support and collaborative dialogue.
• Technical level: Intermediate. The framework builds on accessible concepts (personas, prompting, conversational agents) but the training pipeline relies on reinforcement learning from AI feedback, reward modeling, and GRPO optimization.
• Scope: The paper introduces MASCOT, a multi-agent framework that trains persona-specialized speaker agents and a coordinating director agent to produce more role-consistent, less redundant socio-collaborative dialogue across emotional-support and workplace-collaboration settings.

What This Paper Is About

Multi-agent systems built on large language models are being proposed as companions that can offer emotional and cognitive support, but they frequently break down in two ways: individual agents lose their assigned personalities and regress into generic assistant behavior (persona collapse), and groups of agents drift into agreeable, repetitive, non-constructive exchanges (social sycophancy). MASCOT addresses both failure modes with a two-phase training approach that aligns each agent to its own persona at the individual level, then trains a director agent to coordinate the group at the collective level. The goal is dialogue that is simultaneously role-consistent and collectively productive, rather than merely task-efficient.

Key Contributions

1. Unified framework. MASCOT is presented as a multi-agent framework that enables socio-collaborative companions in multi-perspective group conversations, targeting user-agent interaction quality rather than task efficiency alone.
2. Bi-level optimization. The paper proposes a low-resource pipeline that harmonizes individual persona adherence (Persona-Aware Behavioral Alignment) with collective interaction synergy (Collaborative Dialogue Optimization via a director policy).
3. Extensive evaluation. MASCOT is evaluated on human-grounded contexts drawn from in-domain and out-of-domain datasets using human evaluation, rubric-based evaluation with multiple LLM judges, three-way comparisons, and automatic metrics.
4. Reported gains. MASCOT reports up to +14.1 Persona Consistency on Empathetic Dialogues and up to +10.6 Social Contribution on QMSum relative to strong baselines.

Main Findings

• Persona consistency gains: Consistency improves by up to +14.1 (ED-Positive) and +8.7 (QMSum-Product), the largest category of improvement reported.
• Social contribution gains: Social Contribution improves by up to +7.9 (ED-Positive) and +10.6 (QMSum-Product).
• Overall scores on Empathetic Dialogues: MASCOT reaches 76.9 ± 6.1 Overall on ED-Positive, 76.4 ± 6.4 on ED-Neutral, and 70.5 ± 8.2 on ED-Negative, versus the strongest multi-agent baselines (MultiAgentESC at 67.6 ± 10.2 on Positive, 68.5 ± 6.0 on Neutral, 63.0 ± 6.0 on Negative).
• Empathetic support quality: MASCOT reaches 76.2 (ED-Positive) and 75.6 (ED-Neutral) on Empathetic Support Quality, described as substantially exceeding Zero-Shot CoT.
• Difficulty with negative valence: Performance drops on ED-Negative; MASCOT's Emotional Expressiveness falls from 75.3 (ED-Positive) to 68.2 (ED-Negative). The paper attributes this to the difficulty of calibrating intensity for high-arousal states such as distress or anger from a single turn, and describes a "prudence trade-off" that prioritizes Response Relevance (74.0) over proactive mirroring.
• Persona-level stability: Breaking results down by ED persona role shows all personas maintain strong conversation quality while emphasizing complementary strengths, which the authors take as evidence against collapse into a single generic assistant style.
• Ablation on persona alignment: Disabling Persona-Aware Behavioral Alignment (MASCOT-P) triggers persona collapse, dropping Empathetic Support Quality from 72.8 to 65.1 and Consistency from 76.6 to 72.3.
• Ablation on collaborative optimization: Removing Collaborative Dialogue Optimization (MASCOT-C) leaves individual persona consistency stable but reduces Social Contribution from 71.3 to 70.1 and increases redundancy.
• SFT performs poorly: The evaluated SFT configuration reaches 44.1 Overall on ED-Negative, below the Zero-Shot baseline at 55.1, and has the highest contradiction rate in the automatic metrics (0.195 ± 0.179 versus 0.133 ± 0.148 for MASCOT and 0.008 ± 0.016 for Few-shot CoT). The authors interpret its weakness as poorly coordinated diversity.
• Non-redundancy metrics: MASCOT records Entailment 0.098 ± 0.097, Neutral 0.769 ± 0.177, and Self-BLEU 0.316 ± 0.151 on Empathetic Dialogues, compared with Few-shot CoT at 0.564 ± 0.104, 0.429 ± 0.104, and 0.551 ± 0.058 respectively.
• Human preference: In a Prolific study with 40 US-based native-English annotators and 200 rater-level judgments per baseline, MASCOT was preferred over Few-Shot CoT in 69% of judgments and over MultiAgentESC in 68.5%, with p < 0.0001 for both. Highest preference rates occurred in Neutral/Ambiguous situations (75%+).
• Second human study: In a blinded study of 100 conversations rated on helpfulness, emotional appropriateness, and perceived safety, MASCOT was preferred in 151 of 200 judgments (75.5%), with both annotators agreeing in 57 cases. MASCOT obtained a higher mean in five of six rater-by-metric comparisons; the largest differences were helpfulness (+0.56, p = 0.0006) and emotional appropriateness (+0.40, p = 0.0031).
• Big Five sensitivity: Across 500 conversations generated from 50 Empathetic Dialogues contexts and 10 personality profiles, High-Openness profiles achieved the highest Persona Consistency (79.0 ± 7.3) with Overall 74.4 versus 73.9 for Low-Openness; High-Conscientiousness reduced Overall from 74.5 to 71.5 (Emotional Expressiveness −4.6, Empathetic Support Quality −4.0); High-Agreeableness improved Overall by +1.9 while Low-Agreeableness had the lowest Social Contribution (64.8); High-Extraversion raised Social Contribution by +2.5 and Coherence by +1.9; Neuroticism conditions were stable (70.7 vs. 70.8).
• Model-scale sensitivity: Reducing the director from 8B to 1.7B caused only modest changes (agent-specific 68.4 to 67.2; collective 71.5 to 69.1), but at 0.6B collective performance fell to 51.7 while agent-specific scores stayed at 63.9. Downsizing speakers to 0.6B reduced both agent-specific (57.6) and collective (50.5) performance, indicating speaker scale is a system-wide bottleneck.

Methodology in Plain English

The system is organized as a set of agents, each conditioned on a distinct persona profile specifying traits such as linguistic style, domain expertise, and emotional disposition (examples in the paper include Empathetic Listener, Critical Thinker, Emotional Validator, Action Guide, and Growth Advocate). Training happens in two phases rather than through prompting alone.

In phase one, Persona-Aware Behavioral Alignment, the authors take a supervised fine-tuned base policy and sample multiple candidate responses per context and persona. An LLM judge (GPT-4o, at temperature 0.0) scores each candidate against fine-grained rubric criteria on a 1-to-5 scale; only pairs whose aggregate scores differ by at least a margin are kept, discarding ambiguous comparisons. A reward model is trained on those pairs with a standard pairwise logistic loss, and the speaker policy is then optimized with reinforcement learning using a composite reward combining the learned persona reward with a rule-based format reward that requires a reasoning trace inside <think> tags and penalizes verbose final answers. Group Relative Policy Optimization (GRPO) is used because it estimates the baseline from the group mean of sampled outputs, reducing memory and compute cost.

In phase two, Collaborative Dialogue Optimization, a director agent learns turn-taking and high-level strategy over the now-frozen persona-aligned speakers. At each turn the director observes the conversation history and emits a directive naming the next speaker and giving instructions (for example, telling an agent to amplify a user's pride via active-constructive responding). Complete episodes are generated with a frozen simulated user that speaks with probability 0.5, and the director is trained at randomly sampled decision points using prefix-suffix rollouts: a prefix is constructed with frozen models, the director acts, and the suffix is rolled out for scoring. A sequence-level group reward model scores directives together with the resulting full trajectories, with an added diversity indicator that is +1 when the dialogue block contains contributions from distinct personas and −1 otherwise. The director is then updated with GRPO on group-relative advantages.

Experiments use Qwen3-8B for the director, speakers, and simulated user, with LoRA (r = 16, alpha = 32, dropout = 0.1), batch size 16, and one epoch per policy-tuning phase; trainable parameters are approximately 0.187% of the base model. Reward models are fine-tuned from Qwen3-0.6B for one epoch at batch size 32; 8 candidates are sampled per sample and persona. Evaluation uses LLM-as-a-Judge because the task is open-ended and lacks ground-truth references. Implementation used TRL, vLLM, and HuggingFace transformers on NVIDIA A100 (80GB) GPUs.

Why This Matters

• Impact on research: The paper reframes multi-agent LLM design away from pure task efficiency toward interaction quality, and offers a concrete, low-resource recipe (approximately 0.187% trainable parameters via LoRA) for separating persona alignment from group coordination, an approach that could be reused in other multi-agent dialogue settings.
• Digital social support groups: Persona-based agents could be orchestrated to provide multi-perspective emotional support, with complementary roles such as validation, reframing, and action planning replacing a single generic assistant voice.
• Meeting and workplace collaboration: The QMSum personas (Minutes Scribe, Decision Logger, Action Item Captain, Critic) suggest applications in meeting summarization, decision tracking, and reducing re-litigation of settled topics.
• Edge and on-device deployment: The sensitivity analysis shows a director as small as 1.7B can preserve much of the coordination quality, which matters for latency- and resource-constrained settings, though speaker scale remains a bottleneck.
• Industry relevance: Providers building companion, coaching, or workplace-assistant products can treat persona fidelity and non-redundancy as separately trainable objectives, and the reported human-preference numbers give a benchmark for evaluating whether such products feel role-consistent rather than agreeable and repetitive.

Future Directions

• Proprietary-model adaptation. Because the method relies on parameter-level updates, it currently favors open-source models with accessible weights; the paper suggests black-box alternatives such as specialized few-shot prompting or CoT refinement, but these are not evaluated here.
• Multilingual and cross-cultural persona modeling. The study is limited to English-language interactions, and the authors identify cultural and linguistic variation in persona modeling as an open opportunity.
• Handling negative-valence conversations. The drop from 75.3 to 68.2 Emotional Expressiveness between ED-Positive and ED-Negative highlights unresolved questions about calibrating response intensity for high-arousal states.
• Smaller and more efficient speaker agents. The scale analysis leaves open how to preserve collective and agent-specific quality when speaker models shrink to 0.6B-scale, where performance degraded substantially.

Target Audience

Researchers and practitioners working on multi-agent LLM systems, dialogue and emotional-support agents, persona modeling, and reinforcement learning from AI feedback will benefit most. The paper is also relevant to product teams building companion or workplace-collaboration assistants, and to readers interested in evaluation methodology, given its combination of human preference studies, multiple LLM judges, three-way comparisons, and automatic non-redundancy metrics.

Authors’ abstract

Multi-agent systems (MAS) are emerging as promising socio-collaborative companions for emotional and cognitive support. However, existing systems frequently suffer from persona collapse, where agents revert to generic, homogenized assistant behaviors, and social sycophancy, where agents produce redundant, non-constructive dialogue. We propose MASCOT, a multi-agent framework for multi-perspective socio-collaborative companions. MASCOT introduces a novel bi-level optimization strategy to harmonize individual and collective behaviors: 1) Persona-Aware Behavioral Alignment, an RLAIF-driven pipeline that finetunes individual agents for agent-specific identities; and 2) Collaborative Dialogue Optimization, a group-level adaptation process that promotes complementary, diverse, and productive discourse. We evaluate MASCOT using human-grounded contexts drawn across both in-domain and out-of-domain (OOD) settings against state-of-the-art baselines. MASCOT improves persona consistency by up to +14.1 and social contribution by up to +10.6. A broad evaluation suite, including human evaluation, multiple LLM judges, three-way comparisons, and automatic metrics, further shows that MASCOT produces more role-consistent and less redundant multi-agent dialogue.