What Makes a Social Agent's Interactions Effective? Learning from Verifiable Outcomes in Paired Worlds

AI summary

Overview

Research area: Natural Language Processing / LLM social agents, dialogue evaluation, and reinforcement learning for multi-turn interaction.

Technical level: Intermediate. The evaluation design (paired scenarios, rule-based checking) is accessible, but the training method assumes familiarity with policy optimization, credit assignment, and potential-based reward shaping.

Scope: The paper introduces HICO-Bench, a 180-scenario benchmark that scores social dialogue by whether the agreed plan is verifiable, and PaCPO, a reinforcement learning method that converts end-of-conversation outcomes into turn-level credit using paired scenario variants.

Authors: Caixin Kang, Ke Wang, Yuchuan Wu, Xiaoqiang Wang, Ling Li, Bingxue Qiu, Yongbin Li, Yoichi Sato (Alibaba Token Hub, Alibaba Group; The University of Tokyo). arXiv:2610.20195v1 [cs.CL], 14 Oct 2026, CC BY 4.0.

What This Paper Is About

Existing evaluations of social LLM agents mostly rely on LLM judges that rate the conversation itself, without checking whether the plan the participants actually settled on works for them. That is a problem because a reply like "Don't worry, I'll take over" sounds supportive yet commits to a plan that fails when the other person can attend and only wanted a slide review. The paper builds a benchmark where an acceptable outcome depends on private conditions revealed through dialogue, and then uses those verifiable outcomes as a training signal to improve an 8B model's coordination behavior.

Key Contributions

1. HICO-Bench: A benchmark of 180 paired scenarios across six coordination families (resource allocation, authorization, bounded quantities, joint scheduling, shared-resource scheduling, set-based consent), evaluated by rule-based Feasibility and judge-based Integrity alongside SOTOPIA ratings.
2. PaCPO (Paired-Credit Policy Optimization): A credit-assignment method that compares tentative plans across paired worlds once the private condition that differs between them has been disclosed, redistributing the terminal reward across turns without changing a conversation's total return.
3. A paired-world design: Each scenario has two worlds sharing the same public context but differing in exactly one private condition, chosen so no plan is correct in both, so an agent that responds identically fails at least one world.
4. An evaluation of thirteen LLMs and five RL baselines, showing that highly rated conversations can still yield failed plans, and that outcome-based training improves Verified Success.

Main Findings

• High conversation ratings do not imply correct plans: Pooled over all thirteen models, 35.99% of conversations with Goal ≥ 8 fail Verified Success (reported as 36% in the abstract), because the recorded plan is infeasible or the conversation does not support it.

• The two checks catch different failures: Qwen3-8B passes Feasibility on 49.58% of tasks but reaches only 20.42% Verified Success, mostly due to Integrity failures.

• Rankings differ between ratings and outcomes: GPT-5.6 Luna scores above Gemini 3.7 Flash on Goal (9.61 vs. 9.54) but far below it on Verified Success (52.08% vs. 82.50%).

• Wide model spread: Verified Success ranges from 20.42% to 82.50% across the thirteen models; the strongest is Gemini 3.7 Flash at 82.50%, and GPT-6 Astra reaches 80.83%.

• Outcome training beats conversation-only baselines: PaCPO reaches 28.33% Verified Success on Qwen3-8B, versus 25.00% for PPO, 26.25% for GRPO, 21.25% for Dr.GRPO, 24.58% for GiGPO, and 24.17% for GraphGPO, improving over Base (20.42%) by 7.92 points.

• The gain comes from Integrity, not Feasibility between PaCPO and GRPO: PaCPO and GRPO both reach 52.92% Feasibility, but PaCPO's Integrity is 35.42% versus GRPO's 31.25%.

• Dialogue itself matters: On 12 development scenarios with hidden conditions, dialogue raises Base Feasibility by 50 points over submitting records directly.

• Family-level variation: The largest improvements over Base are in bounded quantities (+22.50 points) and authorization (+10.00 points); resource allocation stays at 17.5% Verified Success for Base, PPO, and PaCPO, and both scheduling families remain difficult.

• Self-play and transfer: Across three generation repeats, PaCPO reaches 28.33% mean Verified Success in self-play, exceeding Base by 6.39 points and PPO by 1.11 points; on SOTOPIA transfer (70 environment–character pairs, GPT-4o scoring), PaCPO leads in both partner settings.

• External task gains: On CaSiNo (86 test dialogues), PaCPO leads agreement (63.95%) and normalized Nash surplus (49.37%) over PPO (60.47%, 45.86%) and GRPO (53.49%, 42.14%); on MultiWOZ (154 test dialogues) the communication gain is small (81.17% vs. 80.52% for PPO and 79.87% for Base and GRPO).

• Ablations confirm both pairing and timing matter: Replacing the other world's plan with a same-world candidate, or shuffling credit across turns, both reduce Verified Success to 26.25% (2.08 points below full PaCPO); opening the disclosure gate early reduces it to 25.83%, the largest drop.

• Human validation: In a blinded pilot with two expert annotators, human judgments agree with Feasibility on 100% of cases and with Qwen3.8-Max Integrity on 82.22%; annotators prefer PaCPO over Base for helpfulness in 8 versus 5 of 24 pairs.

• SFT plus PaCPO is the highest result: Applying PaCPO after supervised fine-tuning raises Verified Success from 25.83% (SFT alone) to 31.67%, mainly through Integrity (36.25%), though its Feasibility (48.75%) remains slightly below Base.

Methodology in Plain English

Each scenario gives two roles the same public context plus a private role card the partner cannot see. The roles alternate free-form turns, and at the end each independently fills in a structured record describing the plan it believes was agreed.

Two checks decide success. Feasibility is computed by a program: the two records must match up to ordering, the outcome kind must be correct (agreement when a plan exists, decline when none does, defer when outside approval is needed), and the terms must satisfy the family's rules (for example, two shares summing to a $240 bill within each person's limit). Integrity is judged by an LLM reading the conversation and both records, flagging four violations with quoted evidence: an uncorrected factual error such as invented authority, repeated exchanges without progress, a record that contradicts the spoken agreement, or output-format text leaking into dialogue. Verified Success is the product of the two.

The pairing is the distinctive part. Each scenario has two worlds differing in one private condition, with disjoint sets of correct plans, so an identical response fails one world.

For training, all RL methods optimize the same terminal reward, R = 0.8·Feasibility + 0.2·Integrity, which gives partial credit when only one check passes. PaCPO's twist is that after each turn it asks the policy, outside the conversation, what plan it would record if the conversation ended there. At the same turn index in both worlds, it checks each tentative plan against each world's rules. Credit is positive when the plan formed in one world fits that world but the plan from the other world does not. A gate opens only once the differing private condition is available in both conversations (either on the tested role's own card or disclosed by the partner, marked by a frozen LLM annotator). The potential Φ = 0.25·within-world feasibility + 0.10·paired credit is turned into redistributed rewards via r' = r + Φ_t − Φ_{t−1}, which telescopes with γ = 1 so each conversation's total return is unchanged. The redistributed rewards are optimized with PPO using GAE.

Training used Qwen3-8B with LoRA against a frozen Qwen3-8B partner, 384 training conversations (one preassigned role per training scenario, two samples per world). Integrity during training was scored by a frozen local Qwen3.8-27B judge. Group-based baselines use G = 4 and cover 48 training scenarios; PPO and PaCPO draw two per world and cover all 96. Evaluation is on 240 test tasks (60 scenarios, two worlds, two roles), with Integrity and SOTOPIA ratings judged by Qwen3.8-Max. On 240 re-judged outputs, Qwen3.8-Max and Kimi-K3 agree on 90.0% of Integrity decisions (Cohen's κ = 0.78).

Why This Matters

Impact on research: The paper argues that conversation ratings and verified outcomes are complementary rather than interchangeable, and provides a paired-world construction that turns sparse terminal outcomes into turn-level learning signal without changing the objective's total return. It also offers a diagnostic — checking the plan, not just the talk — that prior social-agent benchmarks such as SOTOPIA do not provide.

Real-world applications:

• Workplace coordination: Deciding whether a colleague should cover a client pitch or review slides, where the right answer depends on availability and the kind of help wanted.
• Joint scheduling: Finding a shared 20-minute slot or a 10-minute tripod reservation that fits both parties' windows and durations without overlap.
• Cost and resource splitting: Dividing a $240 fence-repair bill so shares respect each neighbor's bounds and sum to the fixed total.
• Bounded quantities and approval: Agreeing on an integer number of crates within both parties' acceptable intervals, or routing an extra review-hour request through delegated authority.

Industry relevance: The training approach targets small deployable models (Qwen3-8B with LoRA) and shows gains on external task-oriented dialogue and negotiation datasets (MultiWOZ, CaSiNo), which matters for customer-service and bargaining agents where an assistant must leave the world in a correct state rather than merely sound helpful.

Future Directions

• Extending verification beyond structured outcomes: The benchmark currently covers six coordination families with structured outcomes; generalizing rule-based checking to less structured interactions is stated as an important next step.
• Reducing reliance on an LLM judge: Integrity depends on a judge, so improving or replacing that component (and expanding human validation) is flagged as future work.
• Improving the difficult families: Resource allocation stayed at 17.5% Verified Success for Base, PPO, and PaCPO, and both scheduling families remain difficult, leaving clear targets for better methods.
• Understanding mixed per-scenario effects: Improvements and regressions occur within the same family, so knowing when paired credit helps versus hurts remains an open question.

Target Audience

Researchers and engineers working on LLM social agents, multi-turn dialogue, agent evaluation, and reinforcement learning from sparse terminal rewards. It is also useful for practitioners building coordination or negotiation assistants who need outcome-level correctness rather than fluent-sounding conversation, and for benchmark designers interested in contrast-set and paired-world evaluation designs.

Authors’ abstract

Effective social interaction requires more than fluent responses: agents must reach plans that fit the participants' needs and circumstances. Existing evaluations of social agents mainly rely on LLM judges that rate the conversation, without checking the resulting plan. We introduce HICO-Bench, a benchmark of 180 everyday coordination scenarios, such as resource allocation, scheduling, and authorization, in which two agents with private information converse and then independently record the plan they agreed on. A program checks Feasibility (the two records match and satisfy both participants' private conditions), and an LLM judge checks Integrity (the conversation supports the recorded plan); passing both is a Verified Success. Each scenario has two versions that share the same public context but differ in one private condition, so the correct plan differs between them. Across thirteen LLMs, 36% of conversations rated highly for goal completion fail Verified Success. To learn from outcomes that arrive only at the end of a conversation, we propose Paired-Credit Policy Optimization (PaCPO). After each turn, PaCPO asks the agent for a tentative plan, checks it in both versions of the scenario, and rewards the turns after which the tentative plans fit their respective versions. PaCPO raises Qwen3-8B's Verified Success from 20.4% to 28.3%.