[Re] Benchmarking LLM Capabilities in Negotiation through Scoreable Games

AI summary

Overview

Research area: Machine learning / large language model evaluation, specifically multi-agent negotiation benchmarks.

Technical level: Intermediate. Readers should be comfortable with LLM benchmarking terminology, prompt ablations, and utility-based evaluation metrics, but the paper is a reproduction study rather than a new model or method.

Scope in one sentence: This paper reproduces and stress-tests the Scoreable Games negotiation benchmark of Abdelnabi et al. (2024) across a wider set of models, new metrics, and code fixes, and finds that its central comparison claim is not well supported.

What This Paper Is About

Benchmarks that measure how well LLMs negotiate and cooperate are still rare, and one widely cited candidate — the Scoreable Games framework from Abdelnabi et al. (2024) — claims to let users objectively compare negotiation skill between models. This paper asks whether that claim actually holds by re-running the benchmark on many more models, adding fairness metrics, and fixing bugs in the released code. The authors conclude that the benchmark is genuinely complex but that its model comparisons are ambiguous, and that several parts of the experimental setup and game design are less general than claimed.

Key Contributions

1. A reproduction study that runs the benchmark on a much broader set of models than the original, including smaller and open-source models and closed-source architectures (GPT-4o mini and GPT-4o), exposing gaps in how weaker models are evaluated and replicating the original adversarial-behavior findings.
2. Identification of unfair and unstable comparisons in the original setup, caused by inconsistent model performance across games and high sensitivity to ablation configuration choices.
3. Discovery and correction of technical limitations in the original open-source codebase, most notably leakage detection that was confounded by formatting failures, an unconditional 10-unit relaxation of p₁'s threshold in the evaluation script, and a bug where a non-final deal from p₁ could be counted as the final deal.
4. Supplementary evaluation metrics (Utilitarian, Egalitarian, and Nash Social Welfare) plus new game-diversity statistics (Sparsity, Intersection over Union, Deal Space) that make fairness and diversity checkable, along with evidence that the framework's built-in prompt generation lacks game diversity.

Main Findings

• Base game is close to solved by some models. On the base game, DeepSeek-R1-Distill-Qwen-32B (int8) reached 95% on 5-way, 55% on 6-way, 95% Any, with 2.50% wrong deals and 0.00% leaked; DeepSeek-R1-Distill-Llama-70B (int4) reached 90%, 65%, 90%, with 0.19% wrong and 0.00% leaked. GPT-4o mini reached 55%, 5%, 90%, with 2.69% wrong and 0.58% leaked; GPT-4o reached 75%, 10%, 85%, with 0.58% wrong and 0.00% leaked.

• Quantization did not prevent strong performance. Despite quantization (int4/int8) applied with bitsandbytes to fit computational constraints, large models outperformed smaller non-quantized counterparts — for example Mistral-Small (int8) at 80% 5-way and Qwen2.5-72B (int4) at 85% 5-way, versus Ministral-8B at 25% and Phi-3.5-mini at 10%.

• Leakage metrics had far greater variance than originally reported. With the leakage bug fixed and smaller models included, the Leaked column varies widely — Llama-3-8B at 69.42%, Qwen2.5-7B at 44.62%, Phi-3.5-mini at 40.77% — whereas models that follow formatting, such as GPT-4o mini, Qwen2.5-72B and Mistral-Small, showed no leakage.

• Ablation settings change the winner. In the 16-configuration ablation study run on GPT-4o mini and Qwen-72B across four factors (previous deals, other preferences, candidates, planning), performance varied sharply. With all four factors ablated, GPT-4o mini scored 65 / 95 (5-way / Any); with the original configuration it scored 60 / 90; one configuration dropped it to 5% 5-way.

• Game difficulty is model-dependent. Across base, base_rewritten, game 1, game 2, and game 3, model rankings shift. Mistral-Small scored 25 on game 1 and 40 on game 2 but 0 on game 3; Qwen2.5-72B scored 80 on game 3 but 30 on game 2; GPT-4o mini scored 55 on game 3 but 15 on game 2. GPT-4 and Mistral-Small found game 1 hardest.

• Negotiation stability differs between weak and strong models. For the poor-performing Phi-3.5-mini, Utilitarian Social Welfare declined over time with a slope of -5.8 and variance of 243. For the high-performing DeepSeek-R1-Qwen-32, variance was 58.2 with a positive slope of 0.877 and a correlation coefficient of 0.687. Egalitarian and Nash metrics largely mimicked the Utilitarian pattern, indicating no single agent was exclusively dominating.

• The provided games show limited diversity in structure. Across the five games, |Π_acc|/|Π| is 55/720 or 57/720, |Π_hard|/|Π| ranges from 12/720 to 35/720, sparsity ranges from 23.68% to 42.98%, and Intersection over Union ranges from about 18.75% to 29.77%. The identical acceptable-deal fractions for base and base_rewritten are by design and do not support a diversity claim.

• New settings are biased toward construction. All five original games and all ten games newly generated with the original prompt were based on a construction project. Removing words such as "project" and "resources" produced genuinely different settings, including military spending budgets and conference planning with floor allocation and catering.

• Adjustability works in the expected direction. Varying thresholds and player count on the base game with GPT-4o mini confirmed the original findings: performance increases as per-player minimum thresholds decrease, and more agreements are reached with fewer players.

• Behavioral effects reproduce, with one surprise. Cooperative, greedy, and adversarial trends replicated. When p₁ is greedy, 5/6-way performance drops to 45 for GPT-4o mini, 50 for Qwen2.5-72B, and 27 for GPT-4. When all players are greedy, it falls to 50, 15, and 26 respectively. Unexpectedly, when the Environmental League is greedy, GPT-4o mini and Qwen2.5-72B show an increase in 5/6-way deals (80 and 95 for the P_const variant).

• Stronger models follow behavioral prompts more closely. In the p_i greedy setting, the last-round distance between the greedy agent and aggregated USW was 12.86 for GPT-4o mini versus 24.85 for Qwen2.5-72B-Instruct. In the untargeted adversarial setting, the distance between the would-be targeted agent and aggregated USW was -33.91 for GPT-4o mini versus only -14.03 for Qwen2.5-72B-Instruct.

• The original rule-based baseline could not be replicated. The original baseline scored 37 (5/6-way) on base, 46 on game 1, 62 on game 2, 79 on game 3; the authors' alternative random-sequence baseline scored 63, 79, 68, and 83 respectively, offering a reproducible alternative.

Methodology in Plain English

The authors take an existing open-source benchmark where six simulated parties negotiate over five issues (for example, how tall to build a tower) for 24 rounds, with p₁ making the final proposal and p₁ and p₂ holding veto power. A deal counts if it exceeds the threshold for at least five of the six players, and a stricter "6-way" category requires all six to accept. Each model is scored on a base game plus four additional game variants, and the headline metric is the percentage of deals achieved over 20 game iterations.

The team rebuilt this setup in their own repository, running models from Hugging Face in pre-trained form without fine-tuning and quantizing larger models with bitsandbytes to fit available hardware. They ran five families of experiments: base-game reproduction across many models, an expanded ablation study covering all 16 combinations of four prompt factors on two models, a cross-game comparison of Mistral-Small, Qwen2.5-72B, and GPT-4o mini, new social-welfare metrics tracked over negotiation rounds, and a rule-based baseline of their own. They also tested game adjustability by querying the original generation prompt ten times and by varying thresholds and player counts, and they replicated the greedy and adversarial prompting experiments.

Along the way they rewrote leakage detection: instead of relying on a GPT-4 judge that was triggered whenever a model omitted a required tag, they separated formatting failures from genuine disclosure of private information, flagged illegal keywords such as <plan> or <scratchpad>, terminated and logged runs that produced no valid public answer, and filtered subsequent experiments to models that showed no leakage.

Why This Matters

For research, the paper shows that a benchmark's complexity does not guarantee that its comparisons are meaningful. If a configuration choice or game selection can flip which model appears better, then reported rankings need to state those choices explicitly. The finding that leakage detection was largely an artifact of parsing failures is a caution for any benchmark that uses an LLM as a judge.

Real-world applications include:

• Automated contract and procurement negotiation, where buyers and suppliers want to know whether an LLM agent will hold its position or concede.
• Multi-agent coordination systems, such as logistics or scheduling agents that must divide limited resources among several stakeholders.
• Human-AI teaming, where an AI negotiator must represent a user's interests without being exploited or becoming adversarial.
• Diplomacy and consensus-building simulations, where a model's willingness to compromise and its stability over long dialogues matter as much as its final answer.

Industry relevance is direct: teams choosing which model to deploy for a negotiation or bargaining agent need comparable numbers, and this paper shows that the most-cited negotiation benchmark produces ambiguous rankings unless the ablation settings, game choice, and leakage handling are all specified.

Future Directions

• Generate a larger and more varied set of games that spans a wider range of sparsity, IoU, and deal-space statistics, and test whether performance saturates (becoming trivially easy or impossible) at the extremes, which would indicate redundancy.
• Develop leakage detection that reliably distinguishes a genuine strategic disclosure from a formatting failure, since the original LLM-judge approach conflated the two.
• Standardize how ablation configurations are reported, because the study shows a single configuration can reverse a model comparison; there is no evidence that one model's optimal configuration is optimal for another.
• Investigate the unexplained result that a single greedy non-first-mover (Environmental League) raised 5/6-way deal rates, and examine why difficulty rankings across games do not align consistently across models.

Target Audience

This paper is most useful to benchmark designers and evaluation researchers, to practitioners selecting LLMs for multi-agent or negotiation deployments, and to readers of the original Abdelnabi et al. (2024) work who want to know which of its results hold up, which were confounded by code issues, and where the framework's limits lie.

Authors’ abstract

Large Language Models (LLMs) demonstrate significant potential in multi-agent negotiation tasks, yet evaluation in this domain remains challenging due to a lack of robust and generalizable benchmarks. Abdelnabi et al. (2024) introduce a negotiation benchmark based on Scoreable Games, with the aim of developing a highly complex and realistic evaluation framework for LLMs. Our work investigates the reproducibility of claims in their benchmark, and provides a deeper understanding of its usability and generalizability. We replicate the original experiments on additional models, and introduce additional metrics to verify negotiation quality and evenness of evaluation. Our findings reveal that while the benchmark is indeed complex, model comparison is ambiguous, raising questions about its objectivity. Furthermore, we identify limitations in the experimental setup, particularly in information leakage detection and thoroughness of the ablation study. By examining and analyzing the behavior of a wider range of models on an extended version of the benchmark, we reveal insights that provide additional context to potential users. Our results highlight the importance of context in model-comparative evaluations.