AgenticPay: A Multi-Agent LLM Negotiation System for Buyer-Seller Transactions

AI summary

Overview

Research area: Multi-agent LLM systems, natural-language negotiation, and benchmark design for economic interaction (agentic commerce).

Technical level: Intermediate. The paper formalizes negotiation as a stochastic language game with private reservation values, but its central artifact — a benchmark suite and scoring algorithm — is described in accessible terms.

Scope: This paper introduces AgenticPay, a benchmark and simulation framework for evaluating LLM agents that negotiate buyer–seller transactions through multi-round natural language dialogue across 111 tasks spanning bilateral bargaining to many-to-many markets.

What This Paper Is About

Existing benchmarks evaluate LLMs mainly on single-agent reasoning, tool use, or preference following, and simplify economic interaction into numeric auctions or short-horizon bargaining that ignore private reservation values, multi-round dialogue, heterogeneous products, and competition among multiple buyers and sellers. AgenticPay addresses this by building a controlled yet expressive testbed in which buyers and sellers hold private constraints and product-dependent valuations and must reach agreements through linguistic negotiation rather than numeric bidding alone. The paper's guiding question is how effectively current LLMs can function as autonomous negotiators in diverse market environments.

Key Contributions

1. A scalable negotiation framework and benchmark. AgenticPay supports over 110 tasks (111 total) ranging from bilateral bargaining to many-to-many markets, with dialogue-to-action grounding and welfare-oriented evaluation metrics. It supports deployment via vLLM, SGLang, and cloud-based LLM APIs.

2. A formalization of language-mediated negotiation. Buyer–seller negotiation is cast as a multi-agent game with private reservation values and dialogue-grounded economic outcomes, where a parser maps exchanged messages to structured actions such as price proposals.

3. A structured task suite and scenario library. The benchmark contains 8 multi-agent configurations composed of 31 basic tasks for core mechanics and 80 realistic tasks drawn from 10 business scenarios organized into four economic domains (Daily Life, Professional Services, Business Procurement, Financial Assets).

4. A benchmarking study of state-of-the-art LLMs. Proprietary and open-weight models are evaluated under a unified inference-only protocol, uncovering persistent limitations in long-horizon strategic reasoning and negotiation efficiency.

Main Findings

• Proprietary models dominate. Claude Opus 4.5 achieves the highest GlobalScore of 86.9, followed by Gemini-3-Flash (82.2) and GPT-5.2 (81.7), all with 100% deal rates and zero timeouts. Open-weight models lag substantially: Qwen3-14B reaches 63.9 GlobalScore with a 20.7% timeout rate, and Llama-3.1-8B reaches 32.5 with 48.6% of negotiations ending in timeout.

• Price overflow distinguishes reliability. Overflow rates — instances where agents propose prices outside acceptable bounds — are near zero for proprietary models but 10.8% for Llama-3.1-8B.

• Efficiency correlates with capability. Stronger models reach agreements faster (Claude Opus 4.5: 3.7 rounds; GPT-5.2: 3.8 rounds) while weaker models need far more turns (Llama-3.1-8B: 15.0 rounds) or fail to agree.

• Systematic buyer–seller asymmetry. All models score higher as sellers than as buyers (GPT-5.2: 81.1 SellerScore vs. 58.5 BuyerScore; Qwen3-14B: 58.9 vs. 47.6). In cross-play, the gap is most pronounced for Qwen3-14B (43.2 points) and persists in Claude Opus 4.5 (26.0 points), suggesting a fundamental bias in LLM negotiation behavior.

• More agents yields better outcomes. GlobalScore increases with buyer and seller multiplicity across most models, with gains ranging from 5 points (GPT-5.2) to over 14 points (Qwen3-14B), attributed to increased market liquidity and more opportunities to find compatible partners.

• Financial assets are the hardest domain. Financial Assets consistently yields the lowest GlobalScores; Gemini-3-Flash drops 20.2 points from Professional Services (88.3) to Financial Assets (68.1).

• Personality configurations matter. Using Claude Opus 4.5 in the 1B1P1S scenario, "Busy Professional" buyers consistently achieve lower GlobalScores, and aggressive sellers score highest with budget-conscious buyers (92.7) but lowest with busy professionals (44.1).

• Sequential vs. parallel modes. Proprietary models maintain consistent performance across both modes; open-weight models gain 4–5 GlobalScore points in parallel execution, though Llama-3.1-8B's overflow rate doubles (0.08 to 0.17).

• Timeouts stem from model limits, not task structure. Failures are uniformly distributed across task categories, with no configuration exceeding 22% of total failures.

• Near-misses reveal convergence deficiencies. Over 40% of failures for Qwen3-14B (43.5%) and Llama-3.1-8B (46.3%) occurred when the buyer–seller price gap was within 5 units — a gap bridgeable with a single concession.

Methodology in Plain English

The researchers built a simulated marketplace where LLM agents play buyers and sellers. Each agent receives public information — a product description, scenario metadata, and the negotiation protocol — plus private information: buyers get a maximum willingness-to-pay and sellers a minimum acceptable price. These reservation values are injected into the system prompt but excluded from the shared dialogue. Negotiation proceeds in alternating rounds up to a maximum number of turns (set to 20), with each party emitting a natural-language message containing an explicit price offer in a required format. A deal is reached when both parties propose the same price, and a transaction is valid only if the agreed price falls within the bargaining zone, defined as the buyer's max price minus the seller's min price.

Tasks vary market structure along three dimensions — number of buyers, number of sellers, and product set size — producing eight configurations from single-buyer-single-seller to full multi-agent markets, and two interaction modes (parallel, where an agent reasons over several negotiations at once, and sequential, where it decides whether to continue, switch, or commit). Outcomes are scored by an algorithm that normalizes each party's utility within the bargaining zone, computes a symmetric quality term that peaks when surplus is split equally, and applies a discount factor for slower deals. Parameters were configured as deal bonus 30, quality bonus 55, efficiency bonus 15, discount factor 0.99, and failure penalty 15. Five models were evaluated — GPT-5.2, Claude Opus 4.5, Gemini 3 Flash, Qwen3-14B, and Llama-3.1-8B — with deterministic decoding at temperature 0 and seed 0, a maximum generation length of 1024 tokens, and 4 NVIDIA A800 GPUs for open-source inference. Each task instance was executed once per model.

Why This Matters

Research impact. AgenticPay fills a gap in evaluating autonomous LLM-based negotiators by unifying ideas from economic theory, multi-agent systems, and language modeling. It provides a controlled testbed for studying multi-agent negotiation, economic alignment, and the co-evolution of language and strategy, and its modular environment/task/agent interfaces allow new market configurations to be added without altering the core protocol. The results establish that strong language generation alone is insufficient for effective economic negotiation.

Real-world applications:

• E-commerce, where agents bargain on behalf of shoppers or merchants over heterogeneous products
• Procurement, including SaaS software and raw materials purchasing across competing suppliers
• Service contracting, such as website development, commercial photography, and home renovation quotes
• Financial asset transactions, including luxury watches and business acquisition deals, which the benchmark shows are the hardest category for current models

Industry relevance. The framework supports deployment through vLLM, SGLang, and cloud APIs, making it usable in production-adjacent pipelines. The paper's impact statement flags a concern that automated bargaining systems may disadvantage less sophisticated human counterparts, and encourages responsible deployment with appropriate safeguards and transparency.

Future Directions

• Investigating the buyer disadvantage. The paper reports a universal buyer disadvantage in cross-play but states that the asymmetry "warrants further investigation in future work," leaving the cause unresolved.
• Improving the "last mile" of convergence. Open-weight models approach the bargaining zone but fail to close small price gaps, pointing to a need for better strategic patience and timely concession behavior.
• Scaling to new market configurations. The modular environment, task, and agent interfaces are designed for systematic expansion to new market structures without changing the core protocol.
• Extending economic alignment work. The authors position AgenticPay as a foundation for research on agentic commerce and the co-evolution of language and strategy in autonomous agents, areas the paper leaves open.

Target Audience

Researchers and practitioners working on multi-agent LLM systems, autonomous negotiation agents, and economic alignment; benchmark designers evaluating LLM strategic reasoning; and industry teams building or auditing automated bargaining, procurement, and commerce agents.

Authors’ abstract

Large language model (LLM)-based agents are increasingly expected to negotiate, coordinate, and transact autonomously, yet existing benchmarks lack principled settings for evaluating language-mediated economic interaction among multiple agents. We introduce AgenticPay, a benchmark and simulation framework for multi-agent buyer-seller negotiation driven by natural language. AgenticPay models markets in which buyers and sellers possess private constraints and product-dependent valuations, and must reach agreements through multi-round linguistic negotiation rather than numeric bidding alone. The framework supports a diverse suite of over 110 tasks ranging from bilateral bargaining to many-to-many markets, with structured action extraction and metrics for feasibility, efficiency, and welfare. Benchmarking state-of-the-art proprietary and open-weight LLMs reveals substantial gaps in negotiation performance and highlights challenges in long-horizon strategic reasoning, establishing AgenticPay as a foundation for studying agentic commerce and language-based market interaction. Code and dataset are available at the link: https://github.com/SafeRL-Lab/AgenticPay.