Do Small Language Models Learn to Negotiate? A Controlled Scaling Study of RL-Trained Sellers

AI summary

Overview

Research area: Reinforcement learning (RL) for LLM agents, specifically multi-turn strategic negotiation with small language models; sits at the intersection of agentic AI, post-training with verifiable rewards, and scaling behaviour.

Technical level: Advanced. The paper assumes familiarity with GRPO-style RL post-training, LoRA-free full-parameter policies, KL penalties, bootstrap confidence intervals, and bid/utility formalisms from the bargaining literature. The prose is readable, but the experimental logic (paired within-scenario contrasts, Holm correction, seed-cluster bootstrapping) is research-grade.

Scope in one sentence: A single controlled study asking whether GRPO on a programmatic utility reward can turn four Gemma 4 checkpoints (2.3B to 31B effective parameters) into competent bilateral negotiators, and how that answer depends on the learning rate.

What This Paper Is About

Small language models are much cheaper to run than frontier models, and companies want agents that can handle sales, procurement and contract talks on their behalf. The problem is that prompting alone has not produced strong LLM negotiators, so the authors instead train sellers with reinforcement learning against a frozen LLM buyer and measure whether the trained model actually captures more value than its untrained base.

The core goal is to test three things at once: whether RL improves each model size over its base, how that improvement changes as models get bigger, and whether the apparent "small models can't learn this" conclusion is really just a learning-rate artefact.

Key Contributions

1. A shared-recipe scaling comparison. Four Gemma 4 checkpoints (2.3B, 4.5B, 12B, 31B effective parameters) trained under one GRPO recipe on a strategic multi-agent task. The paired gain over base runs from +0.001 (not significant) at 2.3B to +0.078 at 31B. Each arm is a single run, and size co-varies with architecture and training FLOPs, so no scaling law is fitted.
2. Evidence that the size pattern depends on the learning rate. At early checkpoints, tripling the learning rate improves on the shared-rate arm at every size, with the largest point estimate at 4.5B (+0.081). Against a shared-rate run trained for 104 steps (versus 56 for the tripled arm), the tripled rate stays ahead at 2.3B and 4.5B, while the difference is not detectable at 12B and 31B.
3. A demonstration that pooled score can hide opponent-specific gains. A 2.3B arm at ten times the shared rate raises pooled score over base, but its gain concentrates on the evaluation buyer that shares a model family with the training pool, accompanied by formatting degeneration.
4. An evaluation protocol built on matched scenario cells, two buyers excluded from training (one sharing the training pool's model family, one not), a held-out domain, seed-cluster bootstrap intervals, and a deviation log.

Main Findings

• Gains rise with size under a shared learning rate. At 104 steps and 10⁻⁶, the paired gain over base is +0.001 [-0.019, +0.021] at 2.3B, +0.023 [+0.011, +0.035] at 4.5B, +0.063 [+0.052, +0.075] at 12B and +0.078 [+0.064, +0.092] at 31B. The 4.5B, 12B and 31B gains survive Holm correction at α = 0.05; the 2.3B gain does not.

• Half the size trend coincides with an architecture change. The gain difference is +0.022 [-0.002, +0.046] within the E-series pair (2.3B, 4.5B), +0.040 [+0.024, +0.056] across the E-series/conventional boundary, and +0.015 [+0.001, +0.029] within the conventional pair (12B, 31B). The paper therefore does not fit a scaling law, and notes an alternative reading: conventional Gemma 4 checkpoints respond to GRPO and the E-series does not.

• Tripling the learning rate helps at every size. Against base, the 3×10⁻⁶ arm gains +0.037 [+0.017, +0.057] at 2.3B, +0.089 [+0.075, +0.102] at 4.5B, +0.076 [+0.059, +0.093] at 12B and +0.080 [+0.063, +0.098] at 31B. Against the shared-rate arm at early checkpoints the tripled rate wins at every size: +0.032, +0.081, +0.051 and +0.058 respectively.

• The rate advantage persists at small sizes after longer training, but not at large ones. Against the 104-step shared-rate arm, the tripled-rate arm stays ahead at 2.3B (+0.036 [+0.018, +0.054]) and 4.5B (+0.065 [+0.053, +0.079]), while at 12B (+0.013 [-0.002, +0.028]) and 31B (+0.003 [-0.014, +0.020]) the intervals include zero. These two arms are not compute-matched: 56 steps versus 104.

• Frontier seller comparisons are exploratory and partly attributable to the base model. The 12B tripled-rate seller (0.601) scores above Claude Sonnet 5 by +0.086 [+0.068, +0.104] and above GPT-5.6-Terra by +0.097 [+0.078, +0.116], but the untrained 12B base (0.525) already scores as high as both reference sellers (0.515 and 0.504). The 4.5B tripled-rate seller shows no detectable difference from either (+0.003 [-0.013, +0.019] and +0.014 [-0.003, +0.031]) and fits on one 48 GB accelerator.

• The 10× learning-rate arm exposes a pooling trap. At 2.3B the 10× arm scores +0.050 [+0.028, +0.073] over base, but its buyer split (score against Terra minus score against Sonnet 5) is +0.175, versus +0.032 to +0.079 for the other 2.3B arms. 91.6% of its seller turns leak a fragment of the JSON action into the prose the buyer reads, and 53.4% begin with "I appreciate" (17.5% for the 3× arm).

• The size ordering does not replicate on the held-out domain. On the withheld car-sale domain (n = 128), only two of eight gains exclude zero: the 104-step 12B arm (+0.032 [+0.003, +0.063]) and the tripled-rate 2.3B arm (+0.059 [+0.023, +0.095]). The 2.3B arms score above the 4.5B arms under both rates, so the authors do not claim the gain pattern replicates.

• Trained large sellers open differently. Base sellers name at least one term on turn 1 in 0.4% to 26.0% of episodes; the tripled-rate 12B and 31B sellers do so in 76.6% and 99.7% of episodes, naming 6.6 and 6.2 terms on average (74% and 69% of the domain's). Accepted as they stand, those openings would score 0.741 and 0.757 for the seller, and the resulting deals score 0.128 and 0.135 lower. The 4.5B tripled-rate seller, which has the largest pooled gain (+0.089), opens with terms in only 20.7%. These descriptors were chosen after the results were known.

• Where the value comes from. At 12B and 31B under both rates, distributive terms account for 75% to 81% of the seller's gain on the scenarios both arms closed. Under the shared rate, 88% and 93% of that gain comes out of the buyer's share and joint welfare does not move detectably (+0.008 and +0.005). Under the tripled rate the buyer's loss is 55% and 48% of the gain and welfare rises (+0.035 [+0.012, +0.057] and +0.042 [+0.023, +0.060]).

• Compute confound is acknowledged, not resolved. At 104 steps the 31B arm used 6–13× the 2.3B arm's training FLOPs, so the shared-rate pattern could also be a compute artefact. At serving time the 12B tripled-rate seller scores 0.601 on one 48 GB device, against the 31B base's 0.542 on four; throughput, latency and cost are not benchmarked.

• KL did not track guard failures. In these runs the logged KL to base did not predict which arms failed the four prespecified guards: the 31B tripled-rate arm has the largest step-56 KL (0.437) and passes all four, while the 10× arm at a KL of 0.313 fails three.

Methodology in Plain English

Authors’ abstract

LLM agents are starting to own the full customer experience. Soon, LLMs may be selling and buying on behalf of companies and customers respectively. Small models are more cost-efficient at scale, but can reinforcement learning train them into competent sellers? We train four Gemma 4 checkpoints (2.3B to 31B effective parameters) with GRPO on a programmatic utility reward for bilateral multi-issue bargaining, and evaluate every arm on the same 1,152 negotiations against two frontier buyers it never saw in training. With the same learning rate ($10^{-6}$) for every size, the gain of the RL model over its base rises from $+0.001$ at 2.3B to $+0.078$ at 31B. Each size was trained once and the two smallest checkpoints use a different architecture, so we fit no scaling law. Tripling the learning rate, with the same or fewer training steps, improves on the shared rate at every size by $+0.032$ (2.3B) to $+0.081$ (4.5B). In exploratory comparisons with two frontier models run as sellers, the 12B seller trained at the tripled rate scores above both, though its untrained base already scores as high as they do. The 4.5B seller at that rate shows no detectable difference from either and fits on one 48 GB GPU. A further 2.3B arm at ten times the shared rate raises pooled score, but its gain concentrates on the evaluation buyer that shares a model family with the training pool. These results suggest tuning the learning rate before concluding that a small model cannot learn to negotiate, and testing against buyers from more than one model family.