MADA-RL: Multi-Agent Debate-Aware Reinforcement Learning for Parameter-Efficient Reasoning in Compact Models

AI summary

Overview

Research area: Large language model reasoning, reinforcement learning post-training, multi-agent debate, and parameter-efficient fine-tuning (LoRA).

Technical level: Advanced. The paper assumes familiarity with reinforcement learning policy optimization (GRPO), advantage functions, LoRA adapters, and multi-agent debate protocols.

Scope: A post-training framework called MADA-RL that specializes a 1.5B-parameter compact model into generator and critic agents trained with a counterfactual critic advantage, evaluated on five mathematical reasoning benchmarks.

What This Paper Is About

Large language models reason well but are expensive to train, and this cost is especially acute for compact models (the paper defines these as ≤4B parameters) trained under limited budgets. Existing options force a choice: reinforcement learning fine-tuning, which is costly and unstable and usually requires updating the full model, or test-time deliberation such as multi-agent debate, which is under-used as a training signal. The paper's goal is to combine them cheaply: post-train a compact model into specialized generator and critic roles using only LoRA adapters, shaping the critics with a new learning signal that rewards them specifically for beating the generator ensemble rather than for merely producing correct answers.

Key Contributions

1. A counterfactual critic advantage. A dynamic, role-conditioned baseline that redefines the critic's advantage as its reward minus the generator ensemble's per-instance accuracy, multiplied by two so it scales with the correctness component of the reward. This sharpens credit assignment without value models, replay buffers, or external verifiers.

2. The MADA-RL post-training method. A parameter-efficient, two-stage framework that applies this signal to compact LLMs via GRPO plus LoRA adapters, together with a suite of LoRA-fine-tuned generator and critic agents built on DeepSeek-R1-Distill-Qwen-1.5B.

3. A controlled analysis of where the gains come from. The paper isolates the source of improvement to learned corrective behaviour in critics rather than to additional test-time deliberation alone.

4. A common-protocol empirical study. An evaluation of existing 1.5B fine-tunings placed under a shared role-specialized protocol, plus an accounting of the training-parameter and inference-token trade-offs involved.

Main Findings

• Accuracy gain on the base model. Across five mathematical reasoning benchmarks, MADA-RL raises the accuracy of DeepSeek-R1-Distill-Qwen-1.5B from 39.9% to 41.9% (+2.0 points, p < 0.001) using 16 times fewer trainable parameters than fully fine-tuned baselines, placing it on the accuracy–trainable-parameter Pareto front.

• It does not beat the strongest baselines. DeepScaleR reaches 44.3 (0.3) and Still-3 reaches 43.1 (0.4), both trained on substantially larger datasets with full-model fine-tuning. MADA-RL remains significantly below both (p < 0.001 against DeepScaleR, p = 0.020 against Still-3). DeepScaleR is significantly ahead of Still-3 (p = 0.025).

• Community fine-tunes are statistically flat. Open-RS1 (40.4), Open-RS2 (39.4), Open-RS3 (40.4), and Tina (39.8) are statistically indistinguishable from the base DeepSeek-R1 model (all p > 0.20), while MADA-RL significantly outperforms each of them.

• At an equal LoRA budget, MADA-RL matches or exceeds re-trained baselines. When DeepScaleR and Still-3 are re-trained with LoRA on the same method and data, their accuracy drops to 40.5 and 41.3 respectively. DeepScaleR-LoRA falls significantly below MADA-RL (p = 0.015); Still-3-LoRA is statistically indistinguishable from it (p = 0.280).

• The strongest critic improvement rate. MADA-RL achieves a critic improvement rate of 19.6%, the highest of any model evaluated, meaning its critics correct the generators' consensus more often than any other model placed in the critic role. This is presented as the mechanism-level signature of the counterfactual advantage.

• Training-parameter efficiency. Gain per parameter, scaled by 10^12, is +182 for MADA-RL, versus +96 for Still-3-LoRA, +42 for DeepScaleR-LoRA, +25 for DeepScaleR, +18 for Still-3, and −17 for Tina.

• A real inference-time price. MADA-RL generates 33,818 tokens per question under the full three-generator, two-round protocol — among the highest of any model and roughly an order of magnitude more than a single forward pass. The paper states plainly that any efficiency claim concerns training parameters, not inference compute.

• More rounds help, with diminishing returns. One round drops to 36.7 (−5.2, p < 0.001); three rounds rise to 43.3 (+1.4, p = 0.016).

• More agents help. One generator plus one critic falls to 39.3 (−2.6, p < 0.001); two generators plus two critics falls to 41.0 (−0.9, p = 0.050).

• Diversity and the counterfactual advantage are directionally supported but not significant alone. A homogeneous configuration is 0.6 points below (p = 0.166); removing the counterfactual component costs 0.8 points (p = 0.095). The paper states it does not rest the case for the counterfactual advantage on this ablation alone, pointing instead to the critic improvement rate.

• MADA-RL depends most on debate. Removing debate causes all models to drop, but the drop is largest for MADA-RL (−5.2 points), suggesting the training procedure encourages specialization.

Methodology in Plain English

The researchers take one compact base model, DeepSeek-R1-Distill-Qwen-1.5B, and train six separate agents from it: three generators and three critics. Each agent gets its own LoRA adapter, so the base weights stay frozen and only a small number of parameters change. The base model contains approximately 1.78 × 10^9 parameters; each MADA-RL agent trains 18.5 × 10^6 of them, for a total of 110 × 10^6.

The training data is the Still-3-Preview-RL-Data dataset, which contains 29,925 mathematical reasoning problems. It is split into three disjoint subsets of 7,500 problems each — one per generator — plus a shared validation set of 750 problems for checkpoint selection.

Stage one trains the generators independently using GRPO. Each generator gets a reward that is a weighted sum of a binary accuracy reward (checked by converting both prediction and ground truth to symbolic form and testing equivalence) and a length reward that penalizes unnecessarily long completions, with a 2:1 weighting in favor of accuracy.

Stage two trains the critics. The researchers run the trained generators on the training questions and build a new dataset in which each example contains the original question, the generators' responses, and the per-instance average accuracy of the generators. The critic then gets a modified advantage: instead of comparing its reward to the average reward of its own sampled outputs, it is compared against two times the generator ensemble's accuracy. A positive value means the critic beat the generator consensus; a negative value means it did worse.

At inference, the agents run a simple protocol the paper calls multi-agent debate: all three generators answer the question in parallel, then the three critics each see the question plus the previous round's answers and produce a revised answer. Accuracy is computed only from the final round. The protocol deliberately omits the summarization step proposed in the prior work it builds on, to keep the method simple and memory efficient.

Statistical comparisons use Welch's two-sample t-test on per-seed average accuracy across the five benchmarks, with ten seeds per model. The benchmarks are Math-500, AIME 2024, AIME 2025, AMC-23, and Minerva-Math.

Why This Matters

Impact on research. The paper contributes a credit-assignment mechanism rather than a new debate protocol, and it makes an unusual move by reporting where its method loses. It shows that much of the apparent advantage of full fine-tuning on strong reasoning baselines shrinks or disappears once those baselines are re-trained under an equal LoRA budget, which reframes the comparison toward accuracy gain per unit of training cost. The counterfactual advantage is also architecturally cheap, requiring no value model, replay buffer, or external verifier.

Real-world applications:

• Edge and on-device reasoning assistants, where a ≤4B model must be fine-tuned and deployed within tight memory budgets.
• Domain-specific verification workflows — such as code review, compliance checking, or document validation — where a critic role trained to catch another model's errors is more valuable than a second general-purpose model.
• Cost-constrained enterprise fine-tuning, where organizations want reasoning improvements without paying for full-model updates on 1.78 × 10^9 parameters.
• Research tooling for small-model reasoning, since the paper evaluates community 1.5B fine-tunings under one shared protocol, providing comparable numbers.

Industry relevance. The authors are affiliated with the Bosch Center for Artificial Intelligence, LMU Munich, the Munich Center for Machine Learning, the University of Oslo, and the University of Cambridge, indicating industrial interest in making compact reasoning models trainable at low cost. The reported inference cost of 33,818 tokens per question is a practical constraint the paper flags as the method's main limitation for latency-sensitive deployment.

Future Directions

• Scaling the agent ensemble. The paper reports that three agents beat smaller ensembles and suggests further scaling could yield gains, but notes that adding agents would require retraining the full pipeline from scratch because of the disjoint data partitioning scheme.
• A better-controlled test of the counterfactual advantage. The accuracy ablation is 0.8 points in the expected direction but not significant at ten seeds (p = 0.095), so the paper suggests larger sample sizes would be needed to establish its marginal contribution to end-task accuracy.
• Reducing inference cost. The 33,818 tokens per question and the six agent calls per question are the method's main practical limitation; the paper leaves efficiency beyond training-parameter count unaddressed.
• Reintroducing summarization. The authors omit the summarization step from prior work to keep the method memory efficient, and note that incorporating a summarizer could be valuable in scenarios with a larger number of agents.

Target Audience

Researchers and engineers working on parameter-efficient fine-tuning, reinforcement learning post-training for language models, and multi-agent reasoning systems. It is most useful to those with a working knowledge of GRPO-style policy optimization and advantage estimation, and to practitioners deciding whether to spend a training budget on full fine-tuning versus LoRA-based role specialization for compact models. Readers primarily interested in inference-time efficiency will find the paper less applicable, since the reported token cost is high.

Authors’ abstract

Large language models achieve strong reasoning performance, but often at prohibitive training cost - a challenge that is especially acute for compact models ($\leq 4 \, \mathrm{B}$ parameters) trained under limited budgets. We introduce MADA-RL, a post-training framework that specializes compact models into generator and critic roles and trains them with a debate-aware learning signal, fine-tuning only a small subset of parameters via LoRA adapters. Our central contribution is a counterfactual critic advantage: a dynamic, role-conditioned baseline that redefines the critic's advantage as its reward minus the generator ensemble's per-instance accuracy. This explicitly optimizes critics to improve over generator consensus rather than to merely reproduce a correct answer, yielding more targeted credit assignment than static mean-reward normalization. At deployment, the specialized agents are composed in a lightweight multi-round protocol. Across five mathematical reasoning benchmarks, MADA-RL raises the accuracy of the DeepSeek-R1-Distill-Qwen-1.5B model from $39.9 \, \%$ to $41.9 \, \%$ ($+2.0$ points, $p &lt; 0.001$) using $16$ times fewer trainable parameters than fully fine-tuned baselines, placing it on the accuracy-trainable-parameter Pareto front. It approaches, but does not surpass, the strongest baselines (DeepScaleR, STILL-3), which are trained on substantially larger datasets; we analyse this gap and the associated inference-time cost directly. A controlled study isolates the source of MADA-RL's gains: the counterfactual advantage produces the highest critic improvement rate of any model evaluated, indicating that trained critics learn to correct generator errors rather than to imitate them.