Don't Just Fine-tune the Agent, Tune the Environment

AI summary

Overview

Research area: Large Language Model (LLM) agents — specifically reinforcement learning (RL) and supervised fine-tuning (SFT) for multi-turn, multi-tool use under scarce training data.

Technical level: Advanced. The paper assumes familiarity with RL for LLMs (policy optimization, reward design, POMDPs) and with function-calling agent benchmarks, though the core ideas are explained conceptually.

Scope: The paper proposes "Environment Tuning," a training paradigm that teaches LLM agents multi-turn tool use from only 400 problem instances by engineering the training environment (curriculum, feedback, rewards) rather than relying on pre-collected expert trajectories.

Paper metadata: arXiv:2510.10197v2 [cs.AI], 30 Jan 2026 (listed publication date 2025-10-11). Authors: Siyuan Lu, Zechuan Wang, Hongxuan Zhang, Qintong Wu, Leilei Gan, Chenyi Zhuang, Jinjie Gu, Tao Lin. Affiliations include Zhejiang University, Shanghai Innovation Institute, Westlake University, AWorld Team (Inclusion AI), and Nanjing University. License: CC BY-NC-ND 4.0. The provided content is truncated; some appendix details (formal analysis, augmentation-generation workflow, full hyperparameters) are referenced but not reproduced here.

What This Paper Is About

Training agents that can call tools across many turns is bottlenecked by data: high-quality multi-turn tool-use datasets are extremely scarce because they require labor-intensive human annotation, and the BFCL V3 multi-turn dataset contains only 800 samples. The two standard fixes both fall short — supervised fine-tuning on synthetic trajectories overfits and collapses out of distribution, while direct RL suffers a "cold-start" problem in complex environments and becomes unstable over long interaction chains. The paper's goal is to train a strong multi-turn tool-using agent from very little data while preserving both generalization and training stability, by tuning the environment the agent learns in rather than only the agent itself.

Key Contributions

1. A new learning paradigm for data-scarce environments. "Environment Tuning" lets agents acquire multi-turn tool-use capabilities directly from problem instances, with no expert demonstrations — shifting from trajectory-based imitation to environment-based exploration.
2. A practical four-stage curriculum with environment engineering. The authors design a curriculum (Stages 1–4) that combines actionable environment augmentation and fine-grained progress rewards to convert sparse feedback into rich learning signals for effective exploration.
3. Empirical validation under extreme data scarcity. Using only 400 training samples, the method works on both base models and SFT-tuned models: it lifts Qwen2.5-7B-Instruct from near-zero in-distribution performance, boosts the SFT-tuned watt-tool-8B to 54.34% (Table 1; the abstract cites 54.25%), and nearly doubles ToolACE-2's out-of-distribution score on ACEBench Agent (8.34% to 15.00% in Table 2; the abstract phrases this as 8.5% to 15.0%).
4. Evidence that environment-centric RL generalizes where SFT does not. The paper documents severe OOD collapse in SFT baselines and shows consistent OOD gains from Environment Tuning across BFCL V4, τ²-bench, and ACEBench Agent.

Main Findings

• Direct RL "cold-starts." When Qwen2.5-7B-Instruct was fine-tuned in a single-stage RL setup with 400 training instances, training collapsed within 70 steps, yielding only a 10% improvement in success rate (reported in Appendix E.2).
• Large in-distribution gains from scratch. Environment Tuning raises Qwen2.5-7B-Instruct on BFCL V3 multi-turn average from 7.00% to 36.92% (+29.92), and Llama-3.1-8B-Instruct from 5.48% to 28.25% (+22.77).
• It also improves already SFT-tuned models. ToolACE-2-Llama-3.1-8B rises from 37.99% to 47.18% (+9.19) and watt-tool-8B from 35.74% to 54.34% (+18.50), the latter exceeding most proprietary models including o3 (49.25% average) and GPT-4o (51.00% average) on the BFCL V3 multi-turn average.
• It beats direct RL (ToolRL) consistently. ToolRL gives only modest gains and is unstable on SFT-tuned models — it degrades ToolACE-2-Llama-3.1-8B by −4.24% — while Environment Tuning improves the same model by +9.19%.
• SFT baselines collapse out of distribution. On BFCL V4 Web Search, xLAM-2-8b-fc-r — which scores 70.50% on BFCL V3 — drops to just 5.00%; BitAgent-8B reaches 4.50%.
• Environment Tuning improves OOD generalization. On the same Web Search task, Llama-3.1-8B-Instruct goes from 1.00% to 15.00%, and ToolACE-2-Llama-3.1-8B from 9.00% to 14.00%. On BFCL V4 average, Llama-3.1-8B-Instruct improves from 8.46% to 16.53%.
• Gains on ACEBench Agent. ToolACE-2-Llama-3.1-8B rises from 8.34% to 15.00% with Environment Tuning, whereas ToolRL causes a drop to 6.65%.
• The curriculum is essential. Ablation on Qwen2.5-7B-Instruct: base 7.00%, direct GRPO 17.42%, +Stage 1 15.50%, +Stage 2 25.83%, +Stage 3 32.00%, +Stage 4 36.92% — a 19.50% increase over the direct GRPO baseline. The direct GRPO baseline used a combined reward of 0.9·R_P + 0.1·R_format on all 400 samples from the start.
• Environment augmentation matters most on hard splits. It produces more stable learning curves across all splits, with improvements of over 20% on the Missing Parameters and Missing Functions splits.
• Dense rewards matter as complexity grows. With a binary reward, Stage 2 (Base split) differences are subtle, but in Stage 3 the Missing Parameters and Missing Functions splits fail completely (performance close to zero); the Progress Reward also gives substantially more stable and effective learning on the Long Context split.
• Note on discrepancies: the abstract/introduction cite 54.25% for watt-tool-8B and 8.5% for ToolACE-2's ACEBench score, while Table 1 and Table 2 report 54.34% and 8.34% respectively.

Methodology in Plain English

The authors model multi-turn tool use as a Partially Observable Markov Decision Process: the agent receives a user request, optionally calls tools, then emits a final answer, after which the environment reveals the next request. Success requires passing every check in every turn, so a single failure fails the whole task.

Instead of feeding the agent pre-written successful trajectories, they shape how the environment behaves during training along three axes:

1. A structured curriculum of four stages.

   • Stage 1 ignores task success entirely and rewards only well-formed output: an XML-like format check and a tool-call validity check, gated by whether the agent attempts a tool call at all. The point is to stop "bad syntax" penalties from being confused with "bad strategy" penalties.
   • Stage 2 moves to the Base split and introduces the progress reward plus augmented feedback.
   • Stage 3 expands training to the Missing Parameters, Missing Functions, and Long-Context splits.
   • Stage 4 trains on the full dataset with the progress reward but turns off environment augmentation, forcing the agent to handle standard, uninformative errors — matching real evaluation conditions.
   • Stage transitions are decided by a rule: advance only when validation accuracy has plateaued and its gradient norm is stable, to guard against gradient explosion in long-horizon RL.

2. Actionable environment augmentation. Standard environments return cryptic errors ("No available route," "FileNotFoundError"). The augmented environment instead returns pedagogical hints — for example, "Invalid airport code[s]:..." (prompting the agent to find the code with another tool) or "Paths are not allowed. Specify only file/directory name..." (revealing an internal tool rule). Dependencies are learned through interaction rather than being handed over as pre-built dependency graphs. The augmentation-generation workflow is LLM-driven (Appendix B).

3. A fine-grained progress reward. Rather than one binary signal at the end of a trajectory, each turn is scored on two binary criteria — correctness of the resulting environment state and of the execution result — multiplied together, and the Progress Reward is the average over all turns. This distinguishes "nearly correct" from "completely wrong" and densifies the learning signal.

Training uses an adapted Group-Relative Policy Optimization (GRPO) with a decoupled clipping mechanism and a KL-divergence penalty. The comparison baseline is ToolRL, which applies GRPO directly with a reward of R_format + R_correct. Evaluation uses 400 samples (100 per split) for training from BFCL V3's 800-sample multi-turn set, with the other 400 held in for in-distribution testing; OOD tests use BFCL V4 web search and memory tracks, τ²-bench, and ACEBench Agent.

Why This Matters

Impact on research. The paper argues for a paradigm shift from supervised fine-tuning on static trajectories to dynamic, environment-based exploration. It offers a concrete counterexample to the assumption that more synthetic trajectories are the answer to data scarcity, and it suggests that feedback design and reward shaping — not dataset size — may be the binding constraint for long-horizon agent training.

Real-world applications (from the paper's framing and task structure):

• Travel planning agents that must resolve ambiguous requests, look up dependent values (such as airport codes), and chain multiple APIs.
• File system and software-engineering agents that must learn environment-specific protocols, such as not using full paths with commands like "rm."
• Cross-domain API orchestration — BFCL V3 spans 8 domains with 84 distinct tools requiring cross-domain calls.
• General tool-using assistants deployed where training data is scarce, where the model must ask for missing parameters or recognize that a required tool is unavailable.

Industry relevance. The method works on top of both base and already SFT-tuned models, which means teams with existing fine-tuned tool models can use it as an online refinement step rather than starting over. Its data efficiency (400 samples) and its OOD robustness target the practical failure mode of shipping agents that look good on a curated benchmark but degrade on real user traffic.

Future Directions

• Automating curriculum and feedback generation. The authors explicitly name automated mechanisms for curriculum and feedback generation as future work; the current augmentation workflow is LLM-driven but described in appendices and appears partly manual.
• Extending to more complex, multi-modal agentic scenarios. The conclusion flags this as a direct extension of Environment Tuning.
• Broadening evaluation beyond the current OOD set. The paper evaluates on BFCL V4, τ²-bench, and ACEBench Agent, but BFCL V3 training is confined to closed-domain API calls; whether the gains extend to fully open-domain settings is not established in the provided content.
• Open questions the paper raises but does not resolve in the visible text: whether the four-stage structure transfers to architectures beyond Qwen2.5-7B-Instruct and Llama-3.1-8B-Instruct, whether stage transitions can be triggered without the validation-plateau-plus-gradient-norm rule, and whether the augmented feedback risks teaching environment-specific shortcuts that Stage 4 only partially removes.

Target Audience

Researchers and engineers working on LLM agents, RL for language models, and tool-use / function-calling systems — particularly those facing scarce or expensive multi-turn training data. It is also relevant to practitioners building production agents who need evidence about generalization behavior under distribution shift, and to benchmark designers interested in how evaluation environments differ from training environments. Readers without RL background will need to consult the appendices and related work on GRPO and reward shaping to follow the training details.

Authors’ abstract

Large Language Model (LLM) agents show great promise for complex, multi-turn tool-use tasks, but their development is often hampered by the extreme scarcity of high-quality training data. Supervised fine-tuning (SFT) on synthetic data leads to overfitting, whereas standard reinforcement learning (RL) struggles with a critical cold-start problem and training instability. To address these challenges, we introduce $\textbf{Environment Tuning}$, a novel training paradigm that enables agents to learn complex behaviors directly from problem instances without relying on pre-collected expert trajectories. $\textbf{Environment Tuning}$ orchestrates this learning process through a structured curriculum, actionable environment augmentation that provides corrective feedback, and fine-grained progress rewards to ensure stable and efficient exploration. Using only 400 problem instances from Berkeley Function-Calling Leaderboard (BFCL) benchmark, our method not only achieves competitive in-distribution performance against strong baselines but also demonstrates superior out-of-distribution generalization, overcoming the performance collapse common to SFT-based approaches. Our work presents a paradigm shift from supervised fine-tuning on static trajectories to dynamic, environment-based exploration, paving the way for training more robust and data-efficient agents. The code is available at https://github.com/inclusionAI/AWorld-RL/tree/main/EnvTuning.