DeepAgent: A General Reasoning Agent with Scalable Toolsets

AI summary

Overview

Research area: Large reasoning models (LRMs) and LLM-powered autonomous agents, specifically tool-augmented reasoning, dynamic tool retrieval, agent memory management, and reinforcement learning for tool use.

Technical level: Advanced. The paper assumes familiarity with chain-of-thought reasoning models, reinforcement learning objectives (GRPO-style policy optimization), dense retrieval, and agent frameworks like ReAct.

Scope: The paper introduces DeepAgent, a framework that folds reasoning, dynamic tool discovery, tool execution, and memory compression into one continuous reasoning stream, trained end-to-end with a new RL method called ToolPO.

What This Paper Is About

Most LLM agents follow rigid, human-designed workflows such as "Reason-Act-Observe" loops, and they rely on a small, fixed set of pre-selected tools (search, browsing, code execution). This limits them on real-world tasks that need thousands of heterogeneous APIs discovered on the fly. DeepAgent addresses this by letting a large reasoning model autonomously think, search for relevant tools, invoke them, and compress its own interaction history — all inside a single, coherent reasoning process — and by training that behavior with reinforcement learning that rewards both correct intermediate tool calls and final task success.

Key Contributions

1. Unified autonomous agentic reasoning framework. DeepAgent is presented as the first framework in which a reasoning model autonomously thinks, discovers tools, and executes actions within one continuous reasoning process, rather than a predefined workflow with a fixed toolset. It can operate over toolsets ranging from tens to over ten thousand tools.

2. Autonomous memory folding with a brain-inspired schema. The agent can trigger a memory fold at any point, compressing its history into three structured components — episodic memory (task-level log), working memory (current sub-goal, obstacles, near-term plans), and tool memory (which tools were used, how, and how well). This lets the agent "take a breath" and abandon failed exploration paths without losing critical detail.

3. ToolPO, an end-to-end RL training method for general tool use. ToolPO uses an LLM-based tool simulator instead of unstable real-world APIs, and applies tool-call advantage attribution — assigning fine-grained credit only to the tokens that constitute tool calls and memory folds, on top of a global task-success reward.

4. Broad empirical validation. Evaluation across eight benchmarks (ToolBench, API-Bank, TMDB, Spotify, ToolHop, ALFWorld, WebShop, GAIA, HLE) in both labeled-tool and open-set retrieval settings, plus ablations, scaling analyses, and cross-backbone generalization tests.

Main Findings

• End-to-end reasoning beats workflow-based agents. On labeled-tool tasks, DeepAgent-32B-RL reaches 89.0% on TMDB and 75.4% on Spotify, versus best 32B baselines of 55.0% and 52.6%. This holds even against much larger models such as GPT-4o and DeepSeek-R1 on several benchmarks.

• Open-set tool discovery is where the gains are largest. With retrieval from full toolsets, DeepAgent-32B-RL scores 64.0% on ToolBench (16k+ APIs) and 40.6% on ToolHop (3.9k tools), versus best baselines of 54.0% and 29.0%. Gains scale with toolset size.

• ToolPO training consistently improves the base model. RL training raises ToolBench success by up to 6.0 percentage points, Spotify (labeled) by 5.2 points, GAIA from 46.7 to 53.3, and ALFWorld from 88.1% to 91.8%, showing that tool-use improvements transfer to interactive downstream environments.

• Memory folding is critical for long-horizon tasks. Removing it drops the average score from 48.1 to 44.2, with GAIA falling from 53.3 to 44.7 — the largest single-component degradation alongside removing training altogether.

• Dynamic tool retrieval outperforms pre-retrieval. Across every framework tested (ReAct, Plan-and-Solve, DeepAgent), retrieving tools on demand during reasoning beats injecting a pre-retrieved tool list up front. DeepAgent benefits most, averaging 52.6 versus 28.5 for the best workflow-based method.

• Training is more stable and reaches higher ceilings than GRPO. ToolPO shows less reward fluctuation during training, attributed to the tool simulator and process-level supervision.

• DeepAgent scales better with longer action budgets. As the maximum action limit increases on WebShop and GAIA, the performance gap over ReAct widens, suggesting the agent selects more task-relevant actions rather than wasting steps.

• Results generalize across backbones. With Qwen3-30B-A3B-Thinking and Qwen3-235B-A22B-Thinking, DeepAgent still outperforms ReAct and Plan-and-Solve, with the largest absolute gains on complex application tasks.

Methodology in Plain English

The agent is built around a main large reasoning model (QwQ-32B in the primary experiments) that emits actions as special text tokens inside its normal chain of thought. Four action types are supported: internal thought, a tool-search query (which triggers dense retrieval over tool documentation embeddings), a structured tool call (JSON with tool name and arguments), and a memory fold trigger.

When the agent searches for a tool, the system retrieves the top-k most similar tool documents by cosine similarity, then an auxiliary LLM (Qwen2.5-32B-Instruct) filters or summarizes the retrieved documentation if it is too long before returning it to the main model's context. Tool outputs are similarly denoised and condensed by the auxiliary model. This division of labor keeps the main reasoning model focused on strategy rather than on sifting through verbose documentation.

Memory folding works the same way: when the agent emits the fold token, the auxiliary LLM processes the entire preceding history and produces three parallel structured memories in JSON format — episodic, working, and tool memory — which then replace the raw history in the context.

For training, the authors avoid calling thousands of real APIs by building an LLM-based tool simulator that mimics real API responses (e.g., RapidAPI), giving a stable, cheap environment. Trajectories are sampled in groups of eight. Two rewards are computed: a task-success reward applied globally to all tokens, and an action reward applied only to tool-call and memory-fold tokens via a mask. The action reward counts correct tool invocations and adds a preference term favoring trajectories that fold memory efficiently (measured by relative length reduction). These produce two separate group-relative advantages that are summed and optimized with a clipped surrogate objective, similar in shape to PPO/GRPO. Training runs for 100 steps with batch size 64 on 64 NVIDIA H20-141GB GPUs.

Why This Matters

Impact on research. The paper argues against the dominant paradigm of hand-engineered agent workflows and fixed toolsets, showing empirically that a single reasoning model trained end-to-end can handle tool discovery, execution, and memory management more effectively. It also introduces a concrete way to do process-level credit assignment in tool-use RL (attributing advantage only to tool-call tokens), which is a reusable technique beyond this specific system. The alignment with the Model Context Protocol (MCP) paradigm for dynamic tool ecosystems is explicitly framed as a direction for the field.

Real-world applications.

• Enterprise API orchestration: agents that discover and call the right internal or third-party API from a catalog of thousands, without a human pre-selecting tools.
• Deep research assistants: multi-step web search, page browsing, and code execution over long horizons, as tested on GAIA and HLE.
• Embodied and robotic task agents: ALFWorld-style instruction following with move/take actions, where memory folding helps recover from failed attempts.
• E-commerce and customer-service agents: WebShop-style shopping interactions plus domain-specific tool routing.

Industry relevance. The reliance on LLM-simulated APIs during training is a practical cost and stability argument for companies that cannot afford to train against live production APIs. The framework's ability to operate over arbitrary-scale toolsets maps directly onto the emerging MCP tool-registry model, and the memory folding mechanism addresses the context-window economics that dominate the cost of long-horizon agent deployments.

Future Directions

• Simulator fidelity. Training on LLM-simulated APIs is stable and cheap, but the gap between simulated and real API behavior — error modes, latency, rate limits, malformed responses — remains unquantified. Closing this sim-to-real gap is an open problem.

• Memory compression fidelity. Folding trades context length for information loss. How much task-critical detail is discarded, and whether folding schemas can be learned rather than hand-designed, are unresolved questions.

• Scaling to shared, evolving tool ecosystems. The paper aligns with MCP, but evaluating agents against live, versioned, mutually incompatible tool registries — where tool documentation itself changes — is untested.

• Safety and reliability of autonomous tool invocation. Giving a reasoning model unrestricted authority to discover and execute tools raises questions about irreversible actions, permission scoping, and failure containment that the paper does not address.

• Beyond the auxiliary LLM. The current design offloads retrieval filtering, output denoising, and memory compression to a separate 32B auxiliary model; whether the main reasoning model can absorb these roles without degrading strategic reasoning is an open architectural question.

Target Audience

Researchers and engineers working on LLM agents, tool-augmented reasoning, and reinforcement learning for language models. It is most useful to readers already comfortable with reasoning-model training pipelines (GRPO/PPO-style objectives), dense retrieval, and agent frameworks such as ReAct — and to practitioners building production agent systems that need to operate over large, changing tool catalogs rather than a handful of hard-coded tools.

Authors’ abstract

Large reasoning models have demonstrated strong problem-solving abilities, yet real-world tasks often require external tools and long-horizon interactions. Existing agent frameworks typically follow predefined workflows, which limit autonomous and global task completion. In this paper, we introduce DeepAgent, an end-to-end deep reasoning agent that performs autonomous thinking, tool discovery, and action execution within a single, coherent reasoning process. To manage long-horizon interactions, we introduce an autonomous memory folding mechanism that compresses past interactions into structured episodic, working, and tool memories, reducing error accumulation while preserving critical information. To teach general-purpose tool use efficiently and stably, we develop an end-to-end reinforcement learning strategy, namely ToolPO, that leverages LLM-simulated APIs and applies tool-call advantage attribution to assign fine-grained credit to the tool invocation tokens. Extensive experiments on eight benchmarks, including general tool-use tasks (ToolBench, API-Bank, TMDB, Spotify, ToolHop) and downstream applications (ALFWorld, WebShop, GAIA, HLE), demonstrate that DeepAgent consistently outperforms baselines across both labeled-tool and open-set tool retrieval scenarios. The code and demo are available at https://github.com/RUC-NLPIR/DeepAgent.