Agent-Reflex: Teaching Language Agents to Act Reflectively

AI summary

Overview

Research area: Large language model agents, specifically long-horizon tool use, noisy-environment robustness, and reflection learning for agentic policies.

Technical level: Advanced. The paper assumes familiarity with supervised fine-tuning, reinforcement learning for LLMs (clipped policy-gradient objectives, group-normalized advantages, KL regularization), and on-policy distillation. The framing of the problem is accessible; the Stage 3 objective is not.

Scope: Agent-Reflex is a three-stage training framework that mines reusable "reflection meta-skills" from agent failures under injected tool noise, internalizes them through supervised fine-tuning, and then reinforces them with agentic RL plus reflection-guided on-policy distillation, built on Qwen3-8B.

What This Paper Is About

Existing agent-training environments assume clean, stable tool feedback and minimal user intervention, while real deployments return timeouts and service errors and users step in to correct the agent. The paper argues that agents therefore lack proactive reflection: the ability to notice noise, failure, or correction intent and revise their actions promptly. Agent-Reflex aims to turn reflection into a learnable, environment-indexed meta-skill that is mined from real failures and internalized into the policy.

Key Contributions

1. Problem framing of two gaps. The authors identify two mismatches between idealized synthetic environments and real deployment: lack of noisy environment feedback, and lack of user-guided correction during execution, highlighting reflection learning as the missing capability.

2. Environment Reflection Skill Discovery (RSD). A stage that builds a scalable noisy-environment ecosystem by injecting perturbations grounded in real tool errors into 557 environment themes, sampling trajectories from a capability-diverse LLM pool, and mining matched success/failure pairs into an offline library of reflection meta-skills.

3. Reflective Supervised Fine-Tuning (ReSFT). Three verifier-validated sampling strategies (Self-Reflection, Skill Recovery, User-Centric Reflection) that convert each mined skill into multi-level reflective supervision, internalized by supervised fine-tuning.

4. On-Policy Reflective Policy Optimization (ORPO). A joint objective combining Noisy Agent RL with reflection-guided on-policy distillation, giving token-level supervision on failure-causing decisions rather than only a trajectory-level scalar return.

Main Findings

• Overall long-horizon performance. Built on Qwen3-8B, Agent-Reflex reaches 58.6 overall on five long-horizon tool-use benchmarks, the highest overall score among the compared 7B–8B baselines. For reference, the largest foundation models in the same table score 64.4 (GPT-5.2), 72.6 (Claude-4.5-Sonnet), 72.8 (Gemini 3 Pro), and the Qwen3-8B backbone scores 40.2.

• Better than inference-time reflection. On VitaBench, Agent-Reflex achieves 17.5% versus 6.8% for Reflexion-8B. The paper attributes this to training directly on verified recovery behavior rather than relying on self-generated critiques or retrieved experience at inference time; both Reflexion-8B and ExpeL-8B fall below the Qwen3-8B backbone (11.4%) on VitaBench.

• Better than trajectory-level reflection training. On ACEBench, Agent-Reflex reaches 78.0% versus 52.0% for NoisyAgent-8B. The authors read this as evidence that environment-level meta-skills give recovery guidance beyond individual correction trajectories or episode-level rewards.

• Complementary to environment scaling. Agent-Reflex improves over EnvScaler-8B and Agent-World-8B on all five benchmark averages, including 62.1% vs. 54.6% on τ²-Bench. On ACEBench the gain is concentrated in multi-turn interaction: 73.3% turn-level versus 85.0% step-level, where it matches the baselines.

• Robustness under noise. On AgentNoiseBench, Agent-Reflex attains the highest Overall score among compared 8B methods at 29.1, versus 25.9 for NoisyAgent-8B. For context, the paper reports that every foundation model degrades markedly under noise; GPT-5.2 goes from 81.0 clean to 61.0 under tool noise and 72.0 under user noise on AgentNoiseBench-τ², for an Overall of 55.8.

• ORPO beats other policy-optimization algorithms. Training ORPO and five representative algorithms from Qwen3-8B for 100 RL steps on the same task pool under matched budgets, ORPO holds a stable advantage over GRPO, DAPO, REINFORCE++, SEED, and noisy Agent RL on BFCL V4, τ²-Bench, and real interaction noise.

• Stable training dynamics. Tracking mean rollout reward over the same 100 steps, ORPO separates from GRPO and SEED early and keeps rising after both flatten. The authors attribute this to the confidence weighting, which emphasizes tokens with a positive skill-induced shift and down-weights skill-neutral or negative ones.

• Ablation ordering. Removing Self-Reflection causes the largest drop (AgentNoiseBench-τ²: 45.9 to 41.3; ANB-Vita: 12.3 to 8.9), followed by Skill Recovery (42.4 / 10.2) and User-Centric Reflection (43.0 / 10.9). Removing OPD also hurts (43.7 / 9.5), and removing only its confidence weighting declines as well (44.5 / 10.4), indicating the token-level reflection signal, not mere skill exposure, drives the gains.

• Failure-family grounding. Replaying agents across nine public agentic benchmarks produced 177,700 tool returns, of which 24,510 are error returns, grouped into eight actionable failure families that are then injected probabilistically.

• Skill-library scale. RSD produced 1,170 reflection meta-skills across 557 environments, with 60 verifier-backed tasks per topic.

Methodology in Plain English

Stage 1 — Building noisy environments and mining reflection skills. The team selected 557 high-usage MCP server specifications from Smithery as environment topics. For each topic, a research agent built a topic-grounded database, and a coding agent generated candidate tools and unit tests; only tools that compile, pass tests, satisfy state-transition constraints, and support executable verification were kept, yielding an environment of a database, a validated tool set, and verifiers. To make noise realistic rather than invented, they replayed agents across nine public agentic benchmarks and logged every tool return, collecting 177,700 returns and 24,510 errors, which they grouped into eight failure families. These failures are injected into tool returns probabilistically (Bernoulli with per-return probability p_inj), balanced across families. Trajectories were then sampled from a pool of nine models across five families (4B to 284B parameters) under one protocol, and an analyzer model plus the environment verifier jointly labeled each rollout as correct or erroneous. Success and failure trajectories from the same task were paired, per-task pair contributions were limited, and each environment had to draw pairs from at least five distinct tasks. An analyzer then converted recurring failure patterns into executable recovery rules, stored per environment as a tuple describing the environment, its tools, its bad patterns, and its skills.

Stage 2 — Turning skills into supervised data. Starting from a frozen behavior policy, three sampling strategies generate supervision. Self-Reflection restores the environment to the state before the first erroneous turn, appends the corrective skill, generates a continuation, and keeps it only if the verifier says the task completes; the skill is stripped from the stored input so the policy learns to recover from its own error state. Skill Recovery instead resets to the initial state and restarts from the original task with the skill, targeting failures that come from a bad initial plan; the skill is again removed from the stored input. User-Centric Reflection uses a user simulator to turn the skill into a natural-language correction; because such a message is deployment-visible, it stays in the stored input while the privileged skill is removed. These three datasets are merged with correct trajectories generated by the target policy and trained with a standard negative log-likelihood objective.

Stage 3 — Reinforcing reflection on the policy's own rollouts. ORPO reuses the EnvScaler RL task pool. For each task it draws one mixed group of trajectories, a fraction rolled out in noisy environments and the rest clean, with none given a reflection-skill prompt. A verifier assigns each trajectory a return, and advantages are normalized over the mixed group as a whole. The clipped agent-RL objective is applied over valid agent-generated tokens (excluding tool returns and simulated user messages), with a KL term to a reference policy. In parallel, the environment's skill card is prepended to the history, and the same policy re-scores exactly the already-sampled action tokens under that skill context. The detached difference between the skill-context score and the ordinary-context score becomes a confidence weight, and the distillation loss raises the student's score on tokens the skill-conditioned "teacher" endorses. The two losses are combined as L_RL + λ_opd · L_OPD. No retrieval or extra matching step is needed because each RL task already specifies its environment and the skill library is organized by environment.

Why This Matters

Impact on research. The paper shifts reflection learning away from per-trajectory repair and toward environment-indexed, reusable meta-skills stored before any policy training, and it adds a token-level reflection signal on top of sparse outcome rewards. It also connects noisy-environment robustness to agentic RL rather than treating noise purely as an inference-time obstacle, and it demonstrates a training recipe that runs on a single 8B backbone.

Real-world applications:

• Customer-facing and internal workflow agents that call MCP tool services, where timeouts and service errors are routine rather than exceptional.
• Enterprise tool-use agents operating over database-backed services such as booking, order management, or record updates, where a wrong initial plan must be diagnosed rather than patched locally.
• Interactive assistants where users interrupt mid-execution to correct the agent's reasoning, requiring the agent to absorb the correction and continue rather than restart.
• Production function-calling deployments that need stable policy behavior when tool feedback is unreliable, since the paper reports ORPO keeps training rewards rising over 100 steps where GRPO and SEED flatten.

Industry relevance. The framework is anchored on MCP server specifications and reuses an existing RL task pool, which makes it closer to deployment plumbing than a purely academic construct. The code and model/data releases are linked (github.com/RUC-NLPIR/Agent-Reflex and a Hugging Face collection), and the reported 8B results land between the Qwen3-8B backbone (40.2 overall) and much larger foundation models, which is the range most deployment budgets can afford.

Future Directions

1. Beyond Qwen3-8B and beyond the tested pool. The work is built on and evaluated from Qwen3-8B; whether the same RSD-to-ORPO pipeline transfers to other backbones or larger scales is not reported in the provided content, and would be a natural next experiment.

2. Skill-library transfer across environments. Skills are stored per environment and loaded directly with the environment. Whether meta-skills mined in one environment family can be reused in a new environment that was never seen during discovery is not reported and is the obvious test of the "reusable" claim.

3. Fine-grained source of the noise-robustness gain. The paper states that Appendix E.1 provides a fine-grained attribution of the observed noise-robustness gains, but that appendix content is not included in the material reviewed here, so the breakdown across noise types remains to be examined.

4. Scope of the failure taxonomy. The taxonomy covers eight failure families mined from 24,510 error returns on nine public benchmarks. Whether that taxonomy remains complete for other tool ecosystems, or for user-intervention styles not captured by the simulator, is left open.

Note: the paper's conclusion does not list an explicit future-work section; the items above are open questions the reported setup and results raise.

Target Audience

Researchers working on LLM agents, agentic reinforcement learning, and long-horizon tool use; engineers building MCP-based production agents who need robustness to tool and user noise; and practitioners interested in on-policy distillation techniques that supply token-level supervision when only sparse verifiable outcomes are available. Readers without RL background will find the problem framing and Stages 1 and 2 accessible but will need to work through the Stage 3 objectives carefully.

Authors’ abstract

Large language models have evolved from single-turn chatbots into interactive agents capable of long-horizon reasoning. In real deployments, MCP tool services can return noisy feedback and users may intervene to correct the agent, but existing methods provide limited supervision for reflection learning and failure recovery. To bridge this gap, we propose Agent-Reflex, a general training framework for incentivizing proactive reflection in agents. Specifically, Agent-Reflex comprises three stages. First, Environment Reflection Skill Discovery (RSD) injects grounded tool-feedback perturbations into environment scaling, samples a capability-diverse LLM pool, mines matched success-failure pairs, and summarizes reusable reflection meta-skills. Building on this skill library, we then apply Reflective Supervised Fine-Tuning (ReSFT) that converts each skill into supervision for self-reflection, skill recovery, and user-centric correction, strengthening complementary reflective behaviors. Finally, we propose On-Policy Reflective Policy Optimization (ORPO) to combine agentic reinforcement learning under noisy interactions with reflection-guided on-policy distillation to further elicit proactive reflection. Built on Qwen3-8B, Agent-Reflex reaches 58.6 overall on five long-horizon tool-use benchmarks and 29.1 on AgentNoiseBench, surpassing strong environment-scaling and reflection baselines. Experiments on six challenging agent benchmark suites demonstrate the effectiveness of Agent-Reflex. Further analyses show that Agent-Reflex improves agents' robustness to noise while maintaining stable training dynamics, supporting scalable agent training. Our code is available at https://github.com/RUC-NLPIR/Agent-Reflex.