ExpSeek: Self-Triggered Experience Seeking for Web Agents

AI summary

Overview

• Research area: Natural Language Processing / LLM-based web agents; specifically, experience-augmented agent reasoning and uncertainty-based intervention.
• Technical level: Intermediate. Familiarity with LLM prompting, ReAct-style agents, and token entropy is helpful, but the core ideas are explained conceptually throughout.
• Scope: The paper introduces ExpSeek, a framework that lets a web agent decide at each step whether it needs external experience guidance and then generates guidance tailored to that step, rather than injecting experience passively at the start of a task.

What This Paper Is About

Web agents — LLM-driven systems that search and browse the web over multiple turns — often explore inefficiently or answer prematurely. A common fix is to feed the agent "experience" (lessons distilled from past successful and failed trajectories), but existing methods dump that experience into the prompt once, before the task even begins, which cannot adapt as the web page content and the agent's situation change. ExpSeek solves this by making experience-seeking proactive and step-level: the agent monitors its own uncertainty (measured by entropy) and requests guidance only when it is genuinely confused, receiving content customized to the current step.

Key Contributions

1. A proactive experience-seeking paradigm. The paper reframes experience intervention from a one-time global injection into an on-demand, step-level mechanism, formalizing the problem as deciding both when to seek experience and what experience to seek.

2. Entropy as a self-trigger signal. The authors show that a model's own step entropy distinguishes correct from incorrect steps, and use logistic regression with bootstrap resampling to derive per-model threshold intervals (separately for "process" steps and "answer" steps) that control intervention timing probabilistically.

3. A structured experience base and a generative experience model. Experience is stored as triplets — Behavior, Mistake, Guidance — organized into topics, extracted by contrasting failed trajectories against successful ones. At inference, an experience model retrieves the top-3 relevant topics and generates situation-specific guidance.

4. Strong empirical validation and analysis. Across four web-agent benchmarks and two backbone models (Qwen3-8B, Qwen3-32B), ExpSeek delivers average absolute gains of 9.3% and 7.5% over vanilla ReAct, with a peak gain of 14.6%, plus analyses of internal entropy dynamics, scaling, transfer, and efficiency.

Main Findings

• Large, consistent accuracy gains: ExpSeek improves Qwen3-8B by 9.3% and Qwen3-32B by 7.5% on average across WebWalkerQA, GAIA, Seal-Hard, and xbench-DeepSearch, beating the strongest passive-injection baselines by 6.7% and 6.0% respectively.

• Passive global injection barely helps: Training-Free GRPO and ReasoningBank+ improve performance by under 3% or even degrade it, indicating that static experience cannot keep up with the noisy, shifting web environment.

• Entropy separates correct from incorrect steps — but unevenly: Correct steps consistently have lower entropy (statistically confirmed via Kolmogorov–Smirnov tests). Discrimination is weak for process steps (AUC = 0.6223) and noticeably better for answer steps (AUC = 0.7187), which motivates separate thresholds for each step type.

• Divergence-then-convergence behavior: After guidance, entropy increases during intermediate process steps (the agent explores alternative paths) and decreases at the final answer step (the agent commits confidently to an answer). This is the paper's main mechanistic explanation for why the method works.

• Answer-step guidance matters most, but both types matter: Guiding only answer steps gets closer to full performance than guiding only process steps, yet neither ablation matches the full method (8B drops of 4.91% / 2.44%; 32B drops of 4.51% / 4.12%).

• Self-triggering is far more efficient than alternatives: Rule-based triggering costs 1.7x more steps and 2.6x more time on GAIA; reward-model triggering cuts steps but costs 2.2–2.9x more time. Entropy triggering keeps accuracy while adapting intervention rate to task difficulty (25.6% higher trigger rate on the harder xbench benchmark).

• Generation beats retrieval: Replacing generated guidance with retrieval of the most similar experience via embeddings substantially lowers accuracy.

• Weak-to-strong guidance works: A 4B experience model improves the 32B agent by 5.2 and 9.7 points on GAIA and xbench, and experience-model size follows a clean scaling law.

• Experience transfers, but is partly model-specific: Swapping experience repositories between 8B and 32B agents yields degraded but still useful performance; even one experience per topic preserves much of the gain, highlighting the value of topic organization.

• Diminishing returns on intervention frequency: Accuracy plateaus around two interventions (43.01% on GAIA) and barely improves beyond six, while reasoning steps keep rising.

Methodology in Plain English

The researchers build the system in three parts.

1. Building an experience library. They sample several trajectories per training query, pair successful ones against failed ones, and use a strong "tool model" to compare them. For each step where the failed trajectory went wrong, the tool model writes a triplet: what the agent did (Behavior), why it was a mistake (Mistake), and what direction it should have taken instead (Guidance). Crucially, guidance gives direction without revealing the answer. These triplets are then clustered into topics (e.g., "repetitive search queries"), stored separately for process steps and answer steps.

2. Deciding when to intervene. While the agent reasons, the researchers average the entropy of every token in each response to get a "step entropy." They fit a logistic regression predicting whether a step is wrong from its entropy, then repeat this on 1,000 bootstrap resamples to get a 95% confidence interval around the decision boundary. This gives a lower and upper threshold per step type and per model. At inference, a step below the lower bound gets no help; above the upper bound it always gets help; in between, help is triggered with a probability that scales linearly across the interval.

3. Generating and injecting guidance. When triggered, an experience model reads the conversation history, picks the three most relevant topics from the library, and writes guidance tailored to the current situation. For process steps, this guidance is appended to the environment observation. For answer steps, the guidance is treated as a new observation so the agent gets another turn to either revise its answer or keep searching. To avoid over-intervention, no intervention is allowed at the step immediately after one fires.

Why This Matters

Impact on research. The paper argues against the prevailing "retrieve-and-prepend" pattern in experience-augmented agents and shows that a model's own uncertainty signal is a cheap, effective, training-free trigger. It also demonstrates weak-to-strong guidance — a small 4B model reliably improving a 32B agent — which is a notable result for cost-conscious deployment.

• Search and research assistants: Deep-research agents that browse for evidence can use this to avoid premature answers and repetitive searches on multi-hop queries.
• Customer-facing web automation: Agents filling forms, comparing products, or navigating portals could self-correct when a page layout confuses them, instead of looping or failing silently.
• Enterprise RPA and IT workflows: Task automation over internal web tools benefits when the agent can detect its own confusion and consult a curated error-pattern library rather than requiring constant global prompting.
• Cost-sensitive deployments: Because the trigger is entropy-based and swappable small experience models suffice, the approach fits settings where running a large model at every step is prohibitively expensive.

Industry relevance. The work comes from a collaboration between the Chinese Academy of Sciences and Alibaba's Tongyi Lab, and the efficiency results (steps and wall-clock time per question) speak directly to serving costs. The finding that a 4B model can guide a 32B model also has clear commercial implications for inference budgeting.

Future Directions

• More robust threshold estimation: The current thresholds depend on the training set and a tool model's step-quality judgments; the authors flag this as a limitation and call for better strategies.
• Beyond web search: Whether ExpSeek extends to non-web domains (e.g., code agents, robotics, tool-rich APIs) is untested.
• Use in agentic reinforcement learning: Given the pass@k improvements, the method might serve as a rollout enhancement to accelerate convergence or improve sampling quality during RL training.
• Understanding and tuning the intervention budget: The efficiency study shows clear diminishing returns past a handful of interventions; principled ways to set the intervention budget per task difficulty remain open.

Target Audience

Researchers and engineers working on LLM agents, tool-use, and multi-turn web reasoning will get the most from this paper. It is also relevant to practitioners building production search or browsing agents who care about inference cost, and to anyone interested in uncertainty quantification, entropy-based error detection, or experience-based agent memory. Readers without a background in LLMs will find the high-level framework accessible, but the threshold-estimation and entropy-analysis sections assume familiarity with token-level probability distributions.

Authors’ abstract

Experience intervention in web agents emerges as a promising technical paradigm, enhancing agent interaction capabilities by providing valuable insights from accumulated experiences. However, existing methods predominantly inject experience passively as global context before task execution, struggling to adapt to dynamically changing contextual observations during agent-environment interaction. We propose ExpSeek, which shifts experience toward step-level proactive seeking: (1) estimating step-level entropy thresholds to determine intervention timing using the model's intrinsic signals; (2) designing step-level tailored experience content. Experiments on Qwen3-8B and 32B models across four challenging web agent benchmarks demonstrate that ExpSeek achieves absolute improvements of 9.3% and 7.5%, respectively. Our experiments validate the feasibility and advantages of entropy as a self-triggering signal, reveal that even a small-scale 4B experience model can significantly boost the performance of larger agent models. The code is released at https://github.com/WYRipple/ExpSeek.