AdaExploreBench: Benchmarking LLM Agents for Proactive Exploration and Test-time Adaptation in Evolving Environments

AI summary

Overview

Research area: Natural Language Processing / LLM agents — specifically benchmark design for agentic exploration and test-time adaptation in non-stationary environments.

Technical level: Advanced. The paper assumes familiarity with LLM agent evaluation, interactive decision-making, and rule-induction benchmarks.

Scope: Introduces AdaExploreBench, a procedurally generated benchmark of 256 evaluation and 1000 training episodes across four latent rule families, and reports an eight-model evaluation showing that current LLM agents acquire evidence poorly and adapt to unannounced rule change only partially.

What This Paper Is About

LLM agents are increasingly dropped into unfamiliar environments where the rules governing how the environment responds to their actions are never stated outright. Existing benchmarks either hand the agent the relevant evidence, leak it from the environment, or keep the environment fixed throughout the interaction. This paper builds a benchmark in which the agent must decide for itself when, what, and how much to probe in order to infer hidden rules — and must then notice, without any announcement, when those rules silently change and re-explore to recover.

Key Contributions

1. A scalable generation pipeline. AdaExploreBench synthesizes evaluation episodes with LLM-assisted generation over predefined task blueprints in finance, healthcare, cybersecurity, and manufacturing, without additional human labor beyond validation, and produces 256 evaluation plus 1000 training instances.
2. An evolving-environment design built on latent rules. Environmental dynamics are instantiated as hidden rules mapping observable fields to an intervention score, spanning four structurally distinct families (additive, conjunctive, disjunctive, lexicographic). Dynamic episodes introduce an unannounced rule drift that replaces one relevant factor and reverses another factor's direction within the same family.
3. A runtime protocol that isolates proactive exploration. Evaluation cases are presented in small sequential batches preceded by exploration windows, completed evaluation cases are masked from context while self-collected probe observations are retained, and the resulting performance curve enables in-process evaluation rather than a single end-of-episode score.
4. A diagnostic evaluation of eight LLMs. The authors combine end-to-end outcome metrics with exploration-behavior and adaptation metrics, plus three ablation settings (Stationary, Evidence Given, Forced-Probe), to isolate where agents actually fail.

Main Findings

• Proactive and adaptive exploration is hard for every model tested. All eight models score below 70% calibrated accuracy in the default setting; Gemini-3.8-Flash is strongest at 67.25% CA and 69.47% EA, while GPT-6-Sol reaches 61.87% CA (the abstract cites 61.9%) and Llama-3.1-8B falls to 8.13% CA.
• Weaker models lose more utility per mistake. The gap between exact accuracy and calibrated accuracy is roughly 2 to 5 points for the top three models but grows to 8 to 22 points for the remaining models, indicating that weaker agents' errors are not just more frequent but costlier.
• More probing does not mean better performance. DeepSeek-V4.1-Flash issues roughly 24 times as many probes as Llama-3.1-8B (113.42 vs. 4.77 mean unique probes) yet scores lower CA than both Gemini-3.8-Flash and GPT-6-Sol, and Gemini-3.8-Flash beats GPT-6-Sol with fewer probes.
• Removing rule drift recovers substantial performance. The Stationary setting raises CA by up to 14.7% for DeepSeek-V4.1-Flash and 6.2% on average, though the weakest models improve by less than 1% and GPT-6-Sol barely moves because its default performance is already near its stationary ceiling.
• Given evidence is exploited far better than self-acquired evidence. In the Evidence Given setting, the paper reports six of seven models improving, four by more than 15 CA points; the abstract and introduction summarize the average gain as 9.4%. Qwen3.8-27B is the only model that degrades, which the authors attribute to trusting stale evidence without re-verifying.
• Forcing exploration does not help. Under Forced-Probe, mean unique probes rise from 33.9 to 53.8 across all models and the most reluctant explorers scale up two to five times, yet mean CA changes by −2.0%.
• Rule drift causes an immediate, unrecovered drop. Averaged across the six models operating above the random baseline, accuracy falls 26.9 points at the first post-change batch, and over the final two batches all six remain 6.9 to 19.2 points below their own pre-change plateau. Llama-3.1-8B falls to 1.3% by the final batch.
• Recognition ability tracks end-to-end performance better than exploration effort does. The four strongest models by CA follow exactly the same ranking on Change Recognition (Gemini-3.8-Flash 63.68%, GPT-6-Sol 47.56%, DeepSeek-V4.1-Flash 44.52%, Qwen3.8-27B 25.93%), while the other models reach at most 13.80%.
• Re-probing correlates with change recognition within models, but not across them. Within-model Spearman correlations between re-probes and Change Recognition are significant (pooled 0.37 for counts, 0.28 for rates), and the across-model count correlation is 0.86, but the across-model rate correlation drops to 0.45 and becomes insignificant after normalizing by pre-change exploration.
• Agents settle on incomplete rules. Precision exceeds recall for both Factor Recognition (mean 67.6 precision vs. 60.9 recall) and Change Recognition (mean 46.4 vs. 25.9). In 36.3% of eligible episodes, agents report identical pre- and post-change assessments despite an actual change — 65.6% for GPT-6-Luna and 54.2% for Qwen3.8-27B.
• Direction reversals are harder to revise than factor replacements. Agents miss the reversal in 17.7% of episodes while identifying the replacement, compared with only 4.1% for the converse case.

Methodology in Plain English

Each benchmark episode hands the agent a task scenario, a set of observable input fields with descriptions and ranges, and four ordered actions ranging from weakest to strongest intervention. Hidden from the agent is which subset of fields actually matters, in which direction each field pushes the decision, and which of the four rule families governs how those fields combine into a single intervention score between −1 and 1. A fixed utility function then rewards actions whose intervention level matches that score, so the agent is effectively trying to reverse-engineer a scoring function from noisy feedback.

The agent interacts through tools: it can query a Probe Batch of cases it designs itself and see the utilities for all four actions, skip probing and request the next evaluation batch, submit actions for an evaluation batch (which returns utility feedback), and call a Calculator for arithmetic. Since evaluation cases are masked after submission, the agent cannot simply read the environment's own evidence — only its own probes persist in context, alongside cumulative utility feedback that may hint something has changed.

To test adaptation, dynamic episodes silently swap one relevant factor for another and flip the direction of a second factor partway through. Because evaluation arrives in small sequential batches, an agent with modest exploration effort can still notice the drift through degraded utilities. The authors then run four configurations — Default (drift on, masking on, self-decided probing), Stationary (no drift), Evidence Given (evaluation batches unmasked), and Forced-Probe (mandatory minimum probing) — to attribute failures to drift, to evidence acquisition, or to probe quality. Separate auxiliary assessments, collected before the drift and at episode end and withheld from the interaction context, measure whether the agent can name the relevant factors and the specific changes.

Why This Matters

Impact on research. The paper separates two capabilities that prior benchmarks conflate — actively acquiring evidence, and recognizing that previously acquired evidence has gone stale. Its ablation logic (Forced-Probe rules out insufficient effort; Evidence Given rules out weak induction) leaves probe informativeness and hypothesis revision as the remaining bottlenecks, which gives the field a more specific target than "agents need to explore more." The 1000 training instances also position the benchmark as a training playground, not just a test set.

Real-world applications:

• Clinical decision support, where which patient factors drive a recommendation, and how strongly, can change as guidelines or formularies are updated without notice.
• Fraud and cybersecurity detection, where adversaries change behavior and a model's learned rules silently decay.
• Industrial and manufacturing process control, where a hidden combination of sensor readings determines the correct intervention strength, and equipment drift invalidates earlier findings.
• Financial risk and trading agents, where the mapping from market signals to appropriate position sizing shifts regime without announcement.

Industry relevance. Teams deploying agents that must operate for long horizons in partially observable systems — rather than answering one-shot questions — face exactly the failure mode documented here: agents that perform well when handed the right context but neither gather it nor notice when it expires. The finding that more interaction does not buy better outcomes argues against using probe counts or tool-call volume as a proxy for agent competence in evaluation dashboards.

Future Directions

• Measuring probe quality directly. The authors explicitly flag that their exploration metrics capture effort rather than quality, and that they avoided spatial-diversity or decision-region-coverage metrics because these favor extensive probing and are incomparable across heterogeneous rule families. A family-agnostic quality metric remains open.
• Beyond within-family drift. The benchmark currently restricts change to factor replacement plus direction reversal within the same rule family; cross-family changes, other change types, and systematically varied change magnitudes are left for future work due to time and compute constraints.
• Richer environment abstractions. The latent-rule formulation is a controlled simplification; extending it toward messier real-world dynamics and uncertainty is an explicit limitation.
• Training against the benchmark. With 1000 training instances released, the natural next step is methods that teach agents to select informative probes and to actively challenge their own established hypotheses rather than settling on plausible but incomplete rules.

Target Audience

Researchers building and evaluating LLM agents for long-horizon, partially observable tasks; benchmark designers interested in procedurally generated, contamination-resistant evaluation; and practitioners who need to know whether an agent can be trusted to operate after deployment conditions shift. Readers primarily interested in single-turn reasoning or pure prompting techniques will find less here, since the paper's contribution is an evaluation environment and a diagnostic analysis rather than a new model or training method.

Authors’ abstract

Large Language Model (LLM) agents are increasingly deployed in unfamiliar and evolving environments, requiring them to proactively explore, understand, and adapt to environmental dynamics. However, existing benchmarks rarely test whether agents can proactively discover how an environment works and adapt as it evolves. To address this gap, we introduce AdaExploreBench, a benchmark for proactive exploration and test-time adaptation in evolving environments. By instantiating environmental dynamics as hidden rules, AdaExploreBench requires agents to actively acquire evidence and infer the rules. Under implicit rule drift, the latent rule changes without announcement, requiring agents to detect from degraded performance that prior knowledge is stale and re-explore. Experiments on eight leading LLMs show that proactive and adaptive exploration remain challenging: GPT-6 Sol achieves only 61.9% calibrated accuracy, while providing sufficient evidence or removing rule drift improves average performance significantly by close to 10%. These results reveal a critical bottleneck in current LLM agents: they can often reason effectively from informative evidence once it is available, but struggle to actively acquire the right evidence and recognize when previously acquired evidence is no longer valid.