Gaia2: Benchmarking LLM Agents on Dynamic and Asynchronous Environments

AI summary

Overview

Research area: Artificial Intelligence — benchmarking and evaluation of large language model (LLM) agents in asynchronous, event-driven simulated environments.

Technical level: Intermediate.

Scope: This paper introduces Gaia2, a benchmark of 1,120 human-annotated scenarios built on the open-source Agents Research Environments (ARE) platform, together with an action-level verifier and an empirical study of state-of-the-art proprietary and open-source models.

What This Paper Is About

Most existing agent benchmarks are static or synchronous: the environment only changes when the agent acts, and evaluation usually looks only at the final answer. That leaves untested many of the challenges agents face in real deployments — asynchronous events, time pressure, noise, ambiguity, and coordination with other agents. The paper's goal is to build a benchmark and an underlying platform where environments evolve independently of the agent, and where every state-changing action can be verified, so that the benchmark is usable both for evaluation and for reinforcement learning from verifiable rewards (RLVR).

Key Contributions

1. ARE framework: The paper releases Agents Research Environments, a general-purpose platform for building asynchronous, event-driven benchmarks that support scalable evaluation and data generation for RL. The authors report validating that ARE can faithfully reimplement existing agentic benchmarks including τ-bench, τ²-bench, GAIA, BFCL-v3, and VendingBench.

2. Gaia2 benchmark: Gaia2 is presented as the first benchmark unifying asynchronous execution, temporal reasoning, noise robustness, ambiguity resolution, and multi-agent collaboration under a verifiable evaluation framework directly usable for RLVR. It consists of 800 unique human-authored core scenarios plus 320 augmentation scenarios (from the 160-scenario Gaia2-mini subset), totaling 1,120 scenarios across 10 universes in a smartphone-like Mobile environment with 101 tools.

3. Write-action verifier: The ARE Verifier checks every state-changing write action against oracle annotations, combining exact argument checks, rubric-guided judgments for flexible fields, and causal and temporal constraints. The authors describe it as a standalone, reusable mechanism for fine-grained, reproducible credit assignment.

4. Empirical study: An evaluation of leading proprietary and open-source models with a shared ReAct-style scaffold, exposing what the authors call fundamental trade-offs between reasoning strength, efficiency, robustness, and cost.

Main Findings

• No model dominates across capabilities: GPT-5 (high) reaches the strongest overall score at 42.1% pass@1 in Table 2 (described as 42% in the abstract and conclusion), maintaining what the paper calls an 8-point margin over Claude-4-Sonnet across all categories. The abstract states that GPT-5 (high) "fails on time-sensitive tasks."

• Open-source leader: Kimi-K2 leads among open-source models with 20.1% overall in Table 2 (described as 21% in the abstract and as 20% in the conclusion), and the paper highlights it particularly on Adaptability (24.0 in Table 2).

• Execution and Search are easiest: These splits are described as the easiest, consistent with prior benchmark saturation. GPT-5 (high) scores 69.2 on Execution and 79.6 on Search; Claude-4-Sonnet scores 57.9 and 59.8 respectively; Claude-4-Sonnet Thinking scores 62.1 and 60.6.

• Ambiguity and Adaptability remain hard: Only Claude-4-Sonnet and GPT-5 (high) are described as achieving robust performance. GPT-5 (high) scores 51.9 on Ambiguity and 40.4 on Adaptability; Kimi-K2 scores 24.0 on Adaptability.

• Time is where frontier models break: Only Gemini 2.5 Pro (7.3) and Claude-4-Sonnet (8.1) achieve meaningful scores on Time, which the authors link to efficiency-latency advantages. GPT-5 (high), Qwen3-235B Thinking, and Grok-4 all score 0.0 on Time in Table 2.

• Removing generation latency helps most for reasoning models: In "instant" mode, Sonnet rises from 8.1% to 26.7% on Gaia2-Time and GPT-5 (high) from 0.0% to 34.4%. The paper reports inverse scaling on Time: models trade Time performance for Execution performance due to longer thinking.

• Noise robustness lags: Most models score below 20 on Noise, with GPT-5 (high) reaching 35.4%.

• Agent2Agent collaboration benefits weaker models more: Collaboration results in a lower incidence of tool call errors for lighter-weight LLMs, and increasing the collaborator ratio "r" improves pass@k scaling for Llama 4 Maverick, but for Claude 4 Sonnet increasing "r" does not improve cost-normalized performance under best-of-k sampling. The paper notes a weak negative effect of collaboration ratio on tool call error frequency for Claude 4 Sonnet.

• Heterogeneous teams beat uniform ones (Gaia2-mini, r=1): Llama-main + Llama-app scores 8.5 ± 1.7; Llama-main + Claude-app 16.2 ± 0.7; Claude-main + Llama-app 18.3 ± 0.7; Claude-main + Claude-app 29.3 ± 2.9.

• The verifier is highly accurate: On 450 hand-labeled validation trajectories, the ARE Verifier achieves 0.98 agreement, 0.99 precision, and 0.95 recall, versus 0.72 agreement, 0.53 precision, and 0.83 recall for an in-context LLM-judge-only verifier.

• Cost and latency trade-offs: Claude 4 Sonnet costs roughly 3× more than GPT-5 (low) for comparable accuracy but operates much faster. Grok-4 is described as inefficient and Kimi-K2 as cost-effective. An average human annotator can solve every task but is slower than all models, partly due to using ARE's GUI rather than a native OS.

• Behavioral drivers: Performance correlates positively with number of tool calls and with output tokens. Claude-4 Sonnet and Kimi-K2 are noted as outliers achieving high performance (35% and 21% respectively as written in that section) while producing relatively few tokens. App usage patterns were nearly identical across models.

• Parallel tool calling does not fix performance: A Parallel Tool Calling (PTC) orchestration can improve efficiency (wall clock time and token usage) but not performance, which the authors say confirms the observed limitations are intrinsic to model capabilities rather than the scaffold.

Methodology in Plain English

The authors first built a platform, ARE, in which "everything is an event." Environments are collections of stateful apps plus a time manager and rules; events are timestamped, scheduled on dependency graphs, and logged; notifications selectively decide which events reach the agent's context; and scenarios specify an initial state plus a DAG of events and a verification method. Time advances in the simulation as events are introduced, and it is decoupled from the agent, so the environment clock keeps moving while the model is still reasoning.

On top of ARE they built Mobile, a consumer smartphone-like environment with twelve apps and 101 tools, where each "universe" is a complete state of all apps around a single user, populated with synthetic data seeded from PersonaHub personas and propagated across apps so that contacts, messages, and calendar entries stay consistent. Universes contain between 400K and 800K tokens of structured and unstructured content (excluding filesystem contents).

Human annotators used ARE's annotation interface to author scenarios that isolate one capability at a time, writing DAGs of write actions and environment events as ground truth, with multiple rounds of independent validation, consistency checks, automated structural guardrails, and post-hoc difficulty calibration.

For evaluation, all models ran under the same model-agnostic ReAct-style scaffold with one structured JSON tool call per step, augmented with pre-step hooks (injecting queued notifications before each LLM call) and post-step hooks (checking termination). Models were evaluated at full context length (≥128K tokens), temperature 0.5, and 16K token generation limits per turn; scenarios were run three times and terminated on 200 steps, context overflow, verification completion, or timeout. Notification verbosity was set to "medium." The ARE Verifier used Llama-3.3-70B-Instruct at temperature 0.

Why This Matters

Impact on research: The paper argues that action-level verification scales better than end-state comparison and supports fine-grained credit assignment, making Gaia2 directly usable for RLVR training rather than only for leaderboard scoring. It also documents failure modes such as "judge-hacking" and argues for cost-normalized evaluation metrics, since comparing model parameters or FLOPs alone inadequately reflects real deployment conditions.

Real-world applications (as motivated in the paper):

• Scheduled task products such as calendars and reminders, reflected in the Gaia2-Time split.
• Interoperable multi-agent systems, reflected in the Agent2Agent split inspired by the recently proposed Agent2Agent protocol.
• Consumer mobile assistants handling email, messaging, calendar, contacts, shopping, cabs, and files, through the Mobile environment.
• Domains the paper says the architecture naturally extends to: desktop automation, customer support, and web browsing.

Industry relevance: Frontier models reach around 42% overall, showing a substantial "sim2real" gap. The reported inverse scaling on time-sensitive tasks and the fact that collaboration helps weaker models more than frontier systems both point to orchestration and adaptive compute — not only model quality — as levers for production agent systems. The heterogeneous-team results suggest a compute-scaling axis where a strong main agent plans and cheaper app-agents execute.

Future Directions

• Adaptive compute: The paper calls for deploying fast, lightweight reasoning for routine tasks while reserving deeper deliberation for complex ones, given the inverse scaling observed on Time-sensitive tasks.
• Robust verifier design and hybrid rewards: The authors state that robust verifier design is critical for both evaluation and RL training, and that hybrid approaches combining scalar rewards with preference signals remain an open direction.
• Parallel orchestration: Some Time scenarios require concurrent actions within narrow windows that the single-threaded scaffold cannot fully express; parallel orchestration is described as a promising direction.
• Better multi-agent orchestration: The A2A results leave open how to scope sub-goals, avoid cascading errors, and decide when decomposition benefits outweigh coordination overhead; the authors also note they expect future RLVR-trained systems to eventually solve Gaia2.

Target Audience

Researchers and engineers working on LLM agents, agent benchmarking, reinforcement learning from verifiable rewards, and multi-agent orchestration. It is also relevant to practitioners building production assistants in consumer or enterprise environments who need to understand cost, latency, and reliability trade-offs, and to platform builders who want an extensible, environment-agnostic framework (ARE) for defining their own dynamic benchmarks.

Authors’ abstract

We introduce Gaia2, a benchmark for evaluating large language model agents in realistic, asynchronous environments. Unlike prior static or synchronous evaluations, Gaia2 introduces scenarios where environments evolve independently of agent actions, requiring agents to operate under temporal constraints, adapt to noisy and dynamic events, resolve ambiguity, and collaborate with other agents. Each scenario is paired with a write-action verifier, enabling fine-grained, action-level evaluation and making Gaia2 directly usable for reinforcement learning from verifiable rewards. Our evaluation of state-of-the-art proprietary and open-source models shows that no model dominates across capabilities: GPT-5 (high) reaches the strongest overall score of 42% pass@1 but fails on time-sensitive tasks, Claude-4 Sonnet trades accuracy and speed for cost, Kimi-K2 leads among open-source models with 21% pass@1. These results highlight fundamental trade-offs between reasoning, efficiency, robustness, and expose challenges in closing the "sim2real" gap. Gaia2 is built on a consumer environment with the open-source Agents Research Environments platform and designed to be easy to extend. By releasing Gaia2 alongside the foundational ARE framework, we aim to provide the community with a flexible infrastructure for developing, benchmarking, and training the next generation of practical agent systems.