Dual Exam: Controlled Evaluation of Multimodal Agents and World Models

AI summary

Overview

Research area: Artificial Intelligence — evaluation methodology for multimodal agents (MAs) and world models (WMs).

Technical level: Intermediate. The framework and metrics are explained in plain terms, but readers will benefit from familiarity with agent benchmarks and video/image generation evaluation.

Scope (one sentence): The paper introduces Dual Exam, a benchmark framework that evaluates multimodal agents and world models separately under controlled conditions — a fixed environment response mechanism for agents and fixed action interventions for world models — across physical, digital, social, and scientific domains, rather than judging them only through coupled rollouts.

What This Paper Is About

World models are promoted as a way to help multimodal agents by simulating experience and predicting the consequences of actions, and agent trajectories are in turn promoted as data for improving world models. The problem is that when such a coupled system fails, the end-to-end score tells you that it failed but not which side caused it. Dual Exam responds by splitting the evaluation in two: agents are tested against a frozen reference environment so that only their action selection, verification, and recovery are measured, and world models are tested with prescribed action sequences so that only their ability to realize those actions while preserving task-relevant state is measured.

Key Contributions

1. A dual-track evaluation framework. Dual Exam evaluates Multimodal Agents under a fixed environment response mechanism and World Models under fixed action interventions, with both tracks verifying recorded execution evidence against explicit task specifications (initial condition, permitted intervention, required change, protected state, evidence rule) rather than self-reported completion.

2. A large agent benchmark. 10 agent configurations are compared on 1,088 evaluation items organized into 15 reporting categories across four domains (524 Physical, 255 Digital, 200 Social, 109 Scientific), with an explicitly defined case-count-weighted Index and a reported equal-domain-weighting alternative.

3. A large world-model benchmark. 16 world-model configurations are evaluated on 400 cases across eight scenarios with 50 cases per scenario, with full-coverage scores kept separate from partial-coverage scores.

4. A 150-case pilot study and trajectory audit. One MA–WM pairing (GPT-5.6 Sol with Seedance 2.5) is run on 150 cases across ten scenarios to demonstrate why coupled outcomes cannot diagnose failure, including a step-by-step packing rollout analysis.

Main Findings

• The coupled pilot produced zero task success. GPT-5.6 Sol interacting with Seedance 2.5 on 150 cases (15 per scenario, one attempt of at most 20 steps, no episode-level retries) yielded 0.0% Task Success overall, a World State Consistency of 26.6%, and an Agent Corrective Response of 9.2%. The authors stress that these aggregate outcomes cannot attribute failures to either component.

• The packing rollout shows locally plausible but globally invalid behavior. Over 14 steps, the agent kept issuing commands despite unconfirmed object acquisition, and later packing and carrying used a different blue pouch, violating the same-backpack requirement. The trajectory ends at 0.316 (6/19 state checks) and 0.286 (2/7 response events); both recorded responses were navigation retries, and none of the four identity discrepancies received a targeted response.

• Agent performance is highly category-specific. The highest displayed agent scores were 9.5 for Manipulation, 11.2 for Action Sequencing, and 5.6 for Desktop Application Interaction, while other settings reached 29.6 for Web Interaction, 42.9 for Mobile Application Interaction, and 47.1 for Game Environment Interaction — showing a strong aggregate score can coexist with very low execution-task scores.

• GPT 6 Astra leads the Agent Index, but not everywhere. Under the selected case-count weights, GPT 6 Astra attains the highest Agent Index at 31.3, followed by GPT 5.6 Sol at 20.7, Gemini 3.6 Flash at 20.1, Qwen 3.8 Max at 20.0, and Kimi K3 at 19.8. Four configurations lead at least one category: GPT 6 Astra leads nine, DeepSeek V4.1 Flash three, GPT 5.6 Sol two, and Seed 2.1 Pro one.

• Aggregate ranking depends on the domain mixture. Under equal-domain weighting, DeepSeek moves from sixth to second and Kimi from fifth to fourth, while GPT 6 Astra remains first.

• Scientific scores mix distinct evaluation units. Across 109 Scientific items, GPT 6 Astra has the highest within-domain case-weighted score of 50.6, ahead of DeepSeek V4.1 Flash at 44.0 and GPT 5.6 Sol at 42.2. Category leaders differ: Astra leads Mathematics & Computation (60.8) and Physical Sciences (52.2), Sol leads Health & Medicine (45.4), and DeepSeek leads Life Sciences (53.7). The 109 items combine 63 ALE tasks, 30 SWE-bench repair tasks, ten derived ToolChoice checkpoints, and six EdgeBench optimization tasks.

• GPT-Image-2 leads full-coverage world-model scores. GPT-Image-2 tops the full-coverage WM Overall at 21.7, ahead of Qwen Image 3.0 at 21.1 and Nano Banana 2 at 19.5, with the remaining full-coverage configurations ranging from 17.3 down to 7.8. Seedance 2.5 scores 18.7 but only on six of eight scenarios (300 cases), and is excluded from the full-coverage ranking.

• Digital scenarios are a shared weakness. No evaluated configuration exceeds 16.5 on Web or 13.3 on Game. MiniMax H3 scores 53.8 on Embodiment but only 3.7 on Web and 8.5 on Game; Qwen-Image-3.0 scores 46.1 on Daily Life but 10.7 on Web and 11.5 on Game.

• Overall world-model rank does not predict suitability for a partner agent. GPT-Image-2 leads overall (21.7) over MiniMax H3 (17.3), but the ordering reverses on Embodiment (20.9 versus 53.8), because WM Overall weights all eight scenarios equally while a given agent faces a different distribution of states and actions.

• Action responsiveness must also preserve accumulated state. Illustrative failures show Qwen-Image-3.0 restoring the apparatus view in a pendulum case but introducing extra cups in a cup-stacking case, and MiniMax H3 preserving cup count while adding an unrequested red tray, whereas GPT-Image-2 preserved the original tray colors.

Methodology in Plain English

The researchers first ran a small pilot: they let one agent drive one world model through 150 tasks in a simulated interaction and audited the trajectories. Every case failed, and the recordings showed state errors and weak corrective responses — but the audit could not say which side was responsible. That motivated splitting the problem in two.

In the agent track, the environment's responses are fixed by a reference mechanism. The agent picks actions from its observation-and-action history, the reference environment returns the next observation, and task-specific verifiers check outcomes against explicit goals and constraints. Tasks are scored on a 0–100 scale that may mean strict completion or graded performance depending on the task, and the overall Index is a case-count-weighted average across 1,088 items. For the scientific subset, the authors combine published ALE category means with strict binary outcomes for SWE-bench and ToolChoice items and graded EdgeBench scores, applying an explicit clipping convention for Borden and DABIC that the authors note is their own, not an official EdgeBench rule.

In the world-model track, the actions are fixed instead. Each case specifies a starting scene, a prescribed intervention sequence, and protected state, and the model must generate observations that realize the requested effects while preserving identities, shared-state consistency, and active rules. Each case is scored against a rubric; the WM Overall is the unweighted mean of the eight scenario scores, and partial coverage (as with Seedance 2.5, which lacks Daily Life and Human Interaction due to face-generation restrictions) is explicitly barred from the full-coverage ranking and not treated as zero.

Crucially, the two tracks share specification principles but not tasks, metrics, or data, and the authors state plainly that the resulting Agent Index and WM Overall are not directly comparable.

Why This Matters

Impact on research: Most prior benchmarks measure one side of the loop — agent benchmarks such as GAIA, AgentBench, WebArena, VisualWebArena, OSWorld 2.0, and Agents' Last Exam, and world-model benchmarks such as VBench, WorldScore, WorldModelBench, MIND, WorldArena, Omni-WorldBench, WorldMark, WBench, WorldExam, and PlayWorld. Dual Exam's distinguishing claim is component isolation: holding one side fixed so that the other can be measured, while also covering all four of the physical, digital, social, and scientific domains and dedicated persistent-state checking.

Real-world applications:

• Web and UI automation: the paper's finding that no world model exceeds 16.5 on Web indicates how far simulated browser environments are from being trustworthy for agent training.
• Robotics and embodied planning: embodiment scores span 53.8 down to 1.1 across configurations, with directly different implications for which model to use in a manipulation pipeline.
• Scientific workflow automation: the Scientific domain mixes code-repair, computation, and optimization tasks, mirroring real research-assistant deployments.
• Content and simulation generation pipelines: the image-vs-video modality split and the "plausible still image cannot establish timing or an oscillation period" observation are directly relevant to quality control for generative video used as a simulator.

Industry relevance: The paper gives model developers and integrators a reproducible way to report component-level profiles rather than a single headline number, and it demonstrates concretely why choosing a world model by its overall leaderboard position can mislead when the partner agent visits an uneven distribution of states.

Future Directions

• Testing whether component scores predict coupled success. The authors repeatedly state that whether these capabilities predict coupled performance remains an empirical question requiring matched interaction experiments, which the pilot is explicitly not.
• Validating reciprocal improvement over successive training rounds. The arrows in the framework figure describe potential uses of the interaction, not demonstrated gains from joint training; a co-evolution study would need evaluation across multiple rounds.
• Closing the digital-domain and persistent-state gaps. Web scores capped at 16.5 and Game at 13.3 across all configurations, and memory consistency and action control are named as open world-model weaknesses.
• Extending observation interfaces. Text-only world models are explicitly out of scope and would require a separately specified observation interface and rubric, and the authors note that per-configuration histories, durations, budgets, and missing provenance limit cross-modality interpretation.

Target Audience

Researchers and engineers building or benchmarking multimodal agents, world models, or agent–simulator pairs; evaluation and safety teams who need to attribute failures to a component rather than to a whole system; and practitioners choosing a world model to pair with an agent, who will benefit most from the scenario-level profiles and the warning that overall ranks can reverse on individual scenarios. Readers new to the area can follow the framework description, but the leaderboard tables assume comfort with benchmark aggregation conventions.

Authors’ abstract

World Models (WMs) can support Multimodal Agents (MAs) through simulated experience, while Agent trajectories can inform World Model refinement. Evaluating this interaction requires separate evidence about action selection and state evolution. We introduce Dual Exam, which tests Agents under fixed environment response mechanisms and World Models under fixed action interventions across physical, digital, social, and scientific domains. The Agent track compares 10 configurations on 1,088 items in 15 reporting categories. The World Model track contains 400 cases across eight scenarios; 15 configurations have full coverage and one covers six scenarios. GPT 6 Astra leads the case-weighted Agent Index at 31.3, and GPT-Image-2 leads the full-coverage WM Overall at 21.7. These scores use different tasks and metrics and are not directly comparable. A 150-case pilot of one coupled pairing records zero task successes and motivates component-level diagnosis. Results reveal uneven task performance and failures to preserve task-relevant state. Their value for predicting coupled success or sustained reciprocal learning remains to be established.