Zero-Shot Mission-Level Evaluation for Aerial MLLM Agents

AI summary

Overview

Research area: Embodied AI and multimodal large language model (MLLM) agents, specifically zero-shot evaluation of UAV (drone) mission execution in simulated 3D aerial environments.

Technical level: Advanced. The paper assumes familiarity with embodied agent benchmarks, vision-language navigation, closed-loop evaluation, and MLLM prompting interfaces.

Scope: The paper introduces MissionBench, a 120-mission benchmark with a closed-loop evaluation framework, and uses it to measure how well 22 frozen, off-the-shelf MLLMs can plan, navigate, position, and report on aerial missions from a single natural-language instruction.

What This Paper Is About

Existing aerial benchmarks mostly test path following, target-distance reaching, or isolated sub-skills such as spatial perception, so they cannot show whether a frozen general-purpose MLLM can coordinate perception, planning, viewpoint control, and reporting inside one episode. The authors build MissionBench to ask that question directly: given only a single high-level directive and egocentric RGB observations, can an untuned MLLM complete a full aerial mission? The result is a measurement of how large the gap remains between general-purpose models and human operators.

Key Contributions

1. MissionBench, a benchmark of 120 mission-level aerial tasks across five high-fidelity environments (Neighborhood, City, Forest, Savannah, AirSimNH) and four task families (Reporting, Inspection, Manipulation, Patrol), each defined by one high-level instruction that requires integrated navigation, viewpoint selection, and reporting.
2. A closed-loop evaluation framework that disentangles mission understanding from spatial execution, using Success Rate (SR), Mission Progress (MP), Oracle Success Rate (OSR), Collision Rate (CR), and Step Efficiency (Eff).
3. Empirical analysis of 22 MLLMs, establishing a large human-model performance gap and showing that successful mission completion requires multifaceted capabilities rather than single-frame spatial perception alone.
4. A design ablation over sampling temperature, input resolution, image history length, and required structured output fields, clarifying which interface choices drive mission completion.

Main Findings

• Overall performance is low: Gemini 3.1 Pro achieves the highest success rate (SR, 34.8%), followed by Gemini Robotics 1.6 (32.2%) and Gemma-4-31B-IT (26.7%). Most remaining models stay below 10% SR. The random baseline reaches 0.0% SR.
• Mission progress tracks success: Gemini 3.1 Pro has the highest MP (73.3), followed by Gemini Robotics 1.6 (70.3) and Gemma-4-31B-IT (59.3). Even the best model succeeds on only about one out of three missions at the strict threshold.
• Large human-model gap: Human operators using continuous keyboard control reach 84.4% SR, 94.2 MP, and 51.1% OSR. Forcing human subjects through the same structured text-and-image interface as the models drops SR to 70.0% (MP 79.0, OSR 36.7, CR 0.1%, Eff 36.0), still far above the best model's 34.8%.
• Budget consumption is high: 17 of 19 models with non-zero SR use over 65% of the available mission-dependent action budget on average (Eff < 35%).
• Two distinct oracle gaps: For some models OSR > SR (Gemini 3 Flash: 20.2% OSR vs. 7.9% SR; Claude Opus 4.6: 10.0% vs. 3.3%), meaning they reached the target but failed the mission. GPT-5.4 shows moderate MP (45.0) with near-zero SR (2.0%) and OSR (0.0%), indicating progress that never converges.
• Scaling helps, but not monotonically: Gemini 3.1 Pro substantially outperforms Flash Lite on MP (73.3 vs. 24.0), and GPT-5.4 outperforms GPT-5.4 Mini (45.0 vs. 20.0). Within Qwen 3.5 dense models, MP rises from 18.1 to 53.2 and SR from 0.0% to 10.0% going from 2B to 27B. However, Claude Sonnet 4.6 beats Opus 4.6 on both MP and SR, and Qwen 3.6 variants do not uniformly improve on Qwen 3.5.
• Spatial awareness does not track task success: Collision rates range from 0.0% (Qwen 3.5 27B) to 7.5% (Qwen 3.5-35B-A3B). Gemini 3.1 Pro, the best overall performer, records a CR of 4.3%, above the median.
• Perception is a weak predictor: Single-frame localization and distance estimation correlate only modestly with mission performance (Pearson r_mIoU,MP = 0.463; r_RMSE,MP = −0.315). Gemini Robotics 1.6 has the highest mIoU (0.66) and strong mission results, but Claude Opus 4.6 has the lowest RMSE (10.7 m) yet underperforms, and Gemini 2.5 Pro reaches MP 54.2 and SR 8.9% despite near-zero mIoU (0.01). Perception is necessary but not sufficient.
• Three recurring failure modes: premature termination (declaring completion within the first few steps while far from the target), drift and oscillation (exhausting the step budget circling near landmarks without converging), and unmet mission objectives (reaching the target vicinity but misidentifying the object or its attributes, e.g., rhino as elephant).
• Ablation findings (5-mission Ablation split, Gemini 3 Flash): default temperature T = 0.7 gives the highest SR (57.1) with MP 68.4; T = 0.0 gives SR 40.0 / MP 68.1; T = 1.0 gives SR 46.7 / MP 59.8. Lowering resolution to 640×480 (SD) raises MP (75.3 vs. 68.4) but cuts SR (42.9 vs. 57.1). Reducing history to 1 image sharply lowers SR (28.6) and MP (33.1); extending to 5 images preserves MP (69.6) but lowers SR (30.8). Removing the reasoning field causes the largest drop (SR 20.0, MP 58.9), while removing the bounding box lowers SR (42.9) but keeps MP high (70.2).

Methodology in Plain English

The authors built five high-fidelity simulated environments on Unreal Engine 5 with Cosys-AirSim physics-based UAV simulation, then hand-designed 120 missions, each specified by a single natural-language instruction. Missions are grounded in real UAV operations: Reporting (e.g., read a license plate), Inspection (e.g., take a closer look at a burning ship), Manipulation (e.g., drop a package next to a tent), and Patrol (e.g., follow the main road). Ground-truth trajectories were produced by a human UAV operator solving each mission; ground-truth trajectory lengths span 13 m to 4,171 m, with a median of 93 m and a mean of 260 m. Missions start from a fixed pose where the target is only partially visible, so the agent cannot succeed from the initial frame.

At each step the agent receives the current egocentric RGB image plus up to two previous images and its action history, all at 1920×1080 resolution with a fixed camera pitch of −45°. It outputs a structured response containing a bounding box, reasoning, an action primitive with magnitude, any information to report, and a binary DONE flag. There are eight directional primitives (forward, backward, strafe left/right, ascend, descend, turn left/right), each usable with magnitude up to d_max meters or 3×d_max degrees; d_max is set to one fourth of the distance from the start pose to the final pose. The maximum action budget is T_max = 50 for patrol missions and T_max = 20 otherwise. Collisions are recorded but do not end episodes.

Success criteria are task-dependent but consolidated into one SR metric. Inspection requires the final position to be within τ_d = 5 m and orientation within τ_ψ = 15° of ground truth. Reporting checks whether the extracted information is correct via soft matching by an LLM-as-a-judge (validated with manual spot checks on 150 random samples) and does not depend on final position. Manipulation requires both spatial thresholds and the task-specific action. Patrol computes IoU between 20 m buffers around the ground-truth and generated trajectories, with success at IoU > 0.5. OSR records whether the agent passed within τ_d of the target at any point.

The 120 missions are split into a 10-mission Proxy set (Neighborhood, mixed types), a 5-mission Ablation set (Neighborhood, easiest subset of Proxy), a 30-mission Test set (Neighborhood, Forest, City; 17 Reporting / 7 Inspection / 3 Manipulation / 3 Patrol), and an 80-mission Held-out set (Neighborhood, Forest, City, Savannah, AirSimNH). Each mission is run three times per model and averaged. Twenty-two open- and closed-source MLLMs were evaluated with no aerial-specific fine-tuning, using identical prompt templates (temperature 0.7). API cost for the full Test split ranges from $19.2 (Nova Pro) to $71.9 (Claude Opus 4.6).

Why This Matters

Impact on research. The paper shows that single-frame perception benchmarks are a poor proxy for mission-level embodied competence, motivating closed-loop evaluation with task-level success criteria rather than distance thresholds. It also provides a held-out 80-mission split specifically reserved for future fine-tuning experiments on whether aerial-specific training closes the perception-execution gap.

Real-world applications.

• Reconnaissance and information extraction, such as reading text, counting objects, or identifying scene attributes from the air.
• Infrastructure inspection and search and rescue, where the agent must reach an observation-ready pose with the target clearly visible.
• Payload delivery, landing, or sample collection, where the agent must locate a spot and then execute a physical action.
• Perimeter security, traffic monitoring, and environmental surveillance, where the agent follows routes along roads, coastlines, or facility layouts.

Industry relevance. The authors argue the results carry both promise and risk: scaling alone produces non-trivial zero-shot embodied capability without task-specific training, which lowers engineering barriers but raises concerns about premature deployment. They note that current models rely on large-scale compute and typically cloud-based inference, making on-device execution infeasible and introducing latency, reliability, and connectivity constraints, so strong benchmark performance should not be read as real-world readiness. They also flag uncontrolled capability emergence from general-purpose pretraining as a challenge for validation and certification.

Future Directions

• Closing the perception-execution gap: use the reserved 80-mission Held-out split to test whether aerial-specific fine-tuning reduces the disconnect between what models perceive and what they execute.
• Sim-to-real transfer: extend MissionBench toward real-world evaluation, since a sim-to-real gap may persist and the benchmark does not capture sensor noise, dynamic obstacles, or adversarial conditions.
• Prompt and interface optimization: per-model prompt tuning may improve performance and reduce model-specific failure modes, since the study used standardized prompts for fair comparison.
• Rebalancing task coverage: the 120 missions skew toward Reporting (N=57) and Inspection (N=38) with fewer Manipulation (N=16) and Patrol (N=9) missions, so a more balanced or larger mission set would test whether the observed failure modes and scaling trends hold across all families.

Target Audience

Researchers and engineers working on embodied AI, MLLM-based agents, and aerial robotics who need a mission-level, closed-loop benchmark rather than a perception or navigation proxy. It is also relevant to safety and policy audiences assessing the deployment readiness of general-purpose models in UAV systems, and to benchmark designers interested in the ablation results on resolution, history length, and structured output.

Authors’ abstract

Multimodal Large Language Models (MLLMs) are emerging as core reasoning modules for embodied agents, yet it remains unclear how well general-purpose models can solve long-horizon embodied tasks from a single high-level instruction. We introduce MissionBench, a benchmark for mission-level evaluation of MLLMs in aerial 3D environments. It comprises 120 missions across five simulated 3D environments and four task families. Agents must autonomously plan, navigate, and report outcomes using only egocentric observations and its action history, without aerial-specific fine-tuning. Across 22 open- and closed-source MLLMs, the strongest model succeeds on fewer than 35% of missions compared to 84.4% human performance, highlighting the difficulty of multi-step embodied tasks. Despite large variations between model families, we observe gains from scaling, indicating that larger general-purpose models possess stronger zero-shot embodied capabilities. Our analysis shows that mission-level competence requires coordinating multiple capabilities beyond spatial perception, including multi-step planning and adaptive reasoning. This motivates closed-loop evaluation and highlights both the promise and risk of scaling-driven improvements for embodied AI.