Evaluating Multimodal LLMs as Generalist Vision-Language-Action Agents for Drone Control: Commanding, Approaching, Tracking and Searching

AI summary

Overview

Research area: Robotics / embodied AI — specifically, using Multimodal Large Language Models (MLLMs) as the decision-making "brain" of a quadrotor drone, with the action space declared entirely in a text prompt rather than learned into model weights.

Technical level: Intermediate. The core ideas (prompting, closed-loop control, success criteria) are accessible; the benchmark design, protocol terminology and metric tables assume some familiarity with embodied AI evaluation.

One-sentence scope: The paper introduces DroneCATS-Agent (a model-agnostic drone agent with a prompt-declared action space) and DroneCATS (a 100-episode benchmark), then uses them to test nine MLLMs across approaching, tracking, searching and multi-drone commanding, finding that small open models often navigate well but fail to follow the arrival-declaration protocol.

The work comes from the Drone AI Team at NAVER Cloud, with code at github.com/naver-ai/DroneCATS.

What This Paper Is About

Most prior drone-control systems with MLLMs deliberately shrink what the model is responsible for: one system uses the model only as a language planner over detections, another makes it point at a pixel while a classical planner flies, and others add verifiers or LiDAR planners so the model's output never reaches the actuators unfiltered. Because those papers vary the system while holding the model fixed, they cannot say which parts of embodied drone control actually depend on the model.

This paper flips the experimental design: it holds the agent fixed and varies the model instead. It drops an MLLM directly into the control loop with its entire action space written in the prompt, gives it only an egocentric monocular RGB camera, and asks it to make four decisions — fly toward a point, yaw to search, pause to deliberate, and declare its own arrival — with no fine-tuning and no function-calling schema. The goal is to measure how far MLLM perception extends into action, and specifically to find what breaks first at the small, deployable end of the model scale range.

Key Contributions

1. DroneCATS-Agent, a model-agnostic drone agent. It defines a pointing interface with four actions (go, rotate, think, finished) that make searching, deliberating and stopping the model's own decisions, all declared in the prompt, so any MLLM that can read it drops in unchanged.

2. A unified closed-loop protocol for approaching, tracking and searching. A single success criterion covers all four cells of the task grid, anchored on the model's own arrival declaration, so scores are comparable across cells and cannot be earned by merely drifting through the goal region.

3. An evaluation spanning commanding, approaching, tracking and searching. Nine models fly over 80 single-drone episodes on two maps, plus a commanding suite of 20 episodes at N=4 in which one context flies four drones toward look-alike candidates, scored by pooling the fleet's declarations. A three-flight variance audit and a per-model failure taxonomy bound what the numbers support.

4. The declaration finding, which localises the small-model gap in protocol adherence rather than navigation: these models reach the target and then misuse the action that would end the episode.

Main Findings

• Approaching is not solved. Flying to a target visible from the first frame succeeds in 13 of 20 episodes for the best model (65% in the results table). Withholding the target from the first frame drops that model to 40%, and no model exceeds 40% on searching.

• Model ordering is unstable across cells. GPT-5 approaches nearly as well as Gemini 3.7 Flash, within one episode of it (60 versus 65), but tracks a moving target in only 15% of episodes, where Gemini 3.7 Flash reaches 80%. The model that navigates to a static target is not necessarily the one that can keep a moving one.

• Embodiment specialisation buys no visible headroom. Gemini Robotics-ER 2 averages 47.5% across the four cells against Gemini 3.7 Flash's 57.5%, a gap comparable to the run-to-run spread reported in the variance audit.

• The Qwen3.5 ladder is monotone in scale, at 33.8%, 21.3%, 12.5% and 0% for the 27B, 9B, 4B and 2B sizes respectively.

• The failure is declaration, not navigation. For frontier models, reaching within the 5 m success radius and declaring arrival track each other. For small open models they diverge. Qwen3.5-9B passes within the success radius in 90% of episodes — more often than any frontier model — but converts only 35%, declaring on average at 0.63 of the start distance.

• Small models fail at both ends of the same competence. Qwen3.5-2B declares in 25% of episodes at 1.28 of the start distance, announcing arrival without having closed any distance, and never succeeds. Cosmos3-Edge-2B flies — entering the success radius in 25% of episodes and closing half its start distance on average — and never declares once. Frontier models declare at 0.20 to 0.39 of start distance.

• Navigation is not the small-model constraint. The 2B and 4B models still close 57% and 62% of the initial distance. The per-cell oracle success rates show the gap is not an approaching artefact: Qwen3.5-9B's trajectory enters the success radius more often than it succeeds in every cell, and Qwen3.5-2B enters it in all four cells and converts none.

• Commanding amplifies the divide. GPT-5, within one episode of the best approacher in the one-drone setting, succeeds in 20% of commanding episodes; Gemini 3.7 Flash succeeds in 80%. Small models fail by blindly copying a single coordinate across distinct views — Qwen3.5-2B declares in 85% of commanding episodes but succeeds in only 5%, while Cosmos3-Edge-2B declares in 0%.

• Passing through the goal region is not success. Among the never-declaring failures in the runs, 61 of 337 had entered the 5 m radius at some point, so crediting trajectory proximity would have inflated the 182 successes by a third.

• think helps rarely. In one worked example, Qwen3.5-9B — described as the roster's best case, not a typical one — is lost and drifting out to 15 m, reads the scene correctly, turns round, re-acquires the car and reaches 2.0 m. In another, Qwen3.5-4B spends five think steps in a row at 5.0 m from the quadrotor it was told to follow; four of five thinking-on replies are the single line {"action":"think"}, the fifth a 90° turn, after which it aims past the target at the tree line (6.6 m ahead, with the chosen pixel being background at 217 m) and never declares.

• The 2B deployment regime is expensive. Serving Cosmos3-Edge-2B with a 32k context takes 10.5 GiB and 0.44 s per 30-token decode, about 2 Hz — measured with vLLM at bfloat16 on one H100, a ceiling no aircraft carries. Qwen3.5-4B needs 13.9 GiB even though its hybrid linear attention makes context nearly free.

Methodology in Plain English

The researchers built a drone agent whose "action space" exists only as text in a prompt. At each step the model receives the drone's egocentric RGB frame, the instruction, and the last five actions, and must return exactly one JSON object. Four actions are available: go (point at an image pixel, normalised to 0–1000 on both axes, plus a depth estimate in metres); rotate (yaw, asked to stay within ±90° per step); think (hover and deliberate); and finished (declare arrival). No model is fine-tuned, and none uses a function-calling schema. The model's only sensor is the camera; even the depth it commands is its own estimate rather than a physical reading, and camera intrinsics are not provided, so depth is treated as a step-size proposal rather than calibrated metric depth.

Everything below the chosen action is fixed and model-agnostic. A rule-based geometric controller back-projects the pixel and depth into a 3D displacement and hands it to the flight stack as a body-frame velocity setpoint; rotate becomes a yaw rate at held position; think and finished hold position. The flight controller's standard cascade (velocity to attitude to rotor commands) is the drone's own and is untouched by the paper. Crucially, a finished declaration does not end the episode — the scaffold records it, keeps flying, and a verifier judges afterwards.

The benchmark varies the task along two binary axes: whether the target moves, and whether it is visible in the first frame. Their product gives four task types — approaching, searching, tracking, and search-and-track. Moving targets travel at 0.3 m/s on the residential map and 0.15 m/s on the campus map, along one-way paths longer than the distance the target can cover in an episode. Episodes run in AirSim on Unreal Engine maps, capped at 300 s, with start distances of 12–48 m.

A single success criterion scores the whole grid: an episode succeeds if at least one arrival declaration was made while the drone was within δ = 5 m of the target in three dimensions and the target was visible to the camera. All declarations are counted, not just the last one, because the scaffold keeps flying and later declarations are measured too. Judging is post hoc from two logs — the declaration events and a 2 Hz pose trace — so δ can be changed without re-flying.

For the organisational axis, the commanding suite puts four drones' views into one context in a fixed order, and the model emits one command per drone in a single response. The scene holds several candidates of the same asset, alike except for an inscription legible only from close range (a licence plate on the residential map, a written sign on Blocks), and the instruction names it. The same criterion applies, pooled across all four drones' declarations. Twenty episodes run at N=4, ten on the residential map and ten on Blocks, for a benchmark total of 100 episodes.

The evaluated roster is four frontier API models (GPT-5, Claude Opus 5, Gemini 3.7 Flash, Gemini Robotics-ER 2), the Qwen3.5 family at 2B/4B/9B/27B, and Cosmos3-Edge-2B. The 2B and 4B sizes were served locally; everything larger went over an API.

Why This Matters

Impact on research: The paper argues that prior systems report backbone-insensitivity (Fly0 found four very different backbones within 1.1 points of one another, at 70.4% to 71.5% success, with a 3B model trailing the best by fewer than 6 points) because the parts that actually separate models have been engineered out of the loop rather than measured. By varying the model and holding the agent fixed, DroneCATS reframes what a failure can mean: with a trained action head a failure is a failure of the policy, while under a prompt-declared action space it can also be a failure to follow the declaration. The paper reports that the second kind dominates below frontier scale.

Real-world applications:

• Autonomous drone delivery or inspection, where a drone that cannot reliably recognise and report its own task completion cannot be trusted to operate without a human watching.
• Edge deployment on battery- and compute-limited aircraft, since the benchmark deliberately scales down to 2B parameters and measures serving cost (10.5 GiB, about 2 Hz for a 32k-context 2B model on an H100).
• Search-and-rescue, where a target outside the initial camera view must be found by sweeping, and the agent must know when the search has succeeded.
• Multi-drone fleet operations, where a single model context allocates several platforms to several look-alike candidates rather than converging on one.

Industry relevance: The finding that navigation holds up while the action protocol does not is directly actionable for anyone building on-device agents: the bottleneck is not flying, but the discipline to sustain a declared protocol and emit the correct terminating action. As the paper puts it, what separates a deployable edge model from a frontier model is that discipline, and the open problem is closing the gap at onboard compute costs.

Future Directions

• Close the declaration gap at onboard compute cost. The stated open problem is yielding a fast model that plans persistently and knows exactly when it is done, at rates a drone can actually carry.
• Evaluate commanding through a hierarchy. A per-drone sub-agent under a commander is the setting the agent is built to support — the paper argues it is expressible only because completion is the agent's own claim — and the evaluation is explicitly left to future work.
• Extend the scale axis and the model set. Only one family (Qwen3.5) is compared at four sizes, and the paper notes that scale is marked as covered when one family is compared at three or more sizes; broader families and more embodiment-specialised models could test the monotone-scale finding.
• Improve the reasoning step. The think action helped in one worked example and was wasted in another, with five consecutive think steps producing no progress — how to make deliberation pay off in a closed loop where every token costs latency is unresolved.

Target Audience

Researchers and engineers working on embodied AI, vision-language-action models, and autonomous drone or UAV systems — particularly those interested in prompt-declared action spaces, benchmark design for closed-loop control, or deploying MLLMs on compute-constrained hardware. It is also useful for practitioners building multi-robot or multi-agent systems that require agents to self-report task completion, and for anyone comparing frontier versus small open models on robotic tasks. Readers without background in embodied evaluation will find the quantitative tables dense, though the central argument — that the failure is the protocol, not the flying — is stated plainly.

Authors’ abstract

Multimodal Large Language Models (MLLMs) are strong perceivers of images and video. We ask how far that reach extends into acting: dropping an MLLM directly into a drone's control loop, with its entire action space declared solely in the prompt. Recent systems approach this setting but increasingly narrow the model's decision-making. We widen it back. We introduce DroneCATS-Agent, an architecture where the MLLM is a swappable component, and DroneCATS, a benchmark treating the model as the independent variable. Beyond merely flying toward a pixel, our agent entrusts the model to yaw and search, deliberate when unsure, and self-declare arrival---all without fine-tuning or function-calling schemas. Evaluating frontier and open models across four core capabilities---approaching a visible target, tracking a moving one, searching outside the initial view, and commanding a multi-drone fleet---reveals that even the simplest embodied settings are far from solved. Crucially, to identify what breaks first at the edge, our roster scales down to 2B parameters. The findings expose a stark paradox: it is not the flying that fails. Small open models often navigate into the success radius more reliably than frontier models, yet lose the episode by declaring arrival prematurely or not at all. Multi-drone commanding amplifies this divide, with small models failing by blindly copying a single coordinate across distinct views. Viewed as vision-language-action agents, the models' spatial perception holds up, but their action protocol does not. What separates a deployable edge model from a frontier model is not navigation, but the discipline to sustain a declared protocol and emit the correct terminating action. The open problem is closing this gap at onboard compute costs---yielding a fast model that plans persistently and knows exactly when it is done---and DroneCATS is built to measure that distance.