AutoFly: Vision-Language-Action Model for UAV Autonomous Navigation in the Wild

AI summary

Overview

Research area: Robotics — Vision-Language Navigation (VLN) and Vision-Language-Action (VLA) models for unmanned aerial vehicles (UAVs), combining multimodal perception with autonomous flight control.

Technical level: Advanced. The paper assumes familiarity with vision-language models, action de-tokenization, reinforcement learning for data collection, and sim-to-real transfer.

Scope: The paper proposes AutoFly, an end-to-end VLA model plus accompanying dataset that lets a UAV navigate to coarse positional or directional goals in unknown outdoor environments using only RGB input, evaluated in both AirSim simulation and real indoor/outdoor flights.

What This Paper Is About

Existing VLN systems for UAVs assume that a human supplies detailed, step-by-step instructions that trace a predetermined route with explicit waypoints and maneuvers — an assumption that rarely holds in unknown outdoor terrain. In the real world, operators typically give only coarse positional or directional guidance, so the UAV must plan, avoid obstacles, and recognize targets on its own. AutoFly addresses this gap with an end-to-end VLA model that maps RGB observations plus short language instructions to low-level UAV control actions, together with a new dataset built for autonomous behavior rather than instruction-following.

Key Contributions

1. AutoFly, an end-to-end VLA model for autonomous UAV navigation. It directly outputs velocity commands for high-level action primitives (obstacle avoidance, object recognition, planning) based on coarse guidance, and adds a pseudo-depth encoder that derives depth-aware features from monocular RGB to strengthen spatial reasoning.
2. A progressive two-stage training strategy. Stage 1 performs vision-language alignment using the prism-siglip-7b configuration from Prismatic-VLMs with a two-layer projector and a 7B-parameter LLaMA2; Stage 2 jointly fine-tunes the pseudo-depth encoder with the VLA backbone, with the Siamese MLP projector selected as the best of three depth-integration paradigms.
3. A large-scale multimodal autonomous navigation dataset. It contains 13,476 trajectories across simulation and real-world outdoor environments, including 1K real-world flight episodes and over 2.5M image-language-action triplets, shifting the paradigm from instruction-following to autonomous behavior modeling.
4. New evaluation instrumentation. The paper introduces the average obstacle encounter metric (10 for their dataset versus "not reported" for all listed prior datasets) alongside standard success rate and collision rate, and reports consistent gains in simulated and real environments.

Main Findings

• Simulation performance: AutoFly achieves an overall success rate (SR) of 47.9%, collision rate (CR) of 21.9%, and path efficiency rate (PER) of 77.3%, versus OpenVLA at 44.0% / 24.5% / 75.1%, RT-2 at 41.9% / 26.0% / 73.7%, and RT-1 at 24.3% / 65.1% / 61.1%. This is a 3.9% higher success rate, 2.6% lower collision rate, and 2.2% improved path efficiency relative to OpenVLA.
• Scene and target generalization: AutoFly scores 55.4% SR on seen scenes versus 42.3% on unseen scenes, and 52.3% on seen targets versus 36.4% on unseen targets. OpenVLA scores 52.7% / 40.3% and 49.3% / 33.6% respectively.
• Real-world flight: AutoFly reaches a 60% success rate indoors (30% collision rate, 76.5% PER) and a 55% success rate in an outdoor campus forest with irregular trees and swaying branches (35% collision, 75.1% PER) — a 5% performance gap.
• Sim-to-real scaling: With indoor simulation-to-real ratios of 0K:1K, 5K:1K, and 10K:1K, success rates rise from 10% to 25% to 60%, and PER rises from 61.1% to 71.3% to 76.5%; collision rate initially rises from 40% to 65% then falls to 30%. The outdoor 10K:1K condition yields 55% SR, 35% CR, 75.1% PER.
• Pseudo-depth encoder ablation: Adding it improves SR from 44% to 47.9% and lowers CR from 24.5% to 21.9% (PER 75.1% to 77.3%).
• Depth projector ablation: The specialized projector (47.9% / 21.9% / 77.3%) substantially beats applying pre-trained SigLIP (42.4% / 23.1% / 64.8%) or DINOv2 (41.3% / 24.2% / 65.1%) directly to depth maps.
• Dataset rebalancing ablation: Without rebalancing, performance collapses to 16.6% SR, 32.9% CR, and 43.7% PER, versus 47.9% / 21.9% / 77.3% with it. The paper quantifies the original imbalance as a KL divergence of approximately 0.36 nats from a uniform distribution.
• Depth-vision-language alignment ablation: Siamese MLP projector (47.9% / 21.9% / 77.3%) outperforms the non-Siamese variant (43.3% / 25.3% / 68.2%) and direct depth input (26.7% / 31.9% / 45.9%), the latter attributed to the modality gap between continuous depth values and discrete LLM tokens.

Methodology in Plain English

The task is framed as learning a control policy that takes the current RGB observation, a language instruction, and coarse positional or directional guidance (encoded as an initial action) and outputs a low-level control action, producing a collision-free trajectory that respects the UAV's kinodynamic constraints and ends in front of the target.

The architecture has three parts. First, a vision-language model built on the LLaVA-based OpenVLA configuration. Second, a pseudo-depth encoder with two pieces: a depth generator that uses Depth Anything V2 to produce depth maps from RGB, and a depth projector that patches, linearly embeds, and aligns those maps into depth tokens matching the visual token space. The authors deliberately avoid depth cameras because AirSim's depth output is overly idealized and does not transfer well, and because RGB-only sensing reduces payload, cost, and hardware complexity. Third, an action de-tokenizer that maps discrete action tokens to continuous 3D action vectors, using the final 256 tokens of the LLaMA2 vocabulary as the action space.

Training proceeds in two stages: vision-language alignment, then spatially-informed robot action fine-tuning in which the depth encoder and VLA backbone are optimized jointly with cross-entropy loss, a learning rate of 2e-5 for the VLM backbone and 1e-4 for the pseudo-depth projector, over 80K gradient steps in an autoregressive paradigm.

For data, the authors build 12 AirSim scenes, each 70m × 70m, populated with irregular obstacles such as trees, walls, rocks, and buildings, with 60 object instances placed at environment boundaries and 3-5 distractor objects per scenario. Because human demonstration is too expensive at scale, they train Soft Actor-Critic (SAC) collection agents per environment until each reaches a 95% evaluation success rate; these agents handle point-to-point navigation only, so object placement is used to generate recognition-based trajectories. The final dataset mixes automated and expert demonstration trajectories and also stores UAV positional and attitude state. Long-horizon imbalance is corrected by segmenting trajectories into obstacle-avoidance versus target-seeking phases using Grounding DINO to query observations with instructions, then rebalancing exposure.

Success is defined as reaching within 5 meters of the target with angular deviation of 15 degrees or less. Evaluation uses three metrics: success rate, collision rate, and path efficiency rate, tested on 7,200 simulated flight episodes and 200 real-world flights, plus 20 independent trials per target on 10 object instances in each real setting.

Why This Matters

Research impact: The paper argues that the prevailing VLN assumption — detailed step-by-step instructions and high-fidelity maps — does not hold in unknown outdoor environments, and that treating planning, recognition, and obstacle avoidance as separate modules is suboptimal because dynamic adaptation requires coordinated deliberation. By integrating these primitives into a single end-to-end VLA model and by publishing a dataset whose instructions average only 12 tokens in length with a vocabulary of 147, it reframes the UAV navigation problem around autonomous behavior modeling. It also adds an obstacle-encounter metric to a field the authors say is "under-measured" on avoidance, and provides sim-to-real scaling evidence (0K:1K → 10K:1K) relevant to anyone studying simulation-to-reality transfer.

Real-world applications:

• Search and rescue, where operators can only give rough location guidance and the aircraft must plan and avoid obstacles itself.
• Environmental monitoring, including flight over unstructured vegetation such as the outdoor forest setting tested here.
• Autonomous delivery systems, enabled by coarse positional targets without detailed route specification.
• Infrastructure or disaster inspection, where an operator points to an object of interest and the UAV navigates to it while recognizing and avoiding surrounding obstacles.

Industry relevance: Because the system uses RGB only — no depth sensor — it reduces payload, cost, and hardware complexity for UAV deployments, which matters for commercial platforms with tight weight and power budgets. The reported 3.9% success-rate gain over state-of-the-art VLA baselines, with model, data, and code released at https://xiaolousun.github.io/AutoFly, gives practitioners a reproducible baseline. The paper's Appendix sections on distributed system architecture, network communication protocol, model acceleration, and parallel inference architecture suggest deployment engineering is part of the contribution, though those details are not contained in the main text provided.

Future Directions

• Closing the remaining sim-to-real gap: Indoor success is 60% and outdoor forest success is 55%, with collision rates of 30% and 35% — the paper's own visualizations show dynamic natural elements such as swaying branches as a persistent challenge.
• Improving unseen-target generalization: Success on unseen targets (36.4%) is far below seen targets (52.3%), indicating recognition and reasoning about novel goal objects remains a weakness.
• Incorporating richer state information: The dataset deliberately collects UAV positional data and attitude angles "to facilitate future research applications," implying state-aware policies are an open extension beyond the current RGB-plus-language-plus-action formulation.
• Extending beyond the current action space and degrees of freedom: The dataset is labeled as 3 DoF with average path length 107.43 m, while prior aerial datasets such as OpenUAV report 6 DoF — a design question the authors leave open. The paper also lists limitations and future work in Appendix A.7, whose content is not included in the provided text.

Target Audience

Robotics and embodied-AI researchers working on VLN, VLA models, and UAV autonomy; graduate students entering multimodal navigation who need a concrete architecture and dataset reference; and industry engineers building GPS-denied or coarse-command UAV stacks who want an RGB-only, sensor-light approach with released code and data. Readers without background in vision-language models, action tokenization, or reinforcement learning will find the method sections advanced.

Authors’ abstract

Vision-language navigation (VLN) requires intelligent agents to navigate environments by interpreting linguistic instructions alongside visual observations, serving as a cornerstone task in Embodied AI. Current VLN research for unmanned aerial vehicles (UAVs) relies on detailed, pre-specified instructions to guide the UAV along predetermined routes. However, real-world outdoor exploration typically occurs in unknown environments where detailed navigation instructions are unavailable. Instead, only coarse-grained positional or directional guidance can be provided, requiring UAVs to autonomously navigate through continuous planning and obstacle avoidance. To bridge this gap, we propose AutoFly, an end-to-end Vision-Language-Action (VLA) model for autonomous UAV navigation. AutoFly incorporates a pseudo-depth encoder that derives depth-aware features from RGB inputs to enhance spatial reasoning, coupled with a progressive two-stage training strategy that effectively aligns visual, depth, and linguistic representations with action policies. Moreover, existing VLN datasets have fundamental limitations for real-world autonomous navigation, stemming from their heavy reliance on explicit instruction-following over autonomous decision-making and insufficient real-world data. To address these issues, we construct a novel autonomous navigation dataset that shifts the paradigm from instruction-following to autonomous behavior modeling through: (1) trajectory collection emphasizing continuous obstacle avoidance, autonomous planning, and recognition workflows; (2) comprehensive real-world data integration. Experimental results demonstrate that AutoFly achieves a 3.9% higher success rate compared to state-of-the-art VLA baselines, with consistent performance across simulated and real environments.