OpenFlyV2: An onboard vision-language-action model for aerial perception and navigation

AI summary

Overview

Research area: Aerial robotics and embodied AI — specifically vision-language-action (VLA) models for unmanned aerial vehicles (UAVs).

Technical level: Advanced. The paper assumes familiarity with vision-language backbones, flow matching / diffusion-style action generation, reinforcement learning from relative rewards, and UAV control stacks.

Scope (one sentence): The paper presents OpenFlyV2, a single 1-billion-parameter vision-language-action model that runs entirely onboard a quadrotor to perform aerial scene understanding, language grounding, spatial reasoning, and closed-loop language-conditioned navigation at 10 Hz.

What This Paper Is About

Aerial robots today tend to be good at one thing at a time: either they understand scenes, or they follow language instructions, or they fly — and the heaviest computation often happens off the aircraft, which makes flight depend on a stable network link. The authors ask whether a single compact model can do all of these on the drone itself, using one shared vision-language representation to both answer questions about a scene and generate the flight waypoints that carry the drone toward a described landmark. To get there they had to solve two resource problems: how to remember where the drone has been without feeding a long image history into the model, and how to generate smooth waypoint sequences in very few computational steps.

Key Contributions

1. A unified onboard VLA architecture. OpenFlyV2 pairs a 0.8B Qwen3.5-VL backbone with a 0.2B action expert. The same shared multimodal context supports semantic understanding, visual grounding, spatial reasoning, and continuous flight control, with the output pathway selected by the task. One checkpoint is used for the vision-language, simulated navigation, and physical flight evaluations.

2. Motion-Adaptive Memory (MAM). Flight history is compressed into a bounded set of event-level tokens built from motion segments and their visual anchors, rather than a dense sequence of past images. This cuts vision-language input from 446 tokens to at most 218 (at least a 51.1% reduction) while representing a longer history, and reduces action-expert denoising on the Jetson Orin NX from 80 ms to 20 ms.

3. Motion-Structured Prior (MSP). Short-horizon UAV trajectories cluster around a few recurring motion patterns, so the authors cluster training chunks with K-means (K = 4) into motion prototypes and use a context-conditioned, adjusted prototype plus within-class covariance as the source distribution for flow-based generation. This lets the action expert produce a full 12-waypoint chunk with a single flow-integration step.

4. The OpenFlyV2-130K dataset and a full onboard demonstration. Data spans 13 environments over 254.81 km² (four AirSim, two Unreal Engine city scenes, and seven 3D Gaussian Splatting reconstructions of five university campuses and two post-disaster ruins), comprising approximately 70,000 navigation trajectories (72,352 in the Fig. 2B breakdown) and 60,006 vision-language instruction–response pairs. The model is then deployed on a DJI M30T with an NVIDIA Jetson Orin NX and flown zero-shot in four real environments.

Main Findings

• Onboard closed-loop flight at 10 Hz: The final policy generates 12-waypoint action chunks at 10 Hz entirely onboard the Jetson Orin NX, with asynchronous execution overlapping generation of a new chunk with execution of the previous one, while a low-level flight controller handles stabilization and waypoint tracking.

• Vision-language performance competitive with much larger models: With a 0.8B backbone, OpenFlyV2 is described as broadly comparable to GPT-5.5 and surpassing Qwen3.5-9B and Gemini 3 Flash across the five perception tasks. It exceeds GPT-5.5 on grounding accuracy (67.1% versus 61.6% Acc@0.5) and has lower distance-estimation error (47.32 versus 52.62 m mean absolute error; 0.27 versus 0.34 relative error). Description quality is essentially matched (6.38 versus 6.36 out of 10, scored by GPT-6.1-sol).

• Best simulated navigation among compared methods: On 800 seen and 300 unseen episodes (success requires an explicit stop within 25 m of the goal), OpenFlyV2 reaches success rates of 62.88% and 52.67%, versus 38.88% and 36.67% for DualVLN — gains of 24.00 and 16.00 percentage points. Terminal navigation error is also lower: 33.72 m versus 65.75 m in seen environments, and 38.81 m versus 57.98 m in unseen ones. It also posts the highest oracle success rate and SPL in both splits.

• Baselines could not run onboard: AeroVLA, DualVLN, the OpenUAV navigation model, and OmniNav retained their original designs and training procedures and were compared only in simulation, because none could run on the Jetson Orin NX used for physical flight.

• Memory compression helps navigation, not just speed: In the ablation, MAM improved all four metrics at both integration budgets. With one step, SR rose from 49.82% to 53.45% and NE fell from 49.18 m to 37.54 m; with two steps, SR rose from 56.36% to 59.45%.

• The motion prior pays off most under a one-step budget: Adding MSP to MAM raised one-step SR from 53.45% to 57.09% and reduced NE from 37.54 m to 33.55 m, and the combined single-step model beat the two-step base on all four metrics. At two steps, MAM alone remained strongest (SR 59.45%), so MSP's advantage was concentrated at the single-step budget.

• Zero-shot sim-to-real transfer: Across 100 physical trials (50 single-landmark and 50 two-landmark instructions) in two seen and two unseen locations, the model achieved an aggregate 48% success rate with no real-world aerial training data and no test-site fine-tuning. Successful flights included sequential landmark-directed goals and instruction-dependent turns.

• A remaining gap between reaching and stopping: For the combined single-step model, OSR (68.36%) exceeded SR (57.09%), indicating a gap between entering the goal region and terminating successfully.

Methodology in Plain English

One model, two output paths. The drone's current camera image, the language instruction, the robot state, and a set of memory tokens are fed into a shared vision-language backbone. If the task is a question, the backbone decodes text or image coordinates. If the task is flight, a small action expert reads the same encoded context and produces a chunk of 12 waypoints, each consisting of three body-frame motion increments plus a stop signal.

Remembering without drowning in images. Instead of stacking past frames, the system watches how the drone actually moved. When consecutive actions change sharply — or when a segment gets too long — it closes an "event." Each event stores the action segment plus the images at its start and end. Learned queries pull visual information out of those two anchor images, and an age encoding is added afterward so the fixed content can be cached and reused across prediction cycles while only the age is recomputed. The result is a fixed budget of at most M×P memory tokens, where M limits retained events and P limits capacity per event.

Generating motion in one shot. Offline, the authors normalize training trajectories by path length so shape is separated from distance traveled, cluster them into four prototypes, and assign every remaining chunk to its nearest prototype. At runtime, a small head predicts which prototype fits the current context and how much to stretch its horizontal extent, vertical profile, and accumulated yaw. That adjusted prototype plus within-class covariance becomes the starting sample for conditional flow matching; the action expert then refines it into the final chunk. One Euler update from flow time τ = 1 to τ = 0 is enough for deployment, and each such update still produces all 12 waypoints — action horizon and flow-integration budget are separate things.

Data. Environments came from AirSim, Unreal Engine, and 3D Gaussian Splatting reconstructions of real campuses and disaster ruins. Landmarks with semantic labels, 3D positions, dimensions, and point clouds were stored in a grid map, and trajectories were generated by sampling unoccluded start viewpoints and chaining landmark goals. GPT-5 wrote the navigation instructions and question–answer pairs; a pool of 200 human annotators verified them under strict criteria, and only 60% of generated samples passed.

Training. Stage one jointly trained on VLN and VQA for two epochs with four historical images plus the current image and no MAM or MSP. Stage two enabled both components and continued joint training for three epochs. A final reinforcement post-training stage adapted group-relative optimization to the stochastic flow policy, sampling 4,000 tasks from eight of the nine training environments with four rollouts per initial condition, updating only the action expert. Ablation variants in Table 1 all received five epochs of supervised training and were evaluated before RL.

Why This Matters

The paper argues that broad vision-language competence and continuous closed-loop flight do not have to be traded off against each other on a resource-constrained aerial platform, and that language-conditioned flight can be learned purely from rendered data and then transferred to a physical aircraft. That reframes onboard compute as a design constraint to be engineered around — through compact memory and structured action initialization — rather than a reason to offload inference.

Real-world applications:

• Language-guided inspection: flying to a described landmark and then answering questions about the view that results, which the authors highlight as a natural extension.
• Search and rescue in damaged sites: the dataset explicitly includes two post-disaster ruin reconstructions, and the model's distance estimation and spatial-relationship reasoning suit surveying cluttered, partially observable terrain.
• Field operations without connectivity: onboard-only inference means missions do not degrade when communication links are unavailable or slow.
• Human-to-robot tasking for non-pilots: operators specify goals in natural language rather than in waypoints or coordinates.

Industry relevance: The demonstration runs on a commercially available airframe (DJI M30T) with a Jetson Orin NX, which is the class of embedded hardware drone manufacturers already ship. A 1-billion-parameter model doing perception, reasoning, and control at 10 Hz on that budget is a concrete datapoint for teams deciding whether onboard generalist policies are deployable rather than merely demonstrable in simulation.

Future Directions

• Detecting and recovering from task deviation. The authors name this directly: recognizing that the drone has drifted from the intended task, recovering from the error, and resuming navigation under changed observations remain open problems, and the OSR–SR gap in the ablation points to a related weakness in knowing when to stop.
• Closing the reach-versus-stop gap. With OSR at 68.36% against SR at 57.09% for the combined single-step model, better termination decisions look like a tractable target.
• From single instructions to dialogue. The discussion proposes extending the interaction to multi-turn dialogue, reference tracking across requests, and concurrent navigation and question answering, pointing to aerial vision-and-dialog navigation as a related setting for studying clarification during flight.
• Broader evaluation across environments and conditions. The authors state that wider evaluation is needed to understand how these capabilities support practical applications, and that reliable real-world aerial VLN "remains a substantial challenge."

Target Audience

Researchers and engineers working on embodied AI, robot learning, and UAV autonomy — particularly those building vision-language-action policies and those concerned with deploying large models under tight onboard compute and latency budgets. It is also useful for robotics practitioners interested in sim-to-real transfer without real flight demonstrations, and for dataset builders interested in automated annotation with human verification (the 60% acceptance rate is a useful calibration point). Readers without a background in flow-based generative models, vision-language architectures, or reinforcement learning will find the methods section demanding.

Authors’ abstract

Human cognition flexibly draws on a shared perceptual foundation to understand environments, attend to referenced entities, reason about spatial relationships, and act. Endowing aerial robots with an analogous functional organization remains challenging, as existing systems are often specialized for individual capabilities, dependent on offboard computation, or unable to sustain closed-loop operation on resource-constrained platforms. We present OpenFlyV2, a unified onboard vision-language-action architecture that uses a shared vision-language representation to support semantic understanding, language-grounded perception, spatial reasoning, and continuous aerial control within a single model. A compact action expert converts the shared multimodal context into short-horizon flight waypoints, while motion-aware temporal memory and action priors reduce the computational cost of closed-loop execution. We construct \dataset{}, a simulation-based dataset comprising approximately 70,000 navigation trajectories and 60,000 vision-language instruction--response pairs. OpenFlyV2 generates action chunks, each comprising 12 waypoints, entirely onboard at 10 Hz using an NVIDIA Jetson Orin NX. It transfers zero-shot from simulation to physical quadrotor flight, achieving a 48\% navigation success rate across four real-world environments without real-world aerial training data. Extensive results demonstrate that broad vision-language competence and continuous closed-loop flight can coexist within a single model on a resource-constrained aerial platform. Code will be made publicly available at https://github.com/Ideal-ljl/AeroBrain.