On Your Own: Pro-level Autonomous Drone Racing in Uninstrumented Arenas

AI summary

Overview

Research area: Robotics — vision-based autonomous aerial navigation, with autonomous drone racing used as the benchmark task.

Technical level: Advanced. The paper assumes familiarity with visual-inertial odometry, Kalman/Extended Kalman filtering, Perspective-n-Point pose estimation, convolutional neural networks, model predictive control, and time-optimal trajectory generation.

Scope (1 sentence): The paper presents and evaluates a full perception, state-estimation, and control stack that lets a quadrotor race autonomously at professional-pilot level both in a motion-capture-instrumented arena and in an uninstrumented hangar where no ground-truth measurements were ever available.

What This Paper Is About

Autonomous drone racing has become the standard test for vision-based aerial autonomy, and prior state-of-the-art work showed autonomous drones can beat human pilots — but typically only in highly controlled, externally instrumented arenas. That makes the results hard to transfer to real field operations, where no motion-capture system exists and the environment is new. This paper asks whether the same level of performance can be reached "on its own" — using only onboard sensing, with no access to ground-truth data for fine-tuning the state-estimation and control modules — and tests it head-to-head against a world-champion human pilot.

Key Contributions

1. Pro-level racing performance in both settings: the authors achieve professional-level autonomous drone racing in a controlled instrumented arena (with external tracking for ground-truth comparison) and in an equally challenging uninstrumented environment where ground-truth measurements were never available.
2. A perception and control stack that avoids ground-truth fine-tuning: the stack does not require fine-tuning with ground truth for residual (drift) estimation and proved resilient to multiple lighting conditions.
3. A public pro-pilot dataset: release of professional human piloting data from the instrumented track in the same format used in prior work ([5]), adding six new flights by a world-champion pilot, totaling 240.77 s of flying time and 2342.98 m of traveled distance at a top speed of 21.29 m/s, hosted at github.com/tii-racing/drone-racing-dataset.

Main Findings

• Instrumented arena: autonomy beats all three humans on average but not on best lap. Against three professional pilots — Thomas Kund (Star23467), Krutharth M. C. (Ion FPV), and world champion Minchan Kim (MCK) — the autonomous system (both MoCap-assisted and VIO-only) surpassed the humans in average lap time and top speed, but fell short of one of the three pilots (MCK) on best individual result. The autonomous drone's average lap time was 4.44 s with MoCap and 4.65 s with onboard VIO alone, versus 7.71 s (Star23467), 6.51 s (Ion FPV), and 4.71 s (MCK).
• Instrumented arena: complete reliability. The autonomous system completed all runs with zero crashes, whereas the humans suffered multiple incidents (2 crashes for Star23467, 7 for Ion FPV, 5 for MCK; the autonomous drone had 0 in both MoCap and VIO configurations across 21 and 6 laps respectively).
• Uninstrumented arena: competitive but behind the champion. MCK achieved both the lower average lap time (5.80 s) and the lower best lap time (5.05 s) than the autonomous drone "on its own" (6.02 s average, 5.92 s best). The autonomous drone, which was 1.27% faster than MCK on average lap time in the instrumented track, gave up a 3.65% margin to the fastest human in the uninstrumented one.
• Autonomy was more consistent; humans more adaptable. In the uninstrumented track the autonomous lap-time standard deviation was 0.06 s versus 0.40 s for MCK. Crashes in the head-to-head races (2 for MCK, 4 for the autonomous system, 3 of which were collisions with the human pilot, who recovered in two of those cases) showed both the autonomous system's consistency "even to a fault" and the human's greater adaptability. The autonomous system was much more vulnerable to shared-track interactions.
• Context against prior literature. The paper's comparison table lists previous human-versus-robot racing work, with robot best results of 9.19 m/s / 12.00 s (2019 AlphaPilot), 19.44 m/s / 5.11 s against a human best of 21.54 m/s / 5.19 s with the robot winning (2023), and 21.83 m/s / 3.20 s against a human best of 9.58 m/s / 6.04 s with the robot winning (2024 dataset work). For this work the table lists a human best of 25.63 m/s / 5.04 s, a robot best of 21.15 m/s / 5.60 s, and the human as head-to-head winner, with external sensing marked as not used.
• Design choices that mattered. The smooth state estimate from the dual-stage filtering (T265 VIO drift correction plus an extended Kalman filter on the 500 Hz flight-controller IMU) enabled the aggressive MPC tuning needed for lap-time optimization; before it, state discontinuities caused instability and crashes. The state predictor used for delay compensation unlocked 3D maneuvers at more sustained speed. Together these were credited with crossing the boundary from pro-level to champion-level performance, corresponding to roughly a 2-second advantage on the reference tracks.
• Skill transfer without ground truth. The system was deployed in two additional uninstrumented venues with novel tracks — public demonstrations at IROS 2024 and the 2024 Abu Dhabi F1 Grand Prix — using only 80 additional fine-tuning images, camera exposure/gain adjustments, and newly generated time-optimal trajectories. There it outperformed professional pilot Ion FPV in both time-trial and head-to-head races, with the F1 track adding outdoor lighting as a challenge.
• Track-specific bottlenecks differ for humans and machines. Human pilots identified the Split-S as problematic (constrained space, placed near a wall) and the spiral section as pivotal for performance; for the autonomous system the hairpin between gates 2 and 3 was the limiting factor, becoming a failure point when trying to execute trajectories generated with a less conservative thrust-to-weight ratio.

Methodology in Plain English

Platform. The quadrotor is built partly on an open design, with the frame changed to carry a stereo camera. It uses T-motor F60 PRO V 2020KV motors and HQProp HeadsUp R38 propellers (chosen because professional pilots unanimously preferred lower blade pitch), giving a thrust-to-weight ratio of about 7 at full battery measured on a thrust bench. A custom damping mechanism reduces image blur. The companion computer is an NVIDIA Orin NX on an A603 carrier board running JetPack 5.1.2, in MAXN power mode. Pilots flew replicas of the same drone with the autonomy electronics replaced by lead-filled 3D-printed parts to match weight and weight distribution, plus an HDZero FPV system.

Seeing the gates. Everything is done on grayscale images from one camera lens. A YOLOv8n model (3.2 million parameters, one class, 640×640 input) finds gate bounding boxes, and a keypoint model based on MobileNetV3-Small (1.1 million parameters, 256×256 input, pretrained on ImageNet-1K) estimates the four inner corners. Both are exported to ONNX (opset v17) and TensorRT (v8.5) engines in FP16 to use the Orin's GPU, giving 24–30 ms per-frame latency.

Training without heavy human labeling. The authors deliberately minimize human supervision: models are pretrained on the grayscale version of an existing racing dataset, then auto-labeled and manually corrected for the instrumented track (3,412 frames). For the uninstrumented track, where no labeled data existed, they use model distillation — Grounding DINO is fine-tuned on just 80 manually labeled frames from human-piloted flights and then used to auto-generate gate labels for whole flights to fine-tune YOLO. As few as 80 human-corrected frames suffice to fine-tune for peak performance in a new environment.

Knowing where you are. The Intel RealSense T265 provides visual-inertial odometry, which drifts, especially under fast maneuvers. Since gates are the only predefined landmarks in drone racing, the system solves a Perspective-n-Point problem for each gate with four visible corners (OpenCV's SOLVEPNP_ITERATIVE, initialized by homography decomposition and refined with Levenberg-Marquardt) and feeds those measurements to a Kalman filter that estimates only the 3D positional drift of the VIO — deliberately not drift velocity, because of loop closure at the camera firmware level. Multiple gates seen in one frame are processed as simultaneous measurements. A second-stage Extended Kalman Filter then fuses the drift-corrected estimate with 500 Hz IMU data from the flight controller (a rate enabled by an optimized MultiWii Serial Protocol link at 1 MBaud, far above the 10 Hz typical of SBUS) to produce a smooth state estimate.

Deciding how to fly. Reference trajectories are generated offline by an open-source time-optimal trajectory generator that accounts for full rigid-body dynamics and actuator constraints (linear aerodynamic drag was excluded). Two waypoints per gate, offset ±0.4 m along the gate frame's x-axis (−0.4 m and +1.25 m for the Split-S) and centered in y and z, keep optimized paths from cutting through gate banners; a conservative TWR of 3.8 is used to absorb modeling error. An open-source MPC framework, with perception-aware objectives disabled and remaining weights tuned for tracking precision and robustness to noisy state estimation, computes Collective Thrust and Body Rates commands. A state predictor based on point-mass dynamics and previous commands compensates for command delay, and Betaflight's internal PID controllers convert the setpoints into rotor signals.

How it was tested. Two tracks were built with 7-by-7 ft gates (213.36 cm outer, 152.4 cm inner opening) matching prominent league standards. The instrumented track recreates the "Track RATM" in an arena 25 m long, 9.7 m wide and 7 m high, equipped with a 32-camera Arqus A12 Qualisys motion capture system tracking 6DoF poses at 275 Hz; the drone carried five 25 mm markers. The uninstrumented track recreates the "Track Split-S" in a larger hangar, with gates placed using a total station (position errors of a few centimeters, orientation errors of at most ten degrees, and a bottom gate 20 cm lower than the original because of a different interlocking system). Because there was no motion capture, quantitative evaluation there was limited to lap and sector times, measured with GoPro cameras filming at 240 fps. Head-to-head races covered three consecutive laps and started on an audio countdown.

Why This Matters

Removing the dependence on instrumented arenas and ground-truth fine-tuning is the step that separates a lab demonstration from a field-deployable system. Vision-based autonomy is what allows drones to work in novel, unstructured environments where traditional navigation methods may be unavailable, and this paper shows a racing-grade stack functioning under those conditions — including varying lighting from sunlight through hangar windows and, at the F1 demonstration, outdoor lighting.

Real-world applications the paper identifies:

• Agriculture
• Logistics and delivery
• Defense
• Infrastructure inspection
• Environmental monitoring

Industry relevance: Autonomous drone racing has been the de facto benchmark for vision-based aerial autonomy since the 2019 AlphaPilot AI Drone Innovation Challenge, and in car racing, leagues such as the Indy Autonomous Challenge and the Abu Dhabi Autonomous Racing League (which organizes "Man vs Machine" events for both drone and car racing) push the same performance frontier. Demonstrations at IROS 2024 and the 2024 Abu Dhabi F1 Grand Prix show the approach surviving venue changes with minimal tuning, which is the kind of evidence industrial adopters look for.

Future Directions

• Monocular sensing: the authors state they will replace the stereo camera with a monocular one, bringing the stack closer to its human counterpart.
• Mixed human-robot racing: safely and effectively managing multi-vehicle racing with humans and robots sharing a track remains open, as the head-to-head crashes illustrate.
• Reliable state estimation at extreme speeds, and coordinating multiple autonomous racers.
• Safety and sim-to-real transfer: ensuring safe deployment alongside humans and improving how algorithms transfer from simulation to reality are named as still-open questions.

Target Audience

Authors’ abstract

Drone technology is proliferating in many industries, including agriculture, logistics, defense, infrastructure, and environmental monitoring. Vision-based autonomy is one of its key enablers, particularly for real-world applications. This is essential for operating in novel, unstructured environments where traditional navigation methods may be unavailable. Autonomous drone racing has become the de facto benchmark for such systems. State-of-the-art research has shown that autonomous systems can surpass human-level performance in racing arenas. However, the direct applicability to commercial and field operations is still limited, as current systems are often trained and evaluated in highly controlled environments. In our contribution, the system's capabilities are analyzed within a controlled environment -- where external tracking is available for ground-truth comparison -- but also demonstrated in a challenging, uninstrumented environment -- where ground-truth measurements were never available. We show that our approach can match the performance of professional human pilots in both scenarios.