IGUANA: Immersive Guidance, Navigation, and Control for Consumer UAV

AI summary

Overview

Research area: Human-Computer Interaction — specifically Mixed Reality (MR) interfaces for controlling consumer unmanned aerial vehicles (UAVs).

Technical level: Intermediate. The paper describes a full hardware-and-software prototype (Unity-based MR application, Android remote controller application, iOS companion application, custom networking protocol) plus a within-subjects user study with inferential statistics. Readers benefit from some familiarity with XR development concepts and basic experimental methods, but the design ideas are explained in accessible terms.

Scope (one sentence): The paper presents IGUANA, an MR-based guidance, navigation, and control system for consumer UAVs built around a 3D map interface, a virtual ball control metaphor, and a spatial overlay, and evaluates it against a conventional DJI dual-stick remote controller with ten participants.

What This Paper Is About

Flying a consumer drone is conceptually simple but mechanically awkward: users must map intended 3D motion onto the limited axes of a dual-stick controller while simultaneously operating a gimbal-mounted camera, and they must split attention between the aircraft in the sky and a live video feed on the controller. Waypoint missions reduce manual flying but still require tedious coordinate entry, and first-person-view goggles trade situational awareness for immersion.

IGUANA asks whether a mixed reality headset can instead provide immersive, spatially grounded control: planning missions on a scaled 3D model of the real terrain, steering with a physically motivated virtual ball, and keeping track of the drone through a persistent augmented-reality overlay even when it is out of sight. The goal is a proof-of-concept design, implementation, and usability evaluation for consumer UAVs.

Key Contributions

1. A 3D map interface for high-level control. A floating circular cutout showing a spatially accurate, scaled-down 3D reconstruction of the real world (built from Cesium-based Google Photorealistic 3D Tiles), with draggable waypoint markers, a virtual camera marker with live preview, height indicators shown near markers, and a UAV marker and user marker. Waypoints are placed and oriented with pinch and release gestures, with an optional thumbs-up gesture for more precise release.

2. A virtual ball interface for low-level control. A spatial control metaphor using a large red sphere for translational movement (forward/backward, left/right, up/down, 3 DoF, bounded inside a larger transparent sphere) and a small blue sphere for yaw and camera pitch (2 DoF, constrained to a vertical circular path). Both spheres can be controlled simultaneously with two hands, and the interface supports two interchangeable schemes: drone-centric and user-centric control.

3. A spatial overlay for situational awareness. An AR visualization anchored to the UAV's actual position, including a UAV frame, horizontal and vertical leading lines (from the user's feet to the UAV's ground position, and from the ground to the UAV's altitude), telemetry panels, and a rendering of the planned waypoint path in the real world.

4. A publicly released implementation and user study. The system is built from a Meta Quest 3 MR application, a DJI Mobile SDK v4.18 Android remote controller application, and a Swift iOS companion application, with telemetry updates at 10 Hz. The source code is publicly available to facilitate future research.

Main Findings

• 3D map interface was well received and reduced workload. Participants rated the 3D map interface significantly easier for Task A than the remote controller (SEQ: t(9) = 4.636, p_adj = .008). On the NASA-TLX for Task A, participants reported significantly lower Mental Demand (t(9) = −5.027, p_adj = .007), Temporal Demand (t(9) = −4.038, p_adj = .015), Effort (t(9) = −4.793, p_adj = .008), and Frustration (t(9) = −4.355, p_adj = .012), and significantly higher subjective Performance (t(9) = 4.836, p_adj = .008). Physical Demand showed no significant difference (Wilcoxon W = 14, p_adj = 1).

• Higher perceived usefulness and adoption intent than the remote controller. TAM ratings were significantly higher for IGUANA on Perceived Usefulness (t(9) = 5.609, p_adj = .002) and Behavioral Intention to Use (t(9) = 5.238, p_adj = .003). Perceived Ease of Use (t(9) = 1.832, p_adj = .3), Trust (t(9) = 1.222, p_adj = .506), and Satisfaction (Wilcoxon W = 4, p_adj = .85) showed no significant differences.

• Usability was comparable, not superior. System Usability Scale scores were numerically higher for IGUANA (M = 80.5, SD = 2.98) than for the remote controller (M = 68.25, SD = 5.89), but the paired t-test was not significant (t(9) = 2.293, p_adj = .192), which the authors interpret as usability comparable to the conventional controller.

• Photo-capture subtask favored IGUANA. IGUANA achieved an 80% success rate on taking a photo of a designated tree at the final waypoint, while only three of ten participants succeeded with the remote controller.

• Trajectory consistency and a time–accuracy trade-off. Flight-log visualizations showed all participants produced similar trajectories with the 3D map interface, in contrast to more varied paths with the remote controller. For IGUANA, a strong significant negative correlation between completion time and mean distance to waypoints (r = −0.693, p = .026) indicated that a longer, more deliberate strategy was linked to greater accuracy; no such relationship appeared for the remote controller (r = −0.278, p = .437).

• No significant difference in raw task metrics or performance score. For Task A, paired t-tests found no significant difference between the 3D map interface and the remote controller for completion time (t(9) = 0.195, p_adj = 1) or mean closest distance to waypoints (t(9) = 2.097, p_adj = .195). The composite performance score, computed from mean distance, completion time, and a binary photo indicator, also showed no significant difference (t(9) = −2.622, p_adj = .112), though the authors note the numerical difference may warrant a larger sample.

• Virtual ball interface was intuitive in concept but limited by missing physical feedback. Participants understood the intent of combining multiple axes and acknowledged its intuitiveness, but some found it harder to use than a standard dual-stick controller. There was no significant difficulty difference for Task B (SEQ: t(9) = −2.667, p_adj = .156), but subjective Performance on the NASA-TLX was significantly lower with the virtual ball (t(9) = −3.482, p_adj = .049). No significant differences were found for Mental Demand (t(9) = 0.506, p_adj = 1), Physical Demand (t(9) = 1.8, p_adj = .525), Effort (t(9) = 1.542, p_adj = .612), Frustration (t(9) = 1.562, p_adj = .612), or Temporal Demand (Wilcoxon W = 10, p_adj = 1).

• Spatial overlay was unanimously considered useful. All participants agreed the overlay helped them quickly locate the UAV in the sky. Reported concerns were about information density and placement: two participants (P1, P5) saw no need for anything beyond distance and height, three (P4, P5, P9) suggested moving additional information closer to the ground, and four (P4, P5, P7, P10) wanted the overlay to scale with distance rather than staying a constant size.

• Participants valued the declarative style. Qualitative feedback described the 3D map interface as "the best option," "intuitive," and "easy to use." One participant contrasted the "declarative approach in the 3D map interface" with the remote controller's "imperative style," and another appreciated that "there is no micromanaging." Several suggested a ground shadow for waypoint markers, movement scaling for finer control, and a separate input for height.

Methodology in Plain English

The researchers designed a three-part interaction concept and then built it as a working prototype spanning four devices: a DJI Mavic Air drone, a Meta Quest 3 headset, a Samsung Galaxy Note20 Ultra running a Java remote controller application built on DJI Mobile SDK v4.18, and an iPhone 16 Pro Max running a Swift companion application that supplies GPS and compass data and displays an AprilTag used for 6 DoF pose alignment.

The headset application (Unity 6 with Meta XR SDK v77) renders the 3D map from Google Photorealistic 3D Tiles via Cesium and positions the spatial overlay using a real-world-scale Cesium georeference, without loading terrain or imagery tiles. Waypoint positions are converted to GPS coordinates and altitude, with the caveat that altitude is relative to the takeoff point rather than WGS84, so the takeoff height is queried from the 3D tiles. High-frequency telemetry at 10 Hz travels from the phone to the headset over a WebRTC data channel, commands travel back over WebSocket for reliable ordering, and the drone's live camera feed also uses WebRTC because it has lower latency than the drone's built-in RTMP streaming.

Evaluation used a within-subjects design in which each participant completed the same tasks with IGUANA and with a baseline DJI Mavic Air remote controller running DJI GO 4 on a Samsung Galaxy S10, with counterbalancing so half started with each system. Ten participants (2 female, 8 male; age 20–42, M = 29.8, SD = 6.3) took part, compensated €20 for roughly 60 minutes. After an indoor tutorial using the DJI Assistant 2 flight simulator for simulated telemetry, sessions moved to an open field of approximately 100 m x 250 m with ground elevation from 215 m to 310 m.

Task A (Exploration) required flying through four waypoints marked on a printed map while avoiding terrain and photographing a designated tree at the final waypoint — done by placing waypoints in the 3D map with IGUANA, or manually with the dual-stick controller. Task B (Going Home) was a qualitative task requiring an upside-down capital 'L' trajectory using the virtual ball in user-centric mode. Flight logs from the DJI system supplied position, velocity, and control inputs; the composite performance score combined average closest distance to waypoints, normalized completion time, and a photo indicator, with a waypoint counted as passed within δ = 10 m. Subjective data came from the SEQ, NASA-TLX, SUS, and TAM questionnaires, plus a semi-structured interview analyzed descriptively. Normality was checked with Shapiro-Wilk tests and Q-Q plots (all p = .061–.978 except TAM Satisfaction p = .003, Task A Physical Demand p = .001, and Task B Temporal Demand p = .034, which used Wilcoxon signed-rank tests), with Holm-Bonferroni correction at α = .05 across three predefined families (m = 6, m = 10, and m = 7). Analyses were conducted in JASP.

Why This Matters

Impact on research. The paper offers a concrete, reproducible instantiation of three ideas that prior work treated mostly in isolation — 3D world-in-miniature waypoint planning on photorealistic tiles, a physical-analogy control metaphor, and a spatially anchored tracking overlay — and tests them together against a real consumer controller in a real outdoor field. It also contributes a device-spanning system architecture (headset plus Android remote-controller app plus iOS companion app, WebRTC and WebSocket transport, AprilTag-based alignment) that other MR-robotics projects can reuse, and it reports negative and null results honestly, including the lack of statistical significance in performance scores and the virtual ball's feedback problem.

Real-world applications (from the use cases the paper motivates):

• Aerial photography and surveying, where consistent waypoint paths and camera pose planning matter.
• Monitoring and inspection of hard-to-reach, large-scale, or hazardous areas such as over water, obstacles, or rough terrain.
• Operations where visual line of sight is interrupted, where the spatial overlay is intended to keep the drone locatable.
• Environments such as dense urban terrain, which the authors cite as the motivation for the 3D map interface, though the study itself was run in an open field for safety.

Industry relevance. The design targets consumer UAVs that support waypoint missions and telemetry through a mobile SDK, and it is built on commodity devices (Meta Quest 3, DJI Mavic Air, standard smartphones), suggesting the interaction model could be layered onto existing consumer drone ecosystems rather than requiring custom hardware.

Future Directions

• Solve the haptic feedback gap in the virtual ball interface. Participants frequently looked down to verify their gestures and reported that visual and auditory cues alone were insufficient for confidence; suggestions included grabbing the ball and adding an axis-locking option.
• Refine the spatial overlay's information presentation. Open questions include scaling the overlay with distance, moving information closer to the ground, and reducing the telemetry shown to just distance and height.
• Improve waypoint placement precision. Suggested refinements include a ground shadow for waypoint markers, movement scaling for finer control, and a separate input for altitude.
• Run a larger study. The authors state that the numerically higher but non-significant performance scores suggest a larger sample may be needed to investigate the effect further, and the truncated discussion leaves the remaining implications of the study unreported in the available text.

Target Audience

This paper is most useful for HCI and XR researchers working on spatial user interfaces, world-in-miniature interaction, and drone or robot teleoperation; developers building mixed reality applications with Meta XR SDK, Unity, and Cesium who want a working reference architecture; and drone industry practitioners evaluating whether headset-based planning and control could complement or replace conventional dual-stick controllers and tablet ground stations. Readers looking for statistically decisive evidence of performance superiority will not find it here — the strongest results are about perceived usefulness, reduced workload, trajectory consistency, and the photo-capture subtask.

Authors’ abstract

As the markets for unmanned aerial vehicles (UAVs) and mixed reality (MR) headsets continue to grow, recent research has increasingly explored their integration, which enables more intuitive, immersive, and situationally aware control systems. We present IGUANA, an MR-based immersive guidance, navigation, and control system for consumer UAVs. IGUANA introduces three key elements beyond conventional control interfaces: (1) a 3D terrain map interface with draggable waypoint markers and live camera preview for high-level control, (2) a novel spatial control metaphor that uses a virtual ball as a physical analogy for low-level control, and (3) a spatial overlay that helps track the UAV when it is not visible with the naked eye or visual line of sight is interrupted. We conducted a user study to evaluate our design, both quantitatively and qualitatively, and found that (1) the 3D map interface is intuitive and easy to use, relieving users from manual control and suggesting improved accuracy and consistency with lower perceived workload relative to conventional dual-stick controller, (2) the virtual ball interface is intuitive but limited by the lack of physical feedback, and (3) the spatial overlay is very useful in enhancing the users' situational awareness.