GenXR-Bench: Benchmarking Coding Agents for Interactive XR Development

AI summary

Overview

Research area: AI for software engineering — specifically benchmarking autonomous coding agents on extended reality (XR) application development, combining spatial computing, WebXR runtime engineering, and automated test generation.

Technical level: Intermediate. Readers should be comfortable with coding-agent benchmarks (Pass@1, held-out tests) and have some familiarity with 3D/XR concepts such as hand tracking, scene graphs, and coordinate frames.

Scope (one sentence): The paper introduces GenXR-Bench, a 112-task expert-authored benchmark plus an automated testing framework (GenXR-Test) for measuring whether coding agents can implement XR SDK features, repair XR bugs, and generate complete interactive XR applications.

What This Paper Is About

Coding agents have progressed from writing single files to repairing whole repositories and building interactive web and desktop apps, but nobody had systematically tested them on XR software, where correctness depends on continuous head and hand motion, spatial coordinate math, and environmental sensing rather than a discrete event loop. The problem is also a measurement problem: existing benchmarks rely on unit tests or browser action runners, which cannot verify spatial scene graphs or 3D tracking input, while 3D generation benchmarks mostly test static shapes rather than reactive software. The authors build a benchmark and an automated simulator-based verification suite so that agent-written XR code can be executed headlessly and judged against expert human ground truth.

Key Contributions

1. GenXR-Bench, an open-source benchmark of 112 expert-authored tasks: 24 SDK feature implementations, 21 SDK bug repairs, and 67 end-to-end application generation tasks, all built on the open-source WebXR framework XR Blocks version 0.21.0.
2. GenXR-Test, an XR testing framework that pairs deterministic scripted kinematic actors with adaptive VLM-controlled agents in simulated 3D environments, checking runtime behavior through both scene-graph assertions and narrowly scoped visual assertions.
3. Hardware-grounded validation: eight blinded XR experts judged 124 candidate applications across 372 runs on consumer XR hardware (Galaxy XR, Meta Quest 3), showing the automated suite reaches 96.5% mean accuracy with 4.0% flakiness.
4. An empirical study of 10 model configurations, identifying where frontier agents break down: coordinate-frame handling, asynchronous sensor lifecycles, and embodied interaction semantics.

Main Findings

• The best model still fails half the time. The strongest configuration, claude-opus-5, reaches only 50.9% overall Pass@1 across all 112 tasks. The weakest evaluated configuration, gemini-3.1-pro-preview, reaches 23.2%.
• Bug repair is far easier than feature implementation. claude-opus-5 scores 85.7% on repair but only 29.2% on SDK feature tasks; deepseek-v4.1-flash and gpt-5.6-sol both hit 95.2% on repair while scoring 20.8% and 29.2% on features respectively.
• Application generation sits in the middle. claude-opus-5 has the highest application-generation Pass@1 at 47.8%, while gpt-5.6-sol earns the highest mean Design Score at 4.03 on the 1–5 Likert scale (claude-opus-5 scores 3.86).
• Coordinate-frame errors dominate failures. 64.4% of failed runs compare positions expressed relative to different parent objects without transforming them into a common frame, such as anchoring buttons to a hand's root joint instead of placing them on the palm surface.
• Sensor initialization races are pervasive. 44.9% of failed runs start gameplay or sensing-dependent calculations before assets or tracking state exist, or check for detected surfaces only once at startup.
• Bimanual and embodied semantics are misunderstood. 34.5% of failures involve treating hands interchangeably, missing asymmetric gesture semantics, treating physical touch as a discrete click via Boolean bounding-box intersection, or ignoring ergonomics by placing UI out of comfortable reach or behind the user.
• Model-written tests do not predict success. The test-to-edit ratio was not indicative of overall success; models tended to overwrite unit tests that did not validate real XR behavior, while models that actually exercised runtime tests were more successful.
• The verifier is reliable but not perfect. Overall precision is 0.989, recall 0.886, Fleiss' κ 0.929, and flakiness 4.0%; disagreements clustered where candidates departed from expected spatial conventions, such as a ball spawning at the wrist rather than the fingertips.
• Smaller models fail on fundamentals. Weaker models exhibited web-development coding hallucinations, poor metric understanding, and API hallucinations; frontier models avoided these but still failed higher-order spatial reasoning tasks like bimanual coordination, room geometry visualization, and physical alignment.

Methodology in Plain English

The team first gathered real development work rather than inventing toy problems. Five XR experts interviewed core XR Blocks contributors and inspected all 552 pull requests and issues available as of August 24, 2026, filtering candidates by relevance to XR engineering, feasibility, and scope. Feature tasks were made by deleting a real implementation while keeping its public interface and surrounding context; repair tasks were made by reproducing a real defect in a pinned snapshot.

For application tasks, the authors derived a taxonomy from 436 peer-reviewed XR systems papers, used gemini-3.5-flash to generate 1,000 candidate app ideas, had experts select 67, and then expanded them into full specifications with gemini-3.6-flash under manual expert review and editing. Specification-and-test refinement ran up to 10 iterations against expert-reviewed valid and adversarial solutions, with no canonical reference solution required since many implementations can be valid.

To make execution reproducible and headless, everything runs on XR Blocks, a web-based WebXR framework, rather than a GUI-heavy desktop game engine. Applications run in Chromium through Playwright with Vitest-based test sessions. The simulator injects fully articulated hand poses computed by forward kinematics with joint limits, injects room geometry into plane, mesh, depth, and object-detection pathways (two scenes are used: Living Room and Office), and exposes explicit wait and frame-stepping so tests can pause, run a VLM check, and advance time deterministically.

Tests combine two kinds of actors and two kinds of assertions. Scripted actors replay fixed action sequences for reproducibility; adaptive VLM agents follow narrow natural-language instructions when layout differences make fixed sequences unreliable. Programmatic assertions check transforms, spatial relationships, collisions, tracking state, sensor outputs, and events; VLM assertions inspect rendered first-person images against a tightly scoped human-written visual criterion rather than judging the whole experience. Because implementations may name objects differently, agents must apply semantic tags to specified objects so tests can find functionally equivalent targets without forcing one implementation.

Finally, all 10 model configurations were run through the mini-swe-agent harness in the Harbor Framework, following DeepSWE procedures, with network access restricted to XR Blocks source, documentation, examples, and dependency docs. Adaptive actors and VLM assertions used gemini-3.8-flash, and blind human judgments from eight XR experts served as ground truth.

Why This Matters

Impact on research. The paper moves agent evaluation from flat-screen software into physically situated computing, where the pass/fail signal depends on continuous motion, sensing, and 3D geometry. It provides a reproducible testbed and a failure taxonomy (coordinate frames, sensor lifecycles, embodied semantics) that other groups can build on, and it shows that simulator-based verification can substitute for device trials at 96.5% agreement.

Real-world applications — the benchmark's application tasks were drawn from a taxonomy spanning:

• Education, such as immersive instructional experiences.
• Healthcare, such as guided or therapeutic XR interactions.
• Daily living, such as spatial assistance in the home.
• Design and entertainment, such as spatial authoring tools and calming games like the paper's bumble-calm example.

Industry relevance. XR platform vendors, engine and framework teams, and agent-tooling companies all need to know whether an agent can be trusted to write spatial code. The finding that repair is easy but feature implementation is hard, and that self-written tests give false confidence, directly informs how such agents should be deployed, gated, and instrumented in real development pipelines.

Future Directions

• Fixing asynchronous sensor lifecycles. Agents need to reason about initialization dependencies and continuous sensing updates, not one-shot startup checks for detected surfaces. The 44.9% failure rate marks this as a concrete engineering target rather than an abstract capability gap.
• Teaching consistent spatial models. With 64.4% of failures involving coordinate frames, a natural next step is training or scaffolding agents to maintain an explicit model of the physical world, the user, and virtual content, and to transform between parent frames before comparing positions.
• Embodied interaction and ergonomics. Beyond detecting gestures, agents must handle asymmetric hand roles, continuous surface penetration rather than Boolean clicks, and comfortable reach and scale. Extending evaluation to these usability properties is an open problem.
• Broadening coverage and design evaluation. The verifier validation covered 62 of the app tasks — those not requiring a specific physical object or setting — leaving object-dependent tasks and the alignment of the VLM design judge with human raters as open areas, along with whether better runtime-testing behavior by agents can be actively elicited rather than merely observed.

Target Audience

Researchers and engineers working on coding agents and agent benchmarks; XR platform, framework, and tooling developers who want to know what AI-generated spatial code can and cannot do; HCI and spatial-computing researchers interested in automated evaluation of interactive 3D software; and QA or test-infrastructure teams building simulator-based verification for embodied applications.

Authors’ abstract

Coding agents are rapidly advancing from single-file generation to long-horizon repository-scale engineering, yet the evaluation of their capabilities to author interactive extended reality (XR) applications remains an open challenge. Unlike traditional software, XR applications operate within dynamic 3D physical environments where runtime robustness depends jointly on continuous user motion, spatial input, and environmental sensing. We introduce GenXR-Bench, the first benchmark for the end-to-end evaluation of coding agents in XR software development. GenXR-Bench contains 112 expert-authored tasks spanning 24 feature implementations, 21 bug repairs, and 67 application generation tasks. Held-out tests combine scripted and adaptive interactions with programmatic and visual assertions to evaluate core functionality and runtime behavior. GenXR-Bench's testing suite achieves 96.5% accuracy against blinded expert judgments of 124 candidate applications executed on consumer XR hardware. Evaluating 10 models on GenXR-Bench, we find that the strongest configuration achieves only 50.9% overall Pass@1. Failure analysis identifies recurring misunderstandings in asynchronous sensor lifecycles, physical grounding, and embodied interaction semantics. GenXR-Bench establishes a reproducible testbed for evaluating coding agents in the interactive and physically situated conditions of spatial computing.