HiSTEVE 1.0: Harnessing Human-Interface VLA Agents for Open-World Games

AI summary

Overview

Research area: Embodied AI / vision-language-action (VLA) models applied to open-world video games, specifically Minecraft, with a focus on agents that play through the same keyboard-and-mouse interface as a human.

Technical level: Advanced. The paper assumes familiarity with VLA architectures, hierarchical planning with LLMs/VLMs, imitation learning, and reinforcement learning (GRPO, DPO-style pairwise losses).

Scope: The paper describes HiSTEVE, a two-level system pairing a VLM planner (the "harness") with a single Qwen2.5-VL-7B-based VLA executor that plays Minecraft via native keyboard-and-mouse input, and evaluates it on three benchmarks: MineStudio Bench, OpenHA, and LongBench.

What This Paper Is About

Existing Minecraft agents tend to be good at one of two things: raw visuomotor control (mapping pixels to keystrokes) without long-horizon autonomy, or high-level planning that relies on environment APIs or predefined skills unavailable in most games. HiSTEVE's goal is to combine both in one closed loop — an agent that keeps track of a long-term objective, picks the next subtask itself, and executes it through the same keyboard-and-mouse interface a human player uses. The authors also address two specific engineering obstacles: deciding when a subtask is finished, and preventing interference between non-GUI control (mouse moves the camera) and GUI control (mouse moves a cursor).

Key Contributions

1. A capability-aware agent harness. A structured-memory orchestration layer with a VLM planner that selects and instantiates atomic subtasks only from schemas within the VLA's execution capabilities, plus evidence-driven handoff, verification, and recovery.
2. A policy-coupled task completion state detector. A lightweight detector that shares the VLA's image encoder and additionally conditions on VLA policy context (hidden states before the first action token) to signal candidate subtask completion — deliberately distinct from failure detection.
3. Regime-specific representations for reliable native-interface control. Regime-Specific RMSNorm (RS-Norm) branches for non-GUI versus GUI interactions (adding only 2Ld parameters over the backbone) and "Dual-Mouse," disjoint mouse-displacement token sets for the two regimes, motivated by a gradient-alignment analysis showing cross-regime gradients are less aligned than within-regime ones.
4. A three-stage VLA training recipe. World knowledge injection, imitation learning, and environment-feedback reinforcement learning (clipped GRPO over same-start branch groups, plus DPO-style pairwise losses on state-matched segments, with world-action replay and KL regularization to a reference policy).
5. State-of-the-art atomic results plus long-horizon capability, including crafting an iron pickaxe from scratch using only human-interface actions.

Main Findings

• MineStudio Bench average of 85.0%. Strict pass@1 scores were 93.3% (Mine), 86.7% (Kill), 66.7% (Craft), and 93.3% (Smelt). Against OpenHA Text/CoA, the four-family average rose from 65.0% to 85.0%, with per-task gains of 20.0, 13.4, 26.7, and 20.0 percentage points; the largest absolute gain was on Craft, which nonetheless remained the weakest category.
• OpenHA average of 37.4%, versus 36.8% for OpenHA Text/CoA. The gain was concentrated in Embodied (37.0% → 46.0%), while GUI (27.2%) and Combat (39.0%) were slightly below the corresponding baseline scores. The authors attribute GUI failures to breakdowns during multi-step item placement, intermediate-state correction, or result retrieval.
• LongBench final-goal success of 42.9% (Resource & Crafting), 28.6% (Exploration & Memory), 14.3% (Recovery), and 14.3% (Open-ended) — improvements of 28.6, 21.5, 11.4, and 12.9 points over OpenHA Text/CoA. The paper notes family-level gains of up to 28.6 points and states the largest long-horizon improvements occur where multi-stage resource production or reuse of exploration information is required.
• The completion detector beats prompted general-purpose VLMs. On a class-balanced validation split of 4,674 VPT-derived samples, the 1.4B-parameter detector reached 0.863 accuracy and 0.863 F1, with precision 0.861, recall 0.865, FPR 0.140, and FNR 0.135. For comparison: random 0.500; Qwen3.5 9B accuracy 0.500 / F1 0.057; Qwen3.5 35B 0.502 / 0.078; GPT-4o 0.698 / 0.690; GPT-5.5 0.780 / 0.776. Qwen3.5 models collapsed toward the incomplete class (recalls of 0.030 and 0.042).
• Staged training helps, but training alone does not solve long horizons. Imitation-only gave MineStudio All 50.4%, OpenHA Avg. 29.5%, LongBench Goal SR 7.1%. Adding world knowledge injection: 65.1%, 35.3%, 14.3%. Adding environment-feedback RL: 80.7%, 37.7%, and LongBench Goal SR unchanged at 14.3% (progress 21.9%).
• RS-Norm and Dual-Mouse are complementary. From a shared-representation baseline of 41.1% on MineStudio All, RS-Norm alone gave 57.3%, Dual-Mouse alone 61.9%, and the combination 80.7%; OpenHA Avg. moved 26.6% → 37.7% and LongBench Goal SR 3.6% → 14.3%.
• The harness is what drives long-horizon performance. With the VLA fixed, fixed-horizon handoff scored 81.2% on MineStudio All, 37.8% on OpenHA Avg., and 10.7% Goal SR on LongBench (17.9% progress); the full harness gave 85.0%, 37.4%, 25.0%, and 28.6%. The authors state the full harness "more than doubles" goal success relative to fixed-horizon handoff, and that stronger atomic execution alone does not yield comparable long-horizon gains.

Methodology in Plain English

The system splits the problem across two timescales.

Slow loop (the harness). A VLM planner (GPT-4o in the experiments) looks at the overall goal, recent observations, a compact structured memory, and a restricted menu of subtask "schemas" that the executor is known to handle. It picks one subtask and turns it into a fixed natural-language instruction. The schemas carry their own applicability conditions, visually observable completion criteria, recognizable failure conditions, and execution budget — a capability boundary rather than a script. Memory is split into four stores: working memory (current state and goal), episodic memory (state before/after each invocation with outcome and cost), procedural memory (reusable dependencies learned from verified successes), and world memory (landmarks and spatial relations).

Fast loop (the VLA). One Qwen2.5-VL-7B policy takes the instruction plus the current first-person frame and emits tokenized keyboard-and-mouse actions at 20 Hz, matching a VPT-style action space where mouse displacement is μ-law transformed and quantized into bins.

Knowing when to stop. A completion detector watches historical frames sampled at 800, 600, 400, and 200 ms before the query, injects the VLA's own policy context via cross-attention after every fourth decoder layer, and fires a candidate completion only when the probability exceeds a threshold for two consecutive queries. Crucially, a handoff is not treated as success: the harness re-examines the visuals and labels the outcome success, failure, or interrupted, then advances, attempts local recovery, or replans.

Avoiding cross-regime interference. Because mouse movement means camera rotation in non-GUI play but cursor motion in GUI menus, the model gets separate RMSNorm scale vectors per regime and separate mouse token sets (an interaction-type token marks which regime is active), while everything else stays shared.

Training. The policy is first given Minecraft world knowledge, then imitation-learned from trajectories, then refined with reinforcement learning using environment feedback — comparing multiple rollouts from the same starting state, scoring them by certified completion, verified progress, cost, and failure penalties, and adding pairwise preferences on matched segments where a correct prefix occurred inside a failed rollout.

Why This Matters

The paper argues that reliable long-horizon autonomy is an orchestration problem, not merely an executor-quality problem — an ablation finding with practical consequences for anyone building agents from pretrained policies. It also offers a cheap, general trick (regime-specific normalization plus separated action tokens) for handling heterogeneous interaction modes in a single model.

Potential real-world applications:

• Game testing and QA automation — agents that play builds through the actual human interface rather than debug APIs.
• Accessibility tooling — agents that operate software via keyboard-and-mouse for users who cannot.
• GUI-plus-3D interaction agents — the non-GUI/GUI interference problem recurs in robotics and desktop automation, where a policy must both move through space and manipulate menus.
• Training data generation and curriculum design for embodied agents in simulated open worlds.

Industry relevance: The regime-separation idea maps directly onto robotics and computer-use agents that mix continuous spatial control with discrete menu interaction. The finding that RL post-training improves local reliability but leaves long-horizon goal success unchanged is a caution for teams expecting post-training alone to deliver autonomy. The detector result (0.863 F1 for a 1.4B model versus 0.776 for GPT-5.5) also suggests specialized, policy-coupled small models can beat much larger prompted generalists on narrow state-estimation tasks.

Future Directions

• Robust multi-step GUI execution. The authors identify multi-step item placement, intermediate-state correction, and result retrieval as the key bottleneck, with GUI success at 27.2% on OpenHA — slightly below baseline — and Craft the weakest MineStudio category at 66.7%.
• Better recovery and weakly structured objectives. Recovery and Open-ended LongBench families both sit at 14.3% final-goal success, which the authors flag as substantially harder and an open challenge.
• Extending RS-Norm beyond GUI versus non-GUI. The paper explicitly proposes extending it to other heterogeneous pairings, such as vision-language understanding versus visuomotor control, and multimodal understanding versus generation.
• Scaling harness-level orchestration. Since the harness changed atomic scores only modestly (81.2% → 85.0% on MineStudio All; 37.8% → 37.4% on OpenHA Avg.) while roughly doubling LongBench goal success (10.7% → 25.0%), the question of how far state-aware orchestration can be pushed with a fixed executor remains open.

Target Audience

Researchers and engineers working on embodied agents, VLA models, and LLM/VLM-based hierarchical planning — particularly those interested in agents that must operate through human interfaces rather than privileged APIs. It is also useful for practitioners building computer-use or game-playing agents who care about subtask handoff timing, state verification, and mixing GUI with spatial control in a single policy. Readers without background in imitation learning and policy-gradient methods will find the training sections dense, but the harness design and ablation results are accessible and transferable.

Authors’ abstract

Building virtual embodied agents that operate in open-ended games like human players is a long-standing goal requiring both long-horizon autonomy and reliable native-interface control. As a procedurally generated open world, Minecraft provides a natural testbed for studying the integration of these capabilities. Existing agents typically provide either native visuomotor control without long-horizon autonomy or high-level planning executed through environment APIs or predefined skills. To bridge this gap, we propose HiSTEVE, a hierarchical VLA agent with a capability-aware agent harness. At the orchestration level, we equip the harness with structured memory for tracking progress and a VLM planner for selecting subtasks within the VLA's capabilities. To reliably determine subtask completion, we further introduce a policy-coupled task completion state detector that conditions on VLA policy context to signal candidate completion. At the control level, a single VLA executes all subtasks with vision-only perception and native keyboard-and-mouse actions. To mitigate interference between heterogeneous GUI and non-GUI interactions, we introduce regime-specific RMSNorm branches and separate mouse-movement token embeddings. We also train the VLA through world knowledge injection, imitation learning, and reinforcement learning with environment feedback to improve reliability and recovery. HiSTEVE achieves state-of-the-art atomic-task performance, scoring 85.0% on MineStudio Bench and a 37.4% average success rate on OpenHA, while also completing long-horizon tasks such as crafting an iron pickaxe from scratch like human players.