Training One Model to Master Cross-Level Agentic Actions via Reinforcement Learning

AI summary

Overview

Research area: Agentic AI and multimodal foundation models, specifically reinforcement learning for embodied and computer-use agents that must operate across heterogeneous action spaces.

Technical level: Advanced. The paper assumes familiarity with reinforcement learning (policy gradients, GRPO, KL-regularized objectives), vision-language models, Markov Decision Processes, and supervised fine-tuning pipelines.

Scope: The paper proposes CrossHA, a single vision-language agentic model trained through a three-stage supervised-plus-reinforcement-learning pipeline to autonomously choose between different action granularities (for example, high-level motion primitives versus low-level mouse-and-keyboard commands) at each step of a task, evaluated on over 800 Minecraft tasks.

What This Paper Is About

Most AI agents are locked into one fixed action space: a GUI agent clicks and types, an API agent calls functions, a robotics policy emits motor commands, and each is designed by hand for its task. The problem is that the best granularity of action is not fixed — it can change from task to task, or even step to step within the same task, so a rigid action space caps an agent's success rate and flexibility. CrossHA is a unified agentic model trained to master several action spaces at once and to select the most appropriate one for each step, without human-specified switching rules.

Key Contributions

1. A unified agentic model with heterogeneous action spaces. The authors propose CrossHA, which operates across multiple action subspaces and autonomously selects the context-appropriate interface rather than relying on human-defined heuristics.

2. A three-stage training pipeline using Multi-Turn GRPO. The pipeline combines cold-start supervised fine-tuning, Single-Turn Reinforcement Learning (STRL), and Multi-Turn Reinforcement Learning (MTRL), enabling adaptive action switching within a single trajectory while balancing task success against execution efficiency.

3. A relabeling/self-training procedure that bootstraps action-space preferences. The model M_strl (produced by STRL) is used to relabel the dataset into D_strl, which initializes the MTRL stage through a model called M_cs2.

4. State-of-the-art results and generalization evidence on over 800 Minecraft tasks, with training on only 30 tasks, demonstrating that adaptive action-space selection generalizes better than static action-space baselines.

Main Findings

• CrossHA leads on average success rate across all tasks. In Table 1, CrossHA reaches an All-Tasks Average Success Rate (ASR) of 54.6 ± 47.6 and Finished Tasks (FT) of 58.7. For comparison, OpenHA records 62.8 FT and 31.5 ± 12.5 ASR, JARVIS-VLA records 63.8 FT and 24.5 ± 28.4 ASR, Game-TARS records an All-Tasks ASR of 42.2, and UI-TARS-1.5 records an All-Tasks ASR of 33.8.

• Strong category-level results. CrossHA reports Mine Blocks FT 45.2 / ASR 40.0 ± 48.3, Kill Entities FT 58.1 / ASR 45.1 ± 43.5, and Craft Items FT 72.7 / ASR 78.8 ± 41.0. The paper highlights peak success rates of 94.7% in Mine Blocks and 83.3% in Craft Items on representative tasks.

• Single-action-space agents specialize narrowly. GroundingHA shows a Kill Entities ASR of 90.1 ± 23.4 in Table 1, MotionHA performs comparatively better on Mine Block tasks, and RawHA is described as advantaged on Craft Item tasks because of fine-grained control — but none of these fixed-space baselines is uniformly strong.

• The STRL stage materially improves outcomes. Removing STRL reduces the All-Tasks ASR in Table 1 from 54.6 ± 47.6 to 41.6 ± 47.9, and the Craft Items ASR from 78.8 ± 41.0 to 58.0 ± 48.4. In the OOD portion of Table 2, CrossHA (w/o STRL) reaches 58.0 ± 48.4 on Craft Items and 39.7 ± 48.1 overall, versus 78.8 ± 41.0 and 49.1 ± 46.6 for the full model.

• RL fine-tuning on 30 tasks transfers to over 800 evaluation tasks. The paper reports consistent ASR and FT gains from RL over SFT-only agents, particularly in fine-grained control domains such as Craft Item, and states that improvements generalize from the 30 training tasks to the full evaluation suite.

• Dynamic selection narrows the in-distribution versus out-of-distribution gap. RawHA-RL achieves 96.2 ± 3.8 on in-distribution Craft Items but drops to 69.8 ± 44.4 out-of-distribution. CrossHA records 83.7 ± 25.5 in-distribution and 78.8 ± 41.0 out-of-distribution on Craft Items, and 68.8 ± 30.5 versus 49.1 ± 46.6 on All Tasks.

• Mixed action spaces converge faster and to higher asymptotic performance. Figure 3 is reported to show CrossHA converging faster and reaching higher asymptotic performance during multi-turn RL than single-space baselines (GroundingHA and MotionHA), which initialize MTRL from only one action-space subset of D_strl.

• The STRL warm start accelerates MTRL. Figure 4 is reported to show that including STRL improves training efficiency and convergence speed in the subsequent MTRL stage, despite its low computational cost; all compared models underwent the same 200-step SFT cold start before MTRL.

Methodology in Plain English

The authors start from a vision-language base model, Qwen2-VL-7B-Instruct, and fine-tune it on Minecraft-specific VQA and captioning datasets to produce a base model. Grounding annotations come from a SAM-based grounding pipeline, and motion annotations from a motion-generation module built on a fine-tuned MineCLIP model; these annotate VPT data and a subset of contractor-collected trajectories. Because not every trajectory can be annotated, the resulting action-space distribution is imbalanced, and the combined annotated data form the mixed-action-space dataset D_mix.

Training then proceeds in three stages. First, Mixed-Space SFT: the model is fine-tuned on D_mix, a balanced dataset drawn from multiple action subspaces, so that it can generate valid actions in each format — this produces M_mix, which understands the syntax of different spaces but does not yet choose between them. Second, Single-Turn RL (STRL): a warm-up stage builds prompts that ask the model to produce candidate actions across all available action spaces, and candidates are kept only if they actually execute successfully (rejection sampling), yielding a rebalanced dataset and a model M_cs1. GRPO is then applied, treating each example as a one-step decision. The reward is deliberately action-space agnostic: r(â, a*) = 1{g(â) = g(a*)}, where g parses an action string into a canonical raw representation, so the model is credited whenever the underlying action matches — regardless of which surface format produced it. This yields M_strl. Third, Multi-Turn RL (MTRL): the preferences of M_strl are distilled back into supervised form by relabeling each example — if the model's predicted action parses to the same raw action as the ground truth, the original label is replaced with the model's chosen format; otherwise the original label is kept. Fine-tuning on this relabeled dataset D_strl gives M_cs2, which is then optimized with multi-turn GRPO against a binary episodic success reward r(τ) = 1{success(τ)}, with an additional penalty −λ l_θ(τ) on the total number of tokens the model generates over the trajectory. This penalty pushes the model toward concise action spaces (for example, high-level APIs) over verbose ones (raw primitive commands) when both succeed.

The general problem is framed as an MDP with a composite action space A = ∪ A_x, where each subspace A_x has its own controller C_x, and the overall objective balances immediate reward against the cost of each action granularity via J = E[Σ_t (r_t − λ_x cost(a_t))].

For the MTRL stage, 10 tasks are selected from each of three OpenHA categories — craft_item, kill_entity, and mine_block — for 30 training tasks total. GRPO is run for over 80 iterations, each involving over 6,400 environment interactions.

The environment is Minecraft version 1.16.5 under the VPT protocol, with observations consisting only of raw RGB screenshots at 640 × 360 and 20 Hz, and interaction through human-aligned mouse and keyboard controls (mouse displacements discretized into 1800 bins). Evaluation uses the OpenHA benchmark suite of over 800 manually designed and verified tasks, split into Mine Blocks, Craft Items, and Kill Entities, with two metrics: Finished Tasks (FT), the proportion of distinct tasks in a category the agent completes at least once, and Average Success Rate (ASR), the mean success rate across all tasks in the category.

Why This Matters

Impact on research. The paper reframes action-space selection as a learnable policy decision rather than an engineering choice, and shows that reinforcement learning over a composite action space yields better out-of-distribution generalization than fine-tuning on any single space. It also contributes a reusable three-stage recipe (SFT, single-turn RL, multi-turn RL) for training cross-level agentic behavior, and all code and models are released at the project's GitHub repository.

Real-world applications:

• Computer-use and web agents that fall back from API calls to precise mouse-and-keyboard GUI manipulation when an API is blocked, for example by CAPTCHA verification.
• Deep research agents that gather most information efficiently through search APIs but switch to GUI-level control for specific protected or authentication-gated pages.
• Tool-calling agents built on services like Model Context Protocol (MCP) that need to decide when a structured tool call is preferable to a manual, lower-level interaction.
• Embodied robotics and vision-language-action systems, where a policy might use a high-level motion primitive for speed and drop to fine motor control for precision — the paper notes real-world robotics as an intended future extension, flagging safety and latency concerns.

Industry relevance. Any product that wraps an LLM or VLM into an agent faces the same trade-off between token cost, latency, and reliability across interaction granularities. A model that learns to pick cheaper high-level actions when they suffice, and only pays for verbose low-level control when needed, directly addresses inference cost and success-rate concerns for deployed agents.

Future Directions

• Improving the efficiency of multi-turn RL, which the authors explicitly name as future work; multi-turn training in this paper required over 80 GRPO iterations with more than 6,400 environment interactions each.
• Extending the framework to real-world robotics settings, where the authors note additional challenges such as safety and latency arise.
• Broadening the set of action subspaces beyond those studied here, since the formulation is general over any collection of action spaces A = ∪ A_x with associated controllers.
• Understanding how cost weighting affects behavior, since the objective uses per-space cost coefficients λ_x and a token-length penalty λ l_θ(τ) to trade off efficiency against success; the paper does not report a systematic study of how these weights change selection behavior.

Target Audience

This paper is most useful to researchers and engineers working on agentic AI, reinforcement learning for multimodal models, and embodied or computer-use agents. It is particularly relevant to readers interested in post-training pipelines that combine supervised fine-tuning with GRPO, in Minecraft-based multi-task benchmarks, and in the question of how agents should decide between high-level and low-level actions. Readers without a background in policy optimization will find the method sections demanding, but the framing of action-space selection as a learnable decision is accessible.

Authors’ abstract

The paradigm of agentic AI is shifting from engineered complex workflows to post-training native models. However, existing agents are typically confined to static, predefined action spaces-such as exclusively using APIs, GUI events, or robotic commands. This rigidity limits their adaptability in dynamic environments where the optimal granularity of interaction varies contextually. To bridge this gap, we propose CrossHA, a unified agentic model that masters heterogeneous action spaces and autonomously selects the most effective interface for each step of a trajectory. We introduce a comprehensive training pipeline that integrates cold-start supervised fine-tuning with a Multi-Turn Group Relative Policy Optimization (GRPO) algorithm. This approach enables the agent to learn adaptive action switching-balancing high-level efficiency with low-level precision-without human-specified rules. Extensive experiments on over 800 tasks in the open-world Minecraft environment demonstrate that CrossHA achieves state-of-the-art performance. By dynamically leveraging the strengths of diverse action spaces, our model significantly outperforms fixed-action baselines, exhibiting superior generalization and efficiency in long-horizon reasoning. All code and models are available at https://github.com/CraftJarvis/OpenHA.