Skip to content
AI.info

Research

Heave-Ho: Decentralized Long-Horizon Multi-Humanoid Collaboration

Overview Research area: Robotics (cs.RO) — humanoid whole-body loco-manipulation, multi-agent reinforcement learning, and vision-language task planning for robot teams. Technical level: Advanced.

arXiv
2611.22275
Published
2026-11-17
Authors
Ting-Hsuan Chen, Siheng Zhao, Han Jiang, Qi Pan, Louison Lu, Jiaheng Luo, Henghui Bao, Yiyu Chen, Quan Nguyen, Yue Wang

AI summary

Overview

  • Research area: Robotics (cs.RO) — humanoid whole-body loco-manipulation, multi-agent reinforcement learning, and vision-language task planning for robot teams.
  • Technical level: Advanced. The paper assumes familiarity with reinforcement learning (PPO/MAPPO, centralized training with decentralized execution), motion tracking controllers, and vision-language models.
  • Scope: The paper presents Heave-Ho, a two-level decentralized framework in which teams of up to ten humanoids carry, push, pull, and hand over shared objects using only local observations and spoken commands, plus HH-Bench, a 12-task benchmark for such teams.

What This Paper Is About

Many real tasks — moving a table through a doorway, carrying a sofa, passing a heavy box down from a truck — exceed what a single humanoid can physically do, so robots must share the load. Doing this well is hard because the robots are physically coupled through the object, the tasks unfold over long sequences of coordinated actions, and every lift or step must start at the same moment. The paper's goal is a team that stays synchronized and coherent while each robot acts only on its own on-board sensing, with no network communication between robots at any point during a task.

Key Contributions

  1. Multi-agent post-training of a frozen foundation tracker. A residual policy, shared across all robots, is learned on top of a frozen humanoid foundation tracking model (SONIC), turning it into a controller for physically coupled collaboration that runs from each robot's local observations alone.
  2. A trained vision-language planner plus speech-based coordination. A fine-tuned Qwen3-VL-2B selects the next team skill from the leader's head-camera image, and a leader–follower speech mechanism synchronizes when the team starts that skill without exchanging network packets.
  3. HH-Bench. A benchmark of 12 collaborative tasks for teams of 2–10 humanoids, including heterogeneous teams, covering constrained transport, team scaling, diverse interaction modes (shoulder carry, pull, push, handover), and tasks that can be chained into longer ones.
  4. Systematic evaluation of scaling and heterogeneity. A policy trained with exactly two identical humanoids is deployed without retraining to teams of four, eight, and ten robots, to a different humanoid platform, and to human partners.

Main Findings

  • Low-level controller dominance in simulation: Heave-Ho succeeds in 94.41% of episodes averaged over the 12 HH-Bench tasks (three seeds of 512 teams each), while no other controller exceeds 28.56%. Comparisons: Pure SONIC 0.22%, Multi-HDMI 28.56%, Heave-Ho without SONIC 16.78%, Heave-Ho with independent actors 22.05%. Heave-Ho also has the lowest mean errors (body 18.62 cm, object 19.59 cm).
  • Tracking alone is not enough: Pure SONIC reproduces body references more closely than several baselines yet almost never completes a task — it does not know how to exert force on the object or share the load with a partner.
  • Planner beats strong baselines: Heave-Ho's planner completes 43 of 50 long-horizon trials versus 21 for zero-shot GPT-6 (Astra), with per-decision latency of 52.5 ms on one RTX 4090 GPU using 5.3 GiB of memory, compared with 78 ms for CHORUS and 6.46 s on average for GPT-6 through its API — a factor of about 120.
  • Per-task planner results (10 trials each): Door 6/10, Corridor 7/10, Handover 10/10, Table 10/10, Sofa 10/10 (average 86%), against 42% for GPT-6 (Astra), 12% for CHORUS, and 98% for the human operator upper bound. CHORUS fails every trial on all tasks except the handover, where it succeeds 6 of 10 times.
  • Why CHORUS fails: Each robot runs its own copy of the policy and advances asynchronously, with no mechanism to align when robots act, so skills start at different moments — fatal when two robots carry an object together.
  • Real-world success: Across all real-world tasks the teams succeed in 92 of 100 trials. Stair carry is the hardest at 7/10, because each step leaves room for little more than one foot.
  • Scaling beyond training: The same policy, trained with two identical G1 robots, drives simulated teams of four, eight, and ten G1 robots carrying a table. Real-world heterogeneous teams (four G1; three G1 plus a person; a G1 plus a PM01 plus a person; a PM01 plus a person) succeed in 37 of 40 trials.
  • Parameter sharing matters: Independent actors reach only 22.05%, whereas the shared actor draws on the experience of all agents and keeps parameter count independent of team size.
  • Shared actor is what enables scaling: The shared actor has no notion of team size, and partners affect each robot only through the object. Note that the paper states simulation and real-world success rates are not directly comparable, since simulation samples initial positions over a wide range while real trials start from a nominal configuration with small perturbations.

Methodology in Plain English

The system has two levels.

Low level — turning a skill into motion. The team has a small vocabulary of team skills: lift, put down, a step in each direction, and a rotation in place. Every skill begins and ends at one of two static "hub" poses — standing empty-handed, or standing while holding the object — so any skill can follow any other and long tasks become chains of discrete choices. Reference motions come from motion-capture-style recovery of video of two people doing the task, retargeted to the robots, plus procedurally generated motions via inverse kinematics. Instead of learning balance from scratch, the authors freeze an existing foundation tracking model as a base policy and learn a residual policy that corrects its output. All robots share one actor, trained with multi-agent PPO under a centralized critic (which sees all agents' observations during training, concatenated in a fixed order, and outputs one value per agent) and decentralized execution. Terminations are coupled: if any agent fails, the episode ends for the whole team. Training uses long random chains of legal skills with random waiting times, varies object size by ±10% and mass by ±20% around the nominal specification the policy is given, and randomizes friction, robot properties, and external pushes.

High level — deciding what to do and when. A leader's head camera and the task instruction are fed to a fine-tuned Qwen3-VL-2B, which outputs one token from the skill vocabulary (plus hold and done). It is queried after each skill completes, so each decision is made from a view of the team at rest. The "when" is handled separately: the leader speaks the skill token, each follower recognizes it on an on-board microphone, and every robot schedules the skill on the next slot of a coarse shared grid of control ticks, with the leader offsetting its own start by the expected acoustic delay. Because the grid is coarser than the variation in acoustic delay, the team starts together without exchanging a single packet. The leader is a role any robot can take, and a person can follow the spoken commands or take over as leader.

Deploying to bigger or different teams. To run N robots, each is assigned a slot around the object, which defines its reference and contact targets; the same policy runs on all of them. Partners with different bodies execute the same skills with their own controller trained with the same recipe.

Why This Matters

Impact on research. The paper argues that decentralized whole-body collaboration among physically coupled humanoids has lacked a common testbed, since single-robot loco-manipulation benchmarks dominate and multi-agent benchmarks rarely involve whole-body contact with a shared object. HH-Bench and the "train with two, deploy with many" result give the field a reusable evaluation target and evidence that shared-actor policies can generalize across team size and embodiment.

Real-world applications:

  • Moving furniture and large or heavy household items — tables, bed frames, sofas — through doors and corridors in homes.
  • Warehouse and logistics handling of large boxes, pallets, and handovers between robots at different heights, such as loading from a truck bed.
  • Construction and industrial carrying, including shoulder-carrying long loads like rebar and pulling hoses.
  • Human–robot teams where a person works directly alongside one or more humanoids, and can even act as the leader by speaking commands.

Industry relevance. The approach adds capability to existing humanoid platforms without retraining per team size and without inter-robot networking, which lowers deployment cost and avoids dependence on communication infrastructure. A frozen foundation tracker plus a lightweight residual and a 2B-parameter planner that runs in 52.5 ms on a single consumer-grade GPU is a practical integration story for robot vendors.

Future Directions

  • Robustness of the speech channel. The authors note synchronization relies on followers hearing the leader, requiring a reasonably quiet environment and reliable sound pickup. Alternatives or fallbacks for noisy sites are left open.
  • Cheaper planner training. Training the planner for a new task requires demonstrations collected manually by teleoperation, which the authors call time-consuming; reducing this data burden is an explicit open problem.
  • Task coverage and composition. Since HH-Bench tasks can be chained (for example, a stair carry, a dolly placement, and a handover of a second box to a third robot), extending evaluation to longer compositions where robots leave one team and join another is a natural next step.
  • Heterogeneity at scale. The paper demonstrates a PM01 and human partners, but systematic study of larger heterogeneous teams and of policies trained with more than one platform in the loop is not addressed.

Target Audience

Researchers and engineers working on humanoid whole-body control, multi-agent reinforcement learning, and robot task planning; benchmark designers interested in physically coupled multi-robot evaluation; and robotics practitioners evaluating whether existing humanoid platforms can be extended to team-based loco-manipulation without retraining or inter-robot networking. Beginners will find the high-level framing accessible, but the training details assume graduate-level familiarity with RL for legged robots.

Authors’ abstract

Collaboration promises to extend humanoid capabilities beyond what individual robots can achieve. However, realizing this potential requires coordinated whole-body interactions, coherent long-horizon planning, and precise synchronization across robots. We introduce Heave-Ho, a decentralized framework for long-horizon collaborative multi-humanoid loco-manipulation. At the low level, we develop a multi-agent post-training framework that adapts a humanoid foundation tracking…

The full abstract on arXiv
Read the original paper