HuGe: Scaling Humanoid Generalist Vision-Language-Action Models with Egocentric Experience

AI summary

Overview

Research area: Robotics — Vision-Language-Action (VLA) models applied to humanoid whole-body control, with a focus on using first-person ("egocentric") human video demonstrations as a scalable substitute for costly robot teleoperation data.

Technical level: Advanced. The paper assumes familiarity with VLA architectures (OFT, GR00T, π), diffusion/flow-matching action heads, reinforcement-learning whole-body controllers, and imitation-learning evaluation protocols.

Scope: The paper introduces HuGe, a family of humanoid generalist VLA models built on three existing VLA architectures that predict latent motion tokens for a pretrained whole-body controller, trained on over 2,000 hours of egocentric human and simulated/real robot data through a three-stage pipeline, and evaluated on the HumanoidArena simulation benchmark and a real Unitree G1 pick-and-drop task.

What This Paper Is About

Building a generalist robot policy for a humanoid is hard for two reasons: the robot must coordinate arms, torso and legs to stay balanced while manipulating objects, and it needs large amounts of diverse demonstration data that is slow, expensive and sometimes unsafe to collect by teleoperating the real robot. First-person human videos are an appealing alternative data source, but human hands and arms look different from robot hardware, and human recordings carry no robot joint-state (proprioception) information. HuGe's goal is to make that human experience actually usable: the authors pair standard VLA architectures with a pretrained whole-body controller, and add targeted data-adaptation tricks so that egocentric human demonstrations can be mixed into humanoid policy training.

Key Contributions

1. A family of humanoid generalist VLAs (HuGe) instantiated from three existing VLA designs — OFT, GR00T and π — without adding new modules or changing the architectures, trained on over 2,000 hours of egocentric human and robot data. All variants share one interface: a 66-dimensional action vector consisting of a 64-dimensional latent motion token in the token space of the pretrained SONIC whole-body controller, plus two binary hand open/close signals.

2. A three-stage training pipeline: Stage 1 egocentric pre-training on human and simulated data, Stage 2 humanoid adaptation on real G1 teleoperation data, and Stage 3 downstream task fine-tuning.

3. Data-adaptation methods for human-to-humanoid transfer: simulation playback generates 34-dimensional pseudo-proprioceptive states for egocentric data that lack robot state during pre-training, and a visual editing pipeline (built on Phantom) removes human hands and arms, inpaints the exposed regions and composites rendered robotic hands during downstream fine-tuning.

4. Empirical validation: state-of-the-art average success on the HumanoidArena simulation benchmark, plus real-world experiments on a Unitree G1 showing that visually adapted egocentric demonstrations improve a cola pick-and-drop task beyond robot demonstrations alone.

Main Findings

• Full-parameter fine-tuning wins on HumanoidArena (no pre-training). Comparing frozen backbone, LoRA and full fine-tuning, full fine-tuning reached average success rates of 55.00% (HuGe-OFT), 58.33% (HuGe-GR00T) and 60.24% (HuGe-Pi), exceeding LoRA by 19.29–38.81 percentage points. Frozen-backbone training was worst (0.00%, 24.05%, 30.24% respectively).

• Pseudo-proprioceptive states help two of three variants. Adding them during egocentric pre-training improved average success by 5.00 percentage points for HuGe-GR00T (61.67% to 66.67%) and 2.86 points for HuGe-Pi (62.14% to 65.00%). SitSofa in particular rose from 61.67% to 78.33% (GR00T) and 68.33% to 83.33% (Pi).

• Pseudo-proprioceptive states hurt the OFT-style variant. HuGe-OFT dropped from 56.43% to 54.05% (−2.38 points); the authors attribute this to the textual state interface, where input truncation leaves incomplete state descriptions.

• Scaling egocentric pre-training helps GR00T- and π-style models. Downstream performance improved progressively with more egocentric pre-training data, reaching average performance of 66.67% and 65.00% with the full corpus. HuGe-OFT, by contrast, declined at the larger scale, again tied to state truncation.

• State-of-the-art on HumanoidArena. HuGe-GR00T (66.67%) and HuGe-Pi (65.00%) exceeded the strongest listed baseline, π0.5 (53.81%), by 12.86 and 11.19 percentage points, and led all listed baselines on both the HOI and HSI averages. HuGe-OFT reached 54.05% but led on Boxing and VisNavi with 73.33% on each.

• A shared weak spot: DoubleDesk. All three HuGe variants scored between 0% and 15.00% on DoubleDesk, which the authors associate with repeated grasping attempts in the demonstrations and the emphasis on whole-body humanoid data during pre-training.

• Visual adaptation measurably narrows the human–robot appearance gap. Fréchet distance to real robot feature distributions dropped from 2.0311 to 1.3499 (33.54%) for Wan2.2, from 7.5697 to 5.5204 (27.07%) for VideoMAE V2, and from 305.5143 to 263.2697 (13.83%) for V-JEPA 2.

• Real-world gains from egocentric data on the Unitree G1. With 110 robot demonstrations only, success was 43.3% (13/30). Adding 400 raw egocentric demonstrations raised it to 60.0% (18/30); using visually adapted versions of the same demonstrations raised it to 66.7% (20/30), with action, state and language supervision unchanged.

• Failure analysis. Of the 10 failed trials in the real-world task, bottle pickup accounted for five, the largest single source of failure; the authors observed balance instability during grasping and attribute it in part to the weight of the dexterous hands.

• Data scale used. Stage 1 totalled 1,669.610 hours / 350,772 episodes (EgoDex 821.800 h, EgoVerse 47.080 h, Xperience-10M 478.330 h, BEHAVIOR-1K 322.400 h). Stage 2 totalled 417.580 hours / 31,463 episodes (Humanoid Everyday 16.470 h, UniFoLM-WBT 22.960 h, HIW-500 378.150 h). Stage 3 used HumanoidArena (3.030 h, 700 episodes), plus 400 egocentric (2.456 h) and 110 teleoperated (0.826 h) cola-task demonstrations.

Methodology in Plain English

The core design decision is to keep the VLA small in scope: rather than outputting raw joint torques, HuGe outputs a compact 64-number "motion token" that a separate, already-trained whole-body controller called SONIC knows how to turn into coordinated joint commands, plus two bits saying whether each hand is open or closed. That splits the problem into "decide what to do" (the VLA) and "stay balanced while doing it" (the controller). The authors then plug the same interface into three off-the-shelf VLA designs — OFT (a lightweight MLP head over action tokens), GR00T (a diffusion transformer generating action chunks via flow matching), and π (a diffusion transformer action expert attending to multiple backbone layers) — all using a Qwen3-VL-2B-Instruct backbone.

Training happens in three stages. Stage 1 learns language-conditioned action prediction from a large pile of human first-person recordings plus simulated robot data, all converted into SONIC motion tokens. Because human videos contain no robot joint angles, the team replays the inferred motions on a simulated G1 and records the resulting 29 joint angles, hand states and 3D gravity direction — a 34-dimensional "pseudo-proprioceptive" state. Stage 2 continues training on real humanoid teleoperation datasets to close the gap between simulated/human and real-robot observations. Stage 3 fine-tunes per task — for the real world, on a cola pick-and-drop task recorded with a Unitree G1 with Inspire dexterous hands, where a PICO 4 Ultra headset and wrist/waist/ankle trackers were used to record 400 human demonstrations. Since human arms would otherwise clash visually with robot arms, those demonstrations are edited: human hands and arms are segmented out, the hole is inpainted, and rendered robotic hands are composited back in, while the action targets, pseudo-proprioceptive states and language instructions stay identical.

Evaluation compares frozen vs LoRA vs full-parameter tuning, ablates the pseudo-proprioceptive state, sweeps the amount of egocentric pre-training data, and benchmarks against ACT, DP, FM, π0.5 (numbers taken from the original HumanoidArena study) and GR00T baselines trained from scratch or initialized from GR00T N1.7. On HumanoidArena, models were trained multitask on seven tasks (three Human-Object Interaction, four Human-Scene Interaction) with 100 demonstrations per task and evaluated over 60 episodes per task (three seed groups of 20).

Why This Matters

Impact on research. The paper reframes a data problem as a data-adaptation problem: it shows that egocentric human video can be injected into a humanoid policy stack if you supply the missing proprioception synthetically and fix the visual embodiment gap, rather than building a whole new human-to-robot architecture. Keeping the three VLA backbones unmodified makes the recipe directly comparable across designs and gives the community a controlled study of training strategy (stage order, state interface, data scale) for humanoid VLAs — a dimension the paper argues is under-explored relative to architecture work. The failure cases are equally informative: the OFT degradation points to a concrete engineering pitfall (text-serialized state truncation), and the uniformly weak DoubleDesk result marks where whole-body humanoid pre-training data does not transfer.

Real-world applications:

• Warehouse and logistics robots that pick, carry and deposit objects while walking — the exact composite of locomotion, grasping and placement tested here.
• Household assistance, where a humanoid must navigate rooms, open doors, sit and manipulate everyday objects (the HumanoidArena task set).
• Faster deployment of new robot skills by recording a human with a headset and wrist/ankle trackers instead of teleoperating a robot that may overheat or require shutdowns.
• Manufacturing or laboratory environments where teleoperation is unsafe or impractical but skilled human workers can be recorded doing the task.

Industry relevance. The pipeline is deliberately built on commodity parts — a 2B-parameter Qwen3-VL backbone, a single node of eight NVIDIA 4090 GPUs (48 GB each), DeepSpeed, with Stage 1 taking roughly 38 hours and Stage 2 roughly 12 hours. That makes the recipe reproducible for well-resourced labs without exotic compute. The stated motivation that joint overheating during teleoperation can trigger robot shutdowns and raise safety concerns is a practical pain point for anyone scaling humanoid data collection, and the demonstrated gain at a fixed robot-demonstration budget (110 robot demos throughout) is the kind of trade-off deployment teams care about.

Future Directions

• Fix the state interface. The OFT-style variant's regression — and its decline with more pre-training data — is attributed to truncation of text-serialized proprioceptive state. A more compact or dedicated state encoding would be the obvious next step, since it currently caps one of the three architectures.

• Bring in robotic-arm manipulation data. The authors explicitly propose incorporating large-scale robotic arm manipulation data into pre-training to address the near-zero DoubleDesk scores (0% to 15.00%), which they link to repeated grasping attempts in demonstrations and the whole-body emphasis of pre-training.

• Address balance under manipulation load. Balance instability during bottle grasping, partly attributed to dexterous-hand weight, was the dominant failure mode (five of ten failures at pickup). Closing this gap likely requires either hardware changes or controller-level compensation.

• Broaden real-world evaluation. Real-world experiments used only the π-style HuGe variant, a single pick-and-drop task, and 30 trials per configuration. Extending to the other two variants, longer-horizon tasks, and more trial repetitions would test whether the recipe generalizes beyond the reported setting.

Target Audience

Robotics researchers and graduate students working on VLA models, humanoid loco-manipulation, and learning from human video. It is most useful to practitioners who already understand imitation learning and transformer-based action heads and want a concrete, reproducible training recipe — stage ordering, state representation, data mixture, visual adaptation — rather than a new architecture. Embodied-AI engineers at companies building humanoid platforms will find the real-robot protocol and the failure analysis directly actionable; readers new to the field should start with the introduction and methodology sections before the tables.

Authors’ abstract

Extending Vision-Language-Action models (VLAs) to humanoid robots requires coordinated whole-body control and diverse demonstrations that are costly to collect in real-world scenarios. Although egocentric human demonstrations provide a complementary source of task experience, differences in visual embodiment and missing robot proprioception complicate transfer to humanoid policies. To address these challenges, we introduce HuGe, a family of Humanoid Generalist VLAs pre-trained on over 2,000 hours of egocentric human and robot data. Specifically, HuGe employs three existing representative VLA architectures to predict latent motion tokens for a pretrained whole-body controller, together with hand-control signals. Based on this, we introduce a three-stage training pipeline that combines egocentric pre-training, humanoid adaptation, and downstream fine-tuning. To facilitate transfer across embodiments, we construct pseudo-proprioceptive states through simulation playback during pre-training and modify arm appearance in egocentric observations through re-rendering during downstream fine-tuning. With this pipeline, HuGe achieves the state-of-the-art on simulation benchmark HumanoidArena. Furthermore, real-world experiments on the Unitree G1 indicate that augmenting robot demonstrations with visually adapted egocentric demonstrations significantly increases success on a pick-and-drop task.