SONIC: Supersizing Motion Tracking for Natural Humanoid Whole-Body Control

AI summary

Overview

Research area: Robotics — reinforcement-learning-based humanoid whole-body control, motion tracking, and vision-language-action (VLA) foundation models.

Technical level: Intermediate. The high-level framing (scaling laws, motion tracking as a foundational task, token interfaces) is accessible, but the methodology assumes familiarity with reinforcement learning, motion capture, and humanoid morphology.

Scope: This paper argues that physics-based motion tracking is a scalable foundational task for humanoid control, and demonstrates a single 42M-parameter policy trained on 100M+ motion frames that generalizes to unseen motions, transfers to real hardware, and serves diverse downstream applications through a unified latent token interface.

What This Paper Is About

Humanoid robots still rely on small, task-specific neural controllers with hand-tuned rewards that do not scale, because each new behavior (walking, dancing, getting up) requires a redesigned objective. This paper asks whether choosing the right foundational task — tracking dense, frame-by-frame motion-capture data — allows humanoid control to scale with model size, data, and compute the way language and vision models have. It then builds a complete system around that tracker so it can be driven by gamepads, VR headsets, video, text, music, and a vision-language-action model.

Key Contributions

1. Motion tracking identified as a scalable foundational task for humanoid control. The authors show that scaling data (4M to 100M frames), model capacity (1.2M to 42M parameters), and compute (~2k to 21k GPU hours) steadily improves tracking success and out-of-distribution generalization, unlike adversarial imitation methods that suffer mode collapse or task-specific reward engineering that plateaus.

2. A universal token space unifying heterogeneous motion inputs. Specialized encoders for robot motion, human motion (SMPL), and hybrid motion (upper-body keypoints plus lower-body robot motion) are quantized via Finite Scalar Quantization into a shared token, which drives a single control policy. A robot-motion decoder provides auxiliary supervision and acts as an implicit human-to-robot retargeting module.

3. A real-time kinematic motion planner for interactive control. An autoregressive planner generates 0.8–2.4 second motion segments conditioned on user commands, running in under 5 ms on a laptop and ~12 ms on a Jetson Orin, with replanning as often as every 100 ms. It enables navigation, squatting, kneeling, crawling, and boxing without retraining either the planner or the tracker.

4. Demonstrated integration with multimodal inputs and VLA foundation models. The same policy is driven by video, text, and music (via the GEM motion generator), VR teleoperation (whole-body and 3-point), and a GR00T N1.5 VLA model that autonomously performs five whole-body loco-manipulation tasks requiring coordinated hand grasping and foot placement.

Main Findings

• Scaling works along all three axes. The largest configuration (42M parameters, 100M frames, 21k GPU hours) reached 99.6% success and 23.8 mm MPJPE-L on unseen motion content, versus 98.0% and 27.7 mm for the smallest (1.2M parameters). Gains were largest on out-of-distribution motions, indicating scale improves generalization rather than just fitting the training distribution. More GPUs also improved asymptotic performance at fixed iteration count because larger batches stabilized optimization.

• The universal tracker beats specialized and generalist baselines. On test-content / test-repetition / PHUMA, SONIC achieved 98.5% / 99.2% / 97.2% success, compared to BeyondMimic's 82.0% / 85.4% / 73.8% and Any2Track's 61.6% / 69.4% / 78.5%. Tracking accuracy was 23.7 mm MPJPE-L, a 42% reduction over BeyondMimic's 40.9 mm. The high PHUMA result is notable because that dataset comes from a different retargeting pipeline and is substantially more out-of-distribution.

• A universal policy outperforms a purpose-built locomotion specialist. Against OpenHomie, a controller explicitly designed for velocity tracking, SONIC achieved 98.5% survival versus 43.0% across a 0–5 m/s command range. OpenHomie collapsed below 20% survival beyond ~2.5 m/s, while SONIC stayed near 100% up to ~4 m/s. OpenHomie's performance also plateaued beyond 8 GPUs, whereas SONIC kept improving with compute.

• Sim-to-real transfer is close to lossless. Across 124 real-world motion sequences, SONIC succeeded on 123 (99.2%) versus 124/124 in simulation, with 25.7 mm MPJPE-L in reality versus 22.3 mm in sim. The gap was smallest for the upper body (22.2 mm vs 21.8 mm) and largest at the feet (53.7 mm vs 29.0 mm), reflecting the difficulty of precise foot placement under real contact dynamics.

• The universal token space is dramatically easier for a VLA to learn than explicit poses. Averaged over three tasks, the FSQ token action space achieved 68% success versus 27% for explicit SMPL poses. The gap widened with task complexity: on the soda-can-to-trash-can task, the token space scored 60% while SMPL poses scored 0%, with the authors noting that pose prediction produced jerky motion and poor directional control.

• VLA-driven whole-body loco-manipulation is feasible. A GR00T N1.5 model fine-tuned on teleoperated data achieved 75% average success across five tasks, including stepping on a trash-can pedal with one foot while balancing on the other, and picking up a soda can, navigating, opening the lid, and throwing the can inside within a single action sequence.

• Multimodal control runs in real time. Video, text, and music inputs all drive the policy at ≥60 fps through a shared motion generation model, with smooth transitions between modalities (for example, starting with video control, switching to text, then handing off to music).

Methodology in Plain English

The core idea is to replace task-specific reward engineering with dense supervision from motion-capture data. Instead of writing a reward function for "walk forward" or "dance," the researchers give the policy a target pose every frame and reward it for matching the pose, velocity, and acceleration of the reference motion, plus penalties for jittery or unstable movement.

They formulate this as a reinforcement learning problem solved with PPO. The policy observes a 10-step history of joint positions, velocities, root angular velocity, gravity direction, and previous actions, plus a motion command. It outputs target joint positions tracked by PD controllers.

To handle multiple input types, they built three encoders: one for robot joint trajectories, one for 3D human poses, and one for hybrid inputs (head and hand keypoints plus lower-body robot motion). All three map into a shared latent space that is quantized with Finite Scalar Quantization into a small token — chosen over VQ-VAE because it is more stable under joint PPO optimization. A control decoder turns the token plus proprioception into motor commands, and an auxiliary decoder reconstructs the robot motion from the token alone, which regularizes the latent space and implicitly handles human-to-robot retargeting.

For real-world use, a separate autoregressive kinematic planner generates short motion segments in the same dataset distribution, and the tracker follows them. Domain randomization on physical parameters, root velocity perturbations, and motion commands keeps the tracker robust to noisy planner output. They evaluated on two held-out splits of their own data (unseen motion categories versus unseen repetitions of known categories) and on the external PHUMA benchmark, then deployed on a Unitree G1.

Why This Matters

Impact on research. The paper provides evidence that humanoid control has a scalable foundational task analogous to next-token prediction in language modeling. This reframes the field's default approach — hand-designed rewards per behavior — and offers a concrete alternative with measurable scaling curves. The universal token space also suggests a standard interface between high-level reasoning models and low-level whole-body control, which could simplify how VLA and world models are applied to robots. The public release of the BONES-SEED dataset (142,220 sequences, 288 hours, 522 actors, in both SOMA and Unitree G1 formats) lowers the barrier for others to build on this line of work.

Real-world applications:

• Teleoperation in hazardous environments. Natural whole-body control with running, jumping, crawling, and squatting supports remote operation in disaster response, inspection, and industrial settings where human-like mobility is required.
• Data collection for robot foundation models. The VR interfaces let operators generate high-quality whole-body teleoperation data, which the authors use directly to fine-tune a VLA — a reusable pipeline for scaling robot learning data.
• Whole-body loco-manipulation in homes and warehouses. Tasks like opening a pedal-operated trash can, carrying boxes, and relocating tools require coordinated hands and feet, which decoupled upper-body/lower-body controllers cannot express.
• Entertainment and character animation. Real-time gamepad-driven navigation with styles like drunken or stealth walking, continuous boxing without expert-model switching, and music-conditioned dancing map directly to games, virtual production, and theme-park robotics.

Industry relevance. This is NVIDIA work targeting a commercial humanoid platform (Unitree G1) with a commercial VLA model (GR00T N1.5), and the training budget (21k GPU hours, 128 GPUs) reflects an industrial rather than academic scale. The onboard inference budget (<5 ms planner on a laptop, ~12 ms on Jetson Orin) is realistic for deployed hardware. For companies building humanoid products, the paper offers a template: train one large tracker, then attach interfaces rather than retrain per task.

Future Directions

• Formal treatment of safety and energy efficiency. The authors explicitly list these as limitations for extended deployment; the tracker can lose balance under extreme conditions or very dynamic motions, and the paper does not quantify power consumption or fall consequences.

• Perception-coupled robustness. The current system relies on domain randomization and a critically damped spring filter to absorb noisy planner commands. Tighter integration with onboard perception — rather than a kinematic planner running on clean desired motions — is the natural next step toward autonomy.

• Closing the foot-placement sim-to-real gap. Foot error roughly doubled from simulation to reality (29.0 mm to 53.7 mm), the largest gap of any body part. Improving contact modeling or tactile feedback could raise precision-critical tasks like pedal stepping.

• Broader VLA generalization and scaling of the token interface. The VLA results cover five tasks with 10–20 trials each and success dropping to 60% on the most complex task. Whether the universal token space supports many more tasks, longer horizons, and instruction-following at the scale of language-conditioned manipulation datasets remains open.

Target Audience

Robotics researchers working on humanoid control, reinforcement learning for locomotion and manipulation, and sim-to-real transfer will find the scaling analysis, baselines, and ablation of the token action space most directly useful. Practitioners building VLA or teleoperation systems benefit from the interface design and the demonstration that compact quantized tokens outperform explicit pose targets. Engineers evaluating the feasibility of deploying large control policies on commercial humanoid hardware will find the compute budget, inference latency, and real-world success rates concrete. Readers primarily interested in the training pipeline will want to consult the accompanying BONES-SEED dataset release.

Authors’ abstract

Despite the rise of billion-parameter foundation models trained across thousands of graphical processing units (GPUs), similar scaling gains have not been shown for humanoid control. Current neural controllers for humanoids remain modest in size, target a limited set of behaviors, and are trained on a handful of GPUs. We show that scaling model capacity, data, and compute yields a generalist humanoid controller capable of natural, robust whole-body movements. We position motion tracking as a scalable task for humanoid control, leveraging dense supervision from diverse motion-capture data to acquire human motion priors without manual reward engineering. We build a foundation model for motion tracking by scaling along three axes: network size (1.2M to 42M parameters), dataset volume (100M+ frames from 700 hours of motion capture), and compute (21k GPU hours). Beyond demonstrating the benefits of scale, we further show downstream utility through a real-time kinematic planner that bridges motion tracking to tasks such as navigation, enabling natural and interactive control, as well as a unified token space that supports virtual reality (VR) teleoperation and vision-language-action (VLA) models with a single policy. Through this interface, we demonstrate autonomous VLA-driven whole-body loco-manipulation requiring coordinated hand and foot placement. Scaling motion tracking exhibits favorable properties: performance improves steadily with compute and data diversity, and learned policies generalize to unseen motions, establishing motion tracking at scale as a practical foundation for humanoid control.