PointWAM: 3D World Action Modeling for Dexterous Robotic Manipulation

AI summary

Overview

• Research area: Robotics — dexterous robotic manipulation, world action models (WAMs), 3D point-based policy learning, and pre-training from human demonstration videos.
• Technical level: Advanced. The paper assumes familiarity with vision-language-action models, transformers, point clouds, and robot retargeting.
• Scope: PointWAM is a single system that represents the world as two sets of 3D point trajectories — scene points and hand keypoints — forecasts how both move together in one space-time coordinate frame, and retargets the forecast hand motion into robot actions, pre-trained on 1.15M human demonstration episodes.

What This Paper Is About

Dexterous manipulation depends on precisely coordinating fingers in 3D space relative to object surfaces, but existing world action models usually represent the world as RGB frames or latent video features and predict actions as end-effector poses or joint angles, so they struggle to capture 3D spatial structure and contact geometry. PointWAM instead decomposes the world into a scene (the environment) and hands (the actor) and jointly forecasts both as 3D point trajectories within a shared space-time coordinate frame. Given a colored point cloud and a language instruction, it predicts how the scene and hands co-evolve over time, then retargets the forecast hand motion into robot actions — allowing pre-training on human demonstration videos that contain no robot actions at all.

Key Contributions

1. PointWAM, a unified 3D world action model. It jointly forecasts the 3D trajectories of scene points and hand keypoints in one shared space-time coordinate frame, unifying world modeling and action modeling.
2. Evidence that 3D world modeling helps dexterous manipulation. Scene-trajectory supervision improves average DexJoCo success by 10.9 percentage points over the same model without it, and the 3D encoder (Mosaic3D) outperforms lifted image (DINOv3) features by 5.4 percentage points.
3. Demonstrated transfer from human videos. Pre-training on human demonstrations from EgoDex and VITRA improves average DexJoCo success by 56.9 percentage points over training from scratch, and the gain scales with both data and model size.
4. State-of-the-art dexterous results, including on a real robot. With pre-training, PointWAM surpasses the prior best multi-task success rate on DexJoCo by 11.7 percentage points and outperforms robot-pre-trained VLAs on a real robot by up to 25 percentage points.

Main Findings

• DexJoCo multi-task result: With one policy across ten DexJoCo tasks in the object-placement setting, PointWAM reaches 69.0% ± 2.0 average success rate, versus GR00T N1.6 at 57.3% ± 1.8, Fast-WAM at 52.9% ± 1.5, π0.5 at 50.0% ± 1.9, PointACT at 33.7% ± 1.8, and DP-T at 31.0% ± 1.7. The margin over the strongest baseline is 11.7 percentage points, and over π0.5 it is 19.0 points. PointWAM is highest on eight of the ten tasks and also ranks first with one policy per task.
• Robustness gain: Under full randomization of camera pose, lighting, and texture, PointWAM's margin over the strongest baseline widens from 5.3 to 22.2 percentage points (Appendix Tables 5 and 6).
• Pre-training transfer is large and specific to the 3D representation: With Volt-H and 60k fine-tuning iterations fixed, EgoDex pre-training raises PointWAM's SR nearly fivefold, from 12.1% to 60.1%, while baselines change by at most 1.4 percentage points (GR00T N1.6 from 57.3% to 56.6%, Fast-WAM from 52.9% to 54.3%, PointACT from 33.7% to 33.9%). Fast-WAM also forecasts the future — but as video frames — and gains little.
• Data scaling: With the backbone and 60k fine-tuning iterations fixed, SR rises from 60.1% on the 324K EgoDex episodes to 64.9% with 736K episodes, and to 69.0% with the full 1.15M episodes. EgoDex alone already surpasses GR00T N1.6 (57.3%), and the full corpus adds 8.9 percentage points.
• Model scaling: SR stays near 50% for Volt-S (39M, 52.3%) and Volt-B (106M, 50.1%) but rises to 64.2% for Volt-L (326M) and 69.0% for Volt-H (655M).
• Ablation — vision encoder: Mosaic3D on the point cloud reaches 64.4% ± 1.7; lifting DINOv3 image features onto the points, as PointWorld does, drops to 59.0% ± 1.3. Encoding 3D structure in the features themselves, not only in their positions, matters.
• Ablation — world modeling: Removing scene-trajectory supervision from both pre-training and fine-tuning lowers SR from 64.4% ± 1.7 to 53.5% ± 2.7. A 2D-frame variant that forecasts DINOv3 latents of the frame at t+T reaches 60.1% ± 1.5 — 6.6 points above no world modeling but 4.3 below the 3D reference. Forecasting 3D point trajectories beats forecasting image latents.
• Ablation — hand representation: Ten anatomical keypoints reach 64.4%, whereas depth-sampled hand surface points reach 43.6% with 10 points and 44.6% with 512 points per hand.
• Ablation — voxel size: 1 cm voxels reach 64.4%, above 0.5 cm (59.9%) and 2 cm (52.2%). Coarser voxels may blur contact geometry; finer ones add little beyond the 224×224 depth input.
• RoboDojo-Precision (gripper transfer): With no robot data beyond the benchmark's own demonstrations, PointWAM reaches an average score of 8.8 and 4.8% SR. GR00T N1.6 reaches 6.2 and 2.3% and leads on three of the eight tasks; Fast-WAM reaches 0.3 and 0.2%; PointACT reaches 1.6 and 0.0% even though it observes depth.
• Real robot: PointWAM outperforms GR00T N1.6 and π0.5 over all trials of each task. On banana pick-and-place it performs comparably with the in-domain banana but reaches 58.3% SR on the out-of-domain unseen box, against 16.7% and 25.0%. On square-peg insertion it reaches 50.0% SR against 16.7% and 25.0%. All three methods usually lift both parts, but PointWAM inserts in 12 of 21 such trials, against 4 of 19 and 6 of 18.

Methodology in Plain English

PointWAM treats a manipulation episode as two families of moving points in one 3D coordinate frame.

Representation. The scene is a colored point cloud from RGBD frames and robot states give the end-effector pose and finger joint angles of each arm. Hands are represented by K keypoints, ten per hand, placed in the scene's coordinate frame. Both the scene points and the hand keypoints are expressed as displacements from the current time t over a horizon T, so the model predicts where each point moves rather than where it is.

Trajectory forecaster. A frozen Mosaic3D encoder, pre-trained to align 3D point features with text, produces a feature per scene point. Each hand keypoint gets a feature from an MLP over its one-hot index plus a hand ID. Scene and hand features are placed on a shared 1 cm voxel grid, aggregated with a sparse-convolutional stem, and divided into patches that are linearly projected into vision tokens (following Volt). Instruction text is projected from a frozen text encoder into instruction tokens. Vision and text tokens are processed by a transformer encoder with global self-attention and axial 3D rotary position embeddings. The output tokens are detokenized back to per-point features and combined with the stem features through a residual MLP to retain local detail. Two MLP heads then predict scene displacements and hand displacements.

Action retargeter. Only the hand forecasts enter the retargeter; scene forecasts supervise the shared backbone. Hand displacements at each future step are encoded, a step embedding and a state embedding are added, and a transformer decoder with learnable action queries produces an action chunk of end-effector poses and finger joint targets.

Training on two data sources. Human videos record no robot states, so both trajectories are estimated from the videos: hand trajectories come from the 3D hand poses annotated in EgoDex and reconstructed by VITRA and are mapped to the robot hand's ten keypoints; scene trajectories come from CoTracker3 dense 2D point tracks lifted to 3D using VGGT-Ω monocular depth scaled to the annotated hands. This yields 1.15M episodes in the same trajectory format as the robot demonstrations. Robot scene trajectories come from replaying DexJoCo demonstrations or, on the real robot, from CoTracker3 tracks lifted with Fast-FoundationStereo depth. The backbone and both trajectory heads are pre-trained from scratch on human demonstrations, then fine-tuned with joint trajectory and action supervision (with λ = 1) while the retargeter is trained from scratch. Scene weights sum to one over points and steps and favor motion without excluding static points.

Why This Matters

Impact on research. The paper argues that observing the scene in 3D is not enough — forecasting it matters, adding 10.9 percentage points over forecasting the hands alone — and that a shared 3D point-trajectory representation, not world modeling alone, is what lets human videos transfer to robots. This reframes how WAMs might be built: rather than predicting future RGB frames or video latents and learning how those map to motor commands, predict the 3D motion of scene points and hands and retarget only the hands to actions. Because scene forecasting needs no task-specific object or keypoint selection, it avoids a dependency that constrains Point Policy, Dexterous Point Policy, General Flow, MolmoMotion, and μ0.

Real-world applications:

• Dexterous robotic manipulation with multi-fingered hands, such as tool use and bimanual assembly tasks that today's policies rarely complete.
• Learning manipulation skills largely from human demonstration video, which is far cheaper to collect at scale than robot teleoperation.
• Precise placement tasks with parallel-jaw grippers, demonstrated on RoboDojo-Precision tasks such as inserting a key or fastening screws.
• Real-world service and household manipulation, as evaluated on the OpenArm humanoid for banana pick-and-place and bimanual square-peg insertion.

Industry relevance. PointWAM pre-trains without any robot demonstrations beyond the target benchmark, yet surpasses baselines (GR00T N1.6, π0.5) that pre-train on large-scale robot and web data. For robot companies, that suggests dexterous skill acquisition could lean on existing human video corpora rather than expensive robot data collection. The paper also reports transfer from human hands to parallel-jaw grippers, indicating the representation is not tied to one embodiment.

Future Directions

• Richer per-point information. PointWAM currently sees the world only through points, so it can miss details finer than the point spacing, such as printed text, and surfaces depth sensing cannot capture, such as transparent objects. The authors propose extending what each point carries — from finer appearance to physical properties such as mass and friction.
• Extending the shared 3D representation to more embodiments and tasks, given that the paper's only embodiment-transfer evidence is the RoboDojo-Precision gripper benchmark, where success rates remain low (4.8% SR) and GR00T N1.6 still leads on three of eight tasks.
• Scaling further. The data-scaling curve was still rising from 736K to the full 1.15M episodes, and the model-scaling curve was still rising from Volt-L (326M) to Volt-H (655M), leaving open how far both can be pushed.
• Reconciling scene prediction with action prediction more tightly, since scene forecasts currently supervise the shared backbone but do not enter the retargeter, and the paper's 2D-frame ablation suggests the choice of forecast target materially changes performance.

Target Audience

Researchers and engineers working on robot learning — particularly world action models, vision-language-action models, point-cloud policies, and dexterous manipulation — who are interested in 3D representations for control and in pre-training from human demonstration video. The paper is also useful to practitioners evaluating whether human video can substitute for robot demonstrations, and to readers tracking how the object-placement benchmark DexJoCo and the RoboDojo-Precision benchmark are being used to test generalization.

Authors’ abstract

World action models jointly learn to forecast world dynamics and predict robot actions, such that the learned internal world dynamics guide accurate actions. Existing approaches typically represent the world as RGB frames or latent counterparts while predicting actions as end-effector poses or joint angles, but they often struggle to capture the 3D spatial structure and contact geometry central to dexterous manipulation. We introduce Point World Action Model (PointWAM), a 3D world action model that decomposes the world into a scene (i.e., environment) and hands (i.e., actor), and jointly forecasts both as 3D point trajectories within a shared space-time coordinate frame. This explicit, disentangled representation enables effective pre-training on large-scale human demonstration videos without requiring any task-specific object or keypoint selection. Given a colored point cloud and a language instruction, PointWAM predicts how the scene and hands co-evolve in 3D space over time, then retargets the forecast hand motion to robot actions. Pre-training on human videos improves average DexJoCo success by 56.9 percentage points, and scene-trajectory supervision adds 10.9 points over forecasting the hands alone. With both, PointWAM surpasses the prior state of the art on ten DexJoCo tasks by 11.7 points and outperforms strong VLAs on a real robot.