DexForesight: Grounding Dexterous Policies with Implicit Spatial Foresight

AI summary

Overview

Research area: Robotics — dexterous manipulation, vision-language-action (VLA) policies, and world-action models (WAMs).

Technical level: Advanced. The paper assumes familiarity with VLA architectures, diffusion/flow-style action experts, forward kinematics, and representation alignment losses.

Scope: A framework that trains an action-conditioned 3D spatial world model on offline depth, hand kinematics, and demonstrations, then transfers its predictive features into a dexterous policy's latent space so that deployment requires only RGB, language, and robot state.

What This Paper Is About

Dexterous hands are hard to control because fingers move in ways cameras often cannot see, and a policy must anticipate how its own articulated hand and the surrounding scene will change under an action, not just recognize the current image. Existing world-action models learn this kind of "spatial foresight" by explicitly predicting future images, motion, or 3D states at inference time, which costs computation and may force the model to reconstruct appearance details irrelevant to action choice. DexForesight instead teaches a policy to carry that predictive knowledge inside its own latent features during training, so that at deployment the world model, depth encoder, and all future decoders are discarded.

Key Contributions

1. Implicit foresight via latent alignment. DexForesight aligns learnable policy queries with the predictive query features of a frozen action-conditioned spatial world model, then uses the aligned tokens to condition the action expert — enabling spatial foresight with no world-model execution, no online depth reconstruction, and no explicit future generation at deployment.

2. An action-conditioned spatial world model combining scene depth and hand kinematics. Trained on approximately 370 hours of demonstrations across 60 tasks in five categories, it jointly predicts point-wise 3D displacements of persistent hand anchors (covering fingers occluded from cameras via forward kinematics) and DINOv3 visual feature residuals, using demonstrated controls as conditioning and depth only as a current-scene input.

3. A cross-layer depth fusion design. Offline depth tokens (from StereoAnyVideo for real data, MuJoCo rendering for simulation, encoded by OmniVGGT) are injected by cross-attention at backbone layers {2, 6, 10, 14, 18, 22, 26} and recombined through a token-wise router at layers {4, 8, 12, 16, 20, 24, 28}, following the Attention Residuals aggregation principle.

4. Evaluation on 11 DexJoCo simulation tasks and three real-robot tasks. DexForesight improves overall simulation success from 49.4% to 60.5% over its GR00T N1.7 backbone, and reaches 58.3% real-robot success versus 50.0% for the strongest baseline, while adding only 3.0 ms (1.3%) to per-step policy inference.

Main Findings

• Simulation gains: On 11 DexJoCo tasks (100 trials per task), DexForesight reaches 60.5% overall — 9.6 points above the strongest baseline Fast-WAM (50.9%) and 11.1 points above GR00T N1.7 (49.4%). It leads every task group at 76.3% single-arm and 41.6% bimanual. Baselines: π0.5 35.1% overall, Cosmos3 49.5%, Fast-WAM 50.9%.

• Real-robot gains: Across Screw On Lid, Close Box, and Insert Battery (20 trials each, held-out object positions, Screw also varying cup color), DexForesight achieves 58.3% overall versus WLA 50.0%, GR00T N1.7 48.3%, QwenVLA 30.0%, Cosmos3 30.0%, and Fast-WAM 10.0%. Per task: Screw 50.0%, Close 45.0%, Insert 80.0%.

• Predictive information survives in the deployed policy: A linear ridge probe from frozen policy features to future 3D hand-point displacements shows DexForesight error of 3.92 / 6.33 / 10.01 mm at 0.13 / 0.27 / 0.54 s, versus architecture-matched QwenVLA action-only at 4.12 / 7.70 / 13.59 mm and a zero-motion prior at 5.08 / 9.79 / 18.48 mm. The 0.54 s gap is a 26.3% error reduction, and the margin grows with horizon.

• Transfer beats direct prediction heads: Attaching prediction objectives straight to the policy is weaker: a hand-flow head reaches 53.1% (3.7 points above GR00T), a DINO future-residual head 45.5%, both heads together 49.5%, and depth with AttnRes 47.4% — all below DexForesight's 60.5%.

• Teacher targets matter: Using pre-action world queries (no control conditioning) yields 52.7%; shuffling the conditioning actions drops success to 41.5%, below the backbone; replacing hand-motion targets with a future-action target gives 48.4%.

• Depth and kinematics are essential teacher ingredients: Removing offline depth lowers simulation success by 10.8 points to 49.7% and real-robot success by 15.0 points to 43.3% (35.0% Screw, 25.0% Close, 70.0% Insert), even though deployed inputs are unchanged. Replacing cross-layer fusion with context depth tokens collapses performance to 42.0%.

• Cheap inference: Policy inference is 227.4 ms for QwenVLA versus 230.4 ms for DexForesight, while Cosmos3 takes 930 ms to generate a 16-frame future video.

Methodology in Plain English

The work splits learning into two stages.

Stage one — teach a model to imagine hand and scene motion. The researchers build a "spatial world model" on a RynnBrain-initialized backbone. Its inputs are multi-view RGB, a language instruction, robot-state history, a point cloud of fixed points on the hand's URDF links computed by forward kinematics (so fingers hidden behind the palm are still represented), and — in a final training stage only — depth maps. The depth maps are estimated offline from stereo video with StereoAnyVideo for real data, and rendered directly in simulation. Depth is folded into the backbone at several layers by cross-attention rather than being pasted in as extra tokens.

A set of learnable "world queries" reads the backbone and is then conditioned on the demonstrated actions through an adapter, using K = 4 future offsets sampled across a 16-frame window. Two experts use those queries to predict (a) the 3D displacement of every hand anchor point between now and each future offset, and (b) the residual in frozen DINOv3 image features over the same interval. Training also randomly truncates the exposed action prefix ("future-horizon dropout") and adds an auxiliary loss reconstructing current hand, state, scene, and instruction features. Training proceeds in three stages: warmup, dynamics pretraining (no depth), then depth injection on a 100-hour subset.

Stage two — squeeze that knowledge into the policy. The world model is frozen. The policy gets its own set of learnable queries that read selected hidden layers of its vision-language model. Those policy queries are matched to the world model's queries with two losses: a token-wise cosine matching loss and a relation loss that matches the pairwise similarity matrices of the two query sets. The matched query tokens are projected and concatenated onto the action expert's context, so they directly condition action generation. Alignment and behavior cloning are optimized jointly.

At deployment the world model, depth encoder, hand-geometry encoder, and both prediction experts are deleted. The policy runs on RGB, language, and robot state alone, computing its query context once per observation and reusing it across action sampling. In simulation the policy is built on GR00T N1.7; on the real robot it is built on QwenVLA, using a Sharpa North robot with 22-DoF Sharpa Wave hands. Simulation uses a Franka Panda with a 16-DoF Allegro hand.

Why This Matters

Impact on research. The paper offers evidence that predictive world knowledge can be distilled into policy latents rather than re-executed at test time, and that this route outperforms simply bolting prediction heads onto a policy. It also argues that for dexterous manipulation, the right prediction target is embodiment-aware — articulated hand-point flow plus scene feature change — rather than full future observation reconstruction. The probe experiment provides a measurement methodology for checking whether transferred knowledge actually persists in a deployed policy.

Real-world applications:

• Warehouse and logistics assembly, where inserting batteries, closing containers, and screwing lids require precise alignment under partial occlusion.
• Manufacturing and small-parts assembly, where fingers frequently hide the contact region from cameras.
• Service and household robotics with multi-fingered hands operating in cluttered, rearranged environments.
• Cost-sensitive embedded deployment, since the approach avoids depth sensors, online reconstruction, and multi-step video generation at every control tick.

Industry relevance. The 3.0 ms (1.3%) inference overhead relative to the base policy, contrasted with 930 ms for generating a 16-frame future video, makes implicit foresight attractive for real-time control loops. The recipe is also backbone-agnostic in practice — demonstrated on both GR00T N1.7 and QwenVLA — which matters for teams that already have a pretrained VLA they want to improve without changing their deployment stack.

Future Directions

• Scaling the demonstration corpus and task taxonomy. The world model was trained on 60 tasks in five categories, but depth was estimated for only a 100-hour subset covering three categories; whether depth-stage coverage should expand to deformable objects and tool use is untested.

• Relaxing the training-time dependence on demonstrated controls. The teacher is conditioned on action sequences the policy never sees, and shuffling those actions degrades success to 41.5%; finding alignment schemes that need less precise control conditioning, or that close the gap, is an open problem.

• Better depth injection mechanisms. Replacing cross-layer fusion with context depth tokens dropped performance to 42.0%, and the authors hypothesize that concatenation perturbs pretrained multimodal features — suggesting the fusion design deserves systematic study.

• Extending beyond hand-point flow and visual residuals. Substituting future-action targets gave only 48.4%, so the question of which predictive targets best capture task-relevant dynamics — contact forces, object pose, latency-aware dynamics — remains open.

Target Audience

Researchers and engineers working on dexterous manipulation, vision-language-action models, and world models for robotics; practitioners who want the benefits of predictive world modeling without paying its inference cost; and graduate students studying representation transfer, auxiliary supervision, or latent alignment between pretrained models. Readers need comfort with VLA architectures and robot kinematics, but the experimental narrative and ablations are legible to anyone familiar with imitation-learning benchmarks.

Authors’ abstract

Dexterous manipulation requires rich spatial foresight over both the surrounding scene and the robot hand. Existing world-action models often acquire such predictive knowledge through explicit future prediction. We introduce DexForesight, a framework that learns spatial foresight implicitly by injecting dynamics-aware world representations into the latent space of a dexterous policy. During training, DexForesight learns an action-conditioned spatial world model from offline…