DynaRend: Learning 3D Dynamics via Masked Future Rendering for Robotic Manipulation

AI summary

Overview

Research area: Robotics — self-supervised visual representation learning for language-conditioned robotic manipulation.

Technical level: Intermediate. The paper assumes familiarity with self-supervised pretraining (masked modeling, contrastive learning), neural radiance fields / volume rendering, and keyframe-based imitation learning, though the core idea is explainable without deep math.

Scope: The paper introduces DynaRend, a pretraining framework that learns 3D-aware, dynamics-informed triplane scene representations from multi-view RGB-D video via masked reconstruction and future prediction with differentiable volumetric rendering, then transfers them to manipulation policies.

What This Paper Is About

Robotic manipulation policies need to understand 3D space, task semantics, and how a scene will change after an action — but existing self-supervised pretraining methods usually capture only one of these: 2D masked image modeling learns static semantics, video prediction models learn 2D dynamics, and explicit 3D methods like dynamic Gaussians are structurally complex and hard to scale. DynaRend's goal is to learn a single unified 3D representation that jointly encodes geometry, semantics, and future dynamics, and to do so using only the fixed camera views available in real-world setups rather than dense calibrated novel views.

Key Contributions

1. A triplane-based masked future rendering framework. DynaRend reconstructs a point cloud from multi-view RGB-D observations, projects it onto three orthogonal feature planes (triplanes), masks a random subset of those features, and passes them through a reconstructive network followed by a predictive network to obtain both current and future scene representations.

2. Rendering-based supervision for three properties at once. Differentiable volumetric rendering produces RGB, semantic, and depth outputs from both the reconstructed and predicted triplanes, giving the model joint supervision over spatial geometry, task semantics, and future dynamics in one objective.

3. Reduced dependence on dense camera setups. Instead of requiring many calibrated novel-view images (feasible in simulation but impractical in the real world), the method synthesizes target views using a pretrained generative model (See3D) with depth estimated by Depth Anything v2, keeping all pretraining data from fixed viewpoints.

4. Systematic empirical validation. A study of pretraining objectives, mask ratio, rendering loss terms, and view synthesis, plus evaluation on RLBench (18-task and 71-task settings), Colosseum (12 perturbation types across 20 tasks), and five real-world tasks.

Main Findings

• RLBench 18-task state of the art: DynaRend reaches an average success rate of 83.2 with an average rank of 1.5, compared with RVT-2 at 81.4, 3D Diffuser Actor at 81.3, 3D-MVP at 67.5, RVT at 62.9, PerAct at 49.4, and C2F-ARM-BC at 20.1. The paper reports a 32.3% average success-rate improvement over the RVT baseline.

• Best efficiency trade-off in the 18-task setting: DynaRend's reported inference speed is 19.6, versus 20.6 for RVT-2, 11.6 for RVT, 11.6 for 3D-MVP, 4.9 for PerAct, and 1.4 for 3D Diffuser Actor.

• Scales to 71 RLBench tasks: DynaRend achieves 76.6 average success rate (Group 1: 81.4, Group 2: 71.8), an 8.1% improvement over two-stage baselines (SPA 70.8, MAE 68.0, VC-1 67.9, DINOv2 67.1, CLIP 66.2, MVP 66.2, MoCov3 63.9) and a 25.2% improvement over single-stage RVT (61.1). Notably, DynaRend is pretrained only on task-relevant multi-view RGB-D data, without large-scale external pretraining datasets.

• Robustness under environmental perturbation (Colosseum): Across 12 perturbation types on 20 tasks, DynaRend shows consistently higher success rates than 2D pretraining methods (MVP, R3M), 3D pretraining (3D-MVP), and RVT trained from scratch, with the largest gains in object and environment texture variations — achieved without domain randomization during training.

• Both pretraining objectives help, and they are complementary: Ablations on 18 RLBench tasks: no pretraining 76.7 (−6.5), reconstruction only 80.7 (−2.5), future prediction only 78.9 (−4.3), both 83.2. Future prediction gives the larger single-objective gain.

• RGB and semantic losses matter most: Removing the RGB loss drops performance to 78.2 (−5.0) and removing the semantic loss to 80.4 (−2.8), while removing depth supervision gives 82.0 (−1.2) — a smaller effect the authors attribute to the triplanes already encoding 3D structure from depth-projected point clouds.

• View augmentation helps: Removing novel-view augmentation reduces the average success rate to 79.8 (−3.4), supporting the claim that synthetic view diversity reduces overfitting to the limited fixed camera views.

• Masking ratio matters: The paper reports that both no masking and an excessively high mask ratio degrade performance, while a moderate ratio improves generalization — the specific optimum value is presented in a figure rather than as a number in the text provided.

• Real-world results: On five tasks (Put Item in Drawer, Stack Blocks, Sort Shapes, Close Pot, Stack Cups) with 20 rollouts each, DynaRend averages 57 versus 37 for 3DA and RVT-2, and 26 for RVT. With distractor objects added at test time, DynaRend averages 45 versus 20 (3DA), 16 (RVT-2), and 10 (RVT). The largest single-task gains include Sort Shapes at 85 versus 60 (RVT-2) and Stack Cups at 40 versus 15 (RVT-2).

Methodology in Plain English

The pipeline has three stages.

1. Build a 3D scene representation. From a set of calibrated multi-view RGB-D images, the method back-projects depth into a scene-level point cloud, encodes each point with an MLP, and projects the point features onto three orthogonal planes (the x-y, x-z, and y-z planes) using axis-aligned max pooling. The result is a "triplane" — three feature maps that together describe the scene more compactly than a voxel grid, point cloud, or 3D Gaussians, while still being spatially structured.

2. Pretrain by masking and future rendering. A random subset of triplane features is replaced with a learnable mask embedding, and CLIP text embeddings of the language instruction are concatenated in. A reconstructive network (a four-layer Transformer using SwiGLU, QK Norm, and RoPE) rebuilds the full current-scene triplane; a predictive network then takes that output and predicts the triplane of the nearest future keyframe. Both feature volumes are rendered into RGB, semantic, and depth maps using differentiable volumetric rendering: rays are cast through the volume, features are queried at sampled points via bilinear interpolation and summed across planes, small MLP heads predict density, color, and semantic features per point, and these are integrated along the ray. Supervision comes from held-out target views — the current frame for reconstruction, a future frame for prediction — using mean squared error on RGB and semantics plus a scale-invariant log loss on depth. Semantic targets come from the vision foundation model RADIOv2.5 rather than DINOv2, which the introduction mentions only as an example of the type of model used. To avoid needing extra cameras, the method perturbs a base camera pose, warps the point cloud to that pose, uses See3D to synthesize a realistic image, and Depth Anything v2 to estimate its depth; these synthetic RGB-D pairs serve as additional supervision.

3. Fine-tune for manipulation. The reconstructive and predictive networks together act as the triplane encoder for a downstream policy. Following RVT, the task is framed as multi-view action value map prediction of the next keyframe (end-effector translation, rotation, and gripper state). Translation is predicted as heatmaps over the three orthogonal planes and supervised with cross entropy; rotation and gripper state are predicted by querying the triplane at the predicted translation and passing the feature through a lightweight MLP, also with cross entropy.

Training details: SE(3) augmentation with translations up to 0.125 m and z-axis rotations up to 45 degrees; a 16×16×16 triplane grid; about 60k pretraining steps and 30k fine-tuning steps in simulation (30k and 10k in the real world); batch size 256; learning rate 1×10⁻⁴ with cosine decay; 8 NVIDIA RTX 3090 GPUs. Real-world training used 30 demonstrations per task on a Franka Research 3 with two Orbbec Femto Bolt RGB-D cameras; simulation used a 7-DoF Franka Emika Panda with four RGB-D views and 100 demonstrations per task.

Why This Matters

Research impact. The paper argues that the field's pretraining paradigms are fragmented: 2D masked modeling captures semantics without dynamics, video prediction captures 2D dynamics without geometry, and explicit 3D representations (dynamic Gaussians, voxel grids) are expensive or complex. DynaRend shows that a single rendering-based objective can supervise semantics, geometry, and dynamics simultaneously, and that this can be done without large external pretraining datasets. It also offers a practical answer to a real bottleneck in rendering-based robot learning — the need for dense novel views — by substituting generative view synthesis.

Real-world applications:

• Household and service robots that must manipulate objects under changing lighting, colors, textures, and object sizes without retraining.
• Warehouse and logistics pick-and-place, where policies need to generalize across unseen items and added clutter.
• Deformable or physically involved tasks such as stacking cups and sorting shapes, where reasoning about how the scene will evolve after an action is essential.
• Lower-cost robot deployments, since pretraining works from a small number of fixed cameras rather than elaborate multi-camera rigs.

Industry relevance. The reported inference speed (19.6 in the 18-task setting, close to the fastest baseline) and the use of standard hardware (8× RTX 3090 for training, off-the-shelf RGB-D cameras and a Franka arm at test time) suggest the approach is compatible with practical deployment constraints, and the task-relevant-only pretraining removes the need to license or process large external video corpora.

Future Directions

• Improving the mask-ratio and objective balance. The paper shows performance is sensitive to the masking ratio and that depth supervision contributes relatively little; finding principled ways to set these — or representations that benefit more from depth — remains open.
• Extending beyond keyframe action value maps. The current formulation follows RVT's keyframe prediction with a low-level motion planner; whether the learned triplane dynamics can drive continuous or closed-loop control is not addressed in the provided content.
• Scaling data and tasks. The results cover 18 and 71 RLBench tasks, 20 Colosseum tasks, and five real-world tasks; behavior at much larger task and object diversity is not reported.
• Reliance on generative view synthesis. The method depends on See3D for novel views and Depth Anything v2 for their depth, so errors or artifacts from those models could propagate; the paper does not report an analysis of this dependence in the content provided. The limitations section is truncated in the available text, so the authors' own stated limitations and future work are not fully reported.

Target Audience

Robotics and embodied-AI researchers working on manipulation policy learning and self-supervised visual pretraining will get the most from this paper, particularly those interested in 3D scene representations, neural rendering as a pretraining signal, or sim-to-real transfer. It is also relevant to practitioners building general-purpose manipulation systems who need policies that survive domain shift without domain randomization, and to students with some background in self-supervised learning and neural radiance fields who want a clear example of combining reconstruction and prediction objectives in one representation.

Authors’ abstract

Learning generalizable robotic manipulation policies remains a key challenge due to the scarcity of diverse real-world training data. While recent approaches have attempted to mitigate this through self-supervised representation learning, most either rely on 2D vision pretraining paradigms such as masked image modeling, which primarily focus on static semantics or scene geometry, or utilize large-scale video prediction models that emphasize 2D dynamics, thus failing to jointly learn the geometry, semantics, and dynamics required for effective manipulation. In this paper, we present DynaRend, a representation learning framework that learns 3D-aware and dynamics-informed triplane features via masked reconstruction and future prediction using differentiable volumetric rendering. By pretraining on multi-view RGB-D video data, DynaRend jointly captures spatial geometry, future dynamics, and task semantics in a unified triplane representation. The learned representations can be effectively transferred to downstream robotic manipulation tasks via action value map prediction. We evaluate DynaRend on two challenging benchmarks, RLBench and Colosseum, as well as in real-world robotic experiments, demonstrating substantial improvements in policy success rate, generalization to environmental perturbations, and real-world applicability across diverse manipulation tasks.