CLAMP: Contrastive Learning for 3D Multi-View Action-Conditioned Robotic Manipulation Pretraining

AI summary

Overview

• Research area: Robot learning / robotic manipulation pre-training, combining 3D perception, contrastive representation learning, and visuomotor policy learning.
• Technical level: Advanced. The paper assumes familiarity with contrastive learning, Vision Transformers, relative positional encoding, and diffusion policies, and includes formal lemmas (Lemma III.1, Lemma III.2) analyzing attention behavior.
• Scope: A single paper presenting a 3D contrastive pre-training framework (CLAMP) built on point clouds and robot actions, evaluated across six simulated tasks and five real-world tasks on an ALOHA 2 bimanual robot.

What This Paper Is About

Robotic manipulation policies commonly reuse pre-trained 2D image representations, but those representations do not capture the 3D spatial structure of objects and scenes that precise manipulation requires. The authors introduce CLAMP, a pre-training framework that learns image and action representations from point clouds plus robot action histories, grounded by text describing objects, their positions, and task progress. The goal is to improve how efficiently and how well a downstream behavior-cloning policy can be fine-tuned on a small number of task demonstrations for tasks it has never seen.

Key Contributions

1. A 3D contrastive pre-training framework for manipulation that uses robot actions and dynamic wrist views together with fixed multi-view re-rendered observations from merged point clouds.
2. First application of STRING relative positional encoding using 3D coordinates derived directly from point clouds (rather than via depth, as in the 3D-STRING variants), which lets tokens from different camera views implicitly correlate when they correspond to nearby 3D regions.
3. A demonstration that dynamic wrist views are critical for high-precision tasks, and that pre-training both the encoders and the Diffusion Policy yields substantially better results than pre-training the encoders alone.
4. Empirical results across six simulated tasks and five real-world tasks showing improved fine-tuning sample efficiency and policy performance over state-of-the-art baselines, with pre-training and fine-tuning tasks that do not overlap.

Main Findings

• Simulation success rates (maximum across checkpoints, averaged over three seeds, 50 trials per checkpoint every 40k steps): "ALOHA Unleashed with CLAMP" reached 94.0% on Can Opener in Caddy, 98.0% on Screwdriver in Caddy, 51.3% on Pen in Container, 95.3% on Mug on Plate, 92.7% on Plate on Rack, and 95.3% on Drawer Open.
• Comparison to unmodified ALOHA Unleashed: the same backbone without CLAMP reached 68.0%, 78.0%, 16.0%, 54.0%, 76.0%, and 80.7% on the same six tasks respectively; "Pretrained ALOHA Unleashed" reached 70.0%, 86.7%, 20.7%, 62.0%, 76.7%, and 88.7%.
• ACT results: ACT with CLAMP encoders only reached 72.7% on Can Opener in Caddy, 92.0% on Screwdriver in Caddy, 10.7% on Pen in Container, 62.7% on Mug on Plate, 88.7% on Plate on Rack, and 28.0% on Drawer Open. Plain ACT reached 34.0%, 82.0%, 9.3%, 38.7%, 70.0%, and 44.7%; "Pretrained ACT" reached 7.3%, 20.0%, 0.7%, 6.0%, 29.3%, and 16.7%.
• VLA Backbone baseline (a π₀-architecture VLA trained from scratch on the same pre-training dataset) reached 7.3%, 38.0%, 2.0%, 14.7%, 56.7%, and 33.3% across the six tasks.
• Sample efficiency: ALOHA Unleashed with CLAMP pre-training "shows strong performance at very early training stages," and on the challenging Pen in Container task performance continued to improve with more training.
• Real-world results (maximum success rate across at most two checkpoints per method per task, reported as partial / success out of 10): ALOHA Unleashed with CLAMP scored 8/10 partial and 7/10 success on Open Drawers, 8/10 and 5/10 on Close Drawers, 8/10 and 8/10 on Mug on Plate, 9/10 and 8/10 on Plate on Rack, and 10/10 and 6/10 on Recycle Cans. ALOHA Unleashed without CLAMP scored 2/10 and 0/10, 8/10 and 0/10, 4/10 and 3/10, 8/10 and 8/10, and 9/10 and 5/10 respectively. The method outperformed the baseline on four of five real tasks and was comparable on Plate on Rack.
• Wrist views matter most for precision: removing virtual wrist views dropped performance by 2.6% on Can Opener in Caddy, 22.0% on Pen in Container, and 28.6% on Mug on Plate.
• Standard orthogonal five views underperform: replacing CLAMP's views with the "standard virtual five views" from prior work (3D-MVP / RVT) cost 9.3%, 9.3%, and 6.6% on the same three tasks.
• Encoder backbones: replacing the ViT with a DP3-style point-cloud encoder cost 15.3%, 15.3%, and 7.3%; removing STRING cost 2.0%, 3.3%, and 4.6%.
• Pre-training both encoders and policy is important: pre-training without the encoders cost 9.3%, 7.3%, and 4.6%; leaving the encoders unfrozen during fine-tuning cost 23.3%, 16.0%, and 3.3%; and ACT with full CLAMP pre-training cost 67.4%, 10.7%, and 59.4%.
• Text encoder contribution is mixed: pre-training without the text encoder changed results by +0.7%, −2.0%, and 0.0% across the three ablated tasks.
• Theoretical result: Lemma III.1 shows the STRING-modulated attention logit reduces to a standard query-key dot product when two tokens correspond to the same sub-point cloud, and Lemma III.2 shows the logit becomes a decreasing function of the distance between sub-point-cloud centers of mass under stated conditions, meaning STRING can implicitly down-weight attention between spatially distant regions across different views.

Methodology in Plain English

The authors build a scene point cloud from RGB-D images using known camera intrinsics and extrinsics, drop the RGB colors for better appearance generalization, crop to a 3D bounding box, and voxel-downsample. They then re-render the scene from five virtual viewpoints — overhead, front-left, back-right, and one view attached to each gripper wrist — producing four-channel images containing depth and 3D coordinates (D, X, Y, Z). The wrist cameras are defined from the left and right gripper poses with a small offset so objects are less occluded.

Three encoders are trained together: a ViT image encoder (94M parameters, 12 layers, 768 wide, B/16, 12 heads, with STRING positional encoding using the average global XYZ of points within each patch), a CLIP text encoder (277M parameters, 12 layers, 768 wide, 12 heads, 256 tokens), and a Transformer action encoder (18M parameters, 2 layers, 768 wide, 12 heads) that consumes a history of previous actions. Each outputs a 768-dimensional embedding. Training uses a SigLIP-style contrastive loss over three pairs — image-text, image-action, and text-action — averaged with equal weight, with labels set to 1 for matched pairs and −1 otherwise, and learnable bias and temperature initialized to b = −10 and t = exp(log 10) = 10.

In parallel, a 192M-parameter Diffusion Policy is pre-trained, consisting of a 4-layer, 768-wide Transformer encoder with 8 attention heads and a 7-layer, 768-wide Transformer decoder with 8 attention heads. It uses per-camera ResNet-50 backbones initialized from ImageNet, concatenates stage-four feature maps with unpooled frozen CLAMP embeddings and proprioceptive features, applies action chunking with a chunk size of 50, outputs 14 absolute target joint positions, and predicts noise of shape 50 × 14 for the next 50 actions.

After pre-training, the policy is fine-tuned on a limited set of task demonstrations, with all weights (including each ResNet backbone) initialized from the pre-trained checkpoint. The text encoder is used only during pre-training, so deployment does not rely on privileged simulator state. For real-world tasks, the policy is pre-trained on real data rather than simulated data to reduce the sim-to-real gap.

Why This Matters

• Research impact: The work argues that 3D perception and robot actions are valuable pre-training modalities, and provides both an architecture and theoretical analysis for cross-view token correlation without explicit view matching. It pushes back on the assumption that 2D image pre-training alone suffices for manipulation.
• Real-world applications:
  • Bimanual tabletop assembly and pick-and-place, such as grasping a specified tool and inserting it into a caddy or container.
  • Household and kitchen manipulation, such as placing a mug on a plate or a plate into a drying rack.
  • Articulated-object interaction, such as opening or closing drawers whose pose varies within the workspace.
  • Sorting and recycling, such as grasping soda cans and placing them into bins.
• Industry relevance: The results suggest that a single pre-training pass on large-scale simulated trajectories (553,592 successful episodes, 358,510,758 image-text-action triplets) can yield a policy that fine-tunes from as few as 63 real demonstrations (Recycle Cans) and still performs well, which is attractive for reducing the cost of real-robot data collection.

Future Directions

• Which pre-training modalities help most is still unclear — the ablation removing the text encoder showed mixed results (+0.7%, −2.0%, 0.0%), so the role of language grounding, object names, positions, and task-progress integers deserves more study.
• The Sim-to-real gap: the authors chose to pre-train real-world policies on real data instead of simulated data; whether pre-training on simulation can transfer to real hardware remains open.
• Scaling and generalization: pre-training tasks did not overlap with fine-tuning tasks, but the paper does not report how performance varies with the number of fine-tuning demonstrations, the number of pre-training tasks, or the diversity of pre-training trajectories.
• Beyond the tested hardware and view setups: all experiments use an ALOHA 2 bimanual robot with camera configurations placed 70 cm above the table; extending the approach to other embodiments, mobile robots, or different camera arrangements is an open question.

Target Audience

Robotics researchers and machine learning engineers working on manipulation policy learning, 3D perception, or multimodal pre-training — particularly those interested in contrastive representation learning, point-cloud-based encoders, diffusion policies, or bimanual robot systems. Readers with a machine learning background but little robotics exposure can follow the high-level framing, while the theoretical lemmas and ablation details will be most useful to specialists designing pre-training pipelines.

Authors’ abstract

Leveraging pre-trained 2D image representations in behavior cloning policies has achieved great success and has become a standard approach for robotic manipulation. However, such representations fail to capture the 3D spatial information about objects and scenes that is essential for precise manipulation. In this work, we introduce Contrastive Learning for 3D Multi-View Action-Conditioned Robotic Manipulation Pretraining (CLAMP), a novel 3D pre-training framework that utilizes point clouds and robot actions. From the merged point cloud computed from RGB-D images and camera extrinsics, we re-render multi-view four-channel image observations with depth and 3D coordinates, including dynamic wrist views, to provide clearer views of target objects for high-precision manipulation tasks. The pre-trained encoders learn to associate the 3D geometric and positional information of objects with robot action patterns via contrastive learning on large-scale simulated robot trajectories. During encoder pre-training, we pre-train a Diffusion Policy to initialize the policy weights for fine-tuning, which is essential for improving fine-tuning sample efficiency and performance. After pre-training, we fine-tune the policy on a limited amount of task demonstrations using the learned image and action representations. We demonstrate that this pre-training and fine-tuning design substantially improves learning efficiency and policy performance on unseen tasks. Furthermore, we show that CLAMP outperforms state-of-the-art baselines across six simulated tasks and five real-world tasks. The project website and videos can be found at https://clamp3d.github.io/CLAMP/.