GauDP: Reinventing Multi-Agent Collaboration through Gaussian-Image Synergy in Diffusion Policies

AI summary

Overview

Research area: Robotics — multi-agent embodied collaboration, imitation learning, and 3D scene representation (3D Gaussian Splatting combined with diffusion policies).

Technical level: Advanced. Familiarity with diffusion policies, 3D Gaussian Splatting, and behavior cloning helps, though the core idea is describable in plain terms.

Scope: The paper proposes GauDP, a method that reconstructs a shared 3D Gaussian field from multiple agents' RGB views and dispatches task-relevant Gaussian features back to each agent, then trains a diffusion policy on the fused image-plus-Gaussian representation for multi-arm manipulation.

What This Paper Is About

Multi-arm robots working on the same task face a trade-off: each arm needs fine, local visual detail for precise grasping and placement, but the team also needs a globally consistent view of the scene to avoid collisions and mis-sequenced actions (for example, one arm trying to put food in a pot before the other has lifted the lid). Existing approaches pick one side of this trade-off — either concatenating all local views into a shared policy, or feeding a single global observation — and each choice degrades either coordination or control precision. GauDP's goal is to build one unified representation that supplies both global consistency and per-agent local detail, using only RGB images and no extra sensing modalities.

Key Contributions

1. A unified image–Gaussian framework for multi-agent collaboration (GauDP). A globally consistent 3D Gaussian field is reconstructed from decentralized RGB observations, and 3D Gaussian attributes are dynamically redistributed back to each agent's local perspective, letting agents query task-critical features while keeping their individual viewpoints.

2. A dynamic global-context allocation and pixel-level fusion mechanism. Rather than giving every agent the whole global context, each agent receives only the subset of Gaussians that originated from its own view — Gaussians that have already absorbed information from other views through cross-view attention. These are turned back into a 2D grid, concatenated with the agent's local image features, and combined by a lightweight convolutional fusion module.

3. Strong results on the RoboFactory benchmark using RGB only. GauDP achieves a 19.67% average success rate across six multi-arm tasks, beating image-based baselines and approaching or exceeding point-cloud-driven methods, with no architectural changes needed as the number of agents grows.

4. Real-robot validation and efficiency analysis. The method is tested on three real multi-arm tasks (Card Box Stacking, Card Box Handover, Grab Roller) and compared against Diffusion Policy on training time, inference speed, and parameter count.

Main Findings

• Reconstruction quality improves substantially after fine-tuning. On 3D Gaussian reconstruction from only two reference viewpoints, the fine-tuned model reaches PSNR 23.424, SSIM 0.779, and LPIPS 0.148, versus PSNR 17.918, SSIM 0.580, and LPIPS 0.492 for the pretrained Noposplat baseline. The paper reports that the pretrained model produces blurry, distorted results and noticeably wrong robot-arm positions, while the fine-tuned model keeps consistent fidelity in both reference and novel views.

• Highest average success rate among compared methods. GauDP averages 19.67% across the six RoboFactory tasks, ahead of DP3(XYZ) at 10.67%, DP3(XYZ+RGB) at 14.33%, 3D Dense Policy at 8.83%, DP at 5.00%, and 2D Dense Policy at 2.33%.

• Large gains on coordinated tasks but not on every task. GauDP reaches 72% on Lift Barrier (2 arms), 26% on Align Camera (3 arms), 15% on Place Food (2 arms), 2% on Two Robots Stack Cube, 3% on Take Photo (4 arms), and 0% on Three Robots Stack Cube. The point-cloud baseline DP3(XYZ+RGB) is higher on Take Photo (11% vs. 3%) and on Three Robots Stack Cube it is also 0%.

• Comparison to point-cloud methods. The paper reports DP3 achieves 30% on Lift Barrier and 21% on Place Food but only 1% on Stack Cubes with 2 arms. GauDP, using only 2D RGB input, surpasses 3D-based counterparts on global-coordination tasks (Align Camera: 26% vs. 18%) and matches them on fine-grained manipulation (Stack Cube: 2% vs. 1%).

• Modest efficiency cost. On the Lift Barrier task, GauDP needs 6.5 GPU hours of training and runs at 1.28 FPS inference, versus DP at 4.8 GPU hours and 1.49 FPS, and DP3 at 2.5 GPU hours and 1.57 FPS. The table caption states training was measured on an A100 GPU and inference on an NVIDIA RTX 5090 GPU, while the experiment-setting section states experiments were run on a single NVIDIA A800 GPU.

• Real-robot results favor GauDP over DP. On Card Box Stacking: place success 23/30 and stack success 17/23, overall 17/30 (DP: 19/30, 11/19, 11/30). On Card Box Handover: place success 24/30, handover success 19/24, overall 19/30 (DP: 22/30, 14/22, 14/30). On Grab Roller: 27/30 (DP: 22/30).

• Every ablated component matters. Removing pixel-level fusion (coarse feature-level concatenation instead) collapses the average to 1.17%. Removing the Gaussian representation leaves only the image input and drops the average to 11.17%. Removing the image input and keeping Gaussians gives 11.17% — wait, that is the "w/o Image" row at 11.17%; removing the Gaussian gives 5.00%. Using a unified world-coordinate system for Gaussians instead of the original camera-coordinate parameterization gives 10.83%.

  Correction for clarity: the four ablation rows are w/ unify coor. (10.83%), w/o prefuse (1.17%), w/o Image (11.17%), and w/o Gaussian (5.00%), all below the full model's 19.67%.

• A shared global policy beats per-arm local policies. Local GauDP averages 5.33% (3% Lift Barrier, 12% Place Food, 0% Stack Cube 2-arm, 15% Align Camera, 0% Stack Cube 3-arm, 2% Take Photo), against 19.67% for the global version.

• The gain is not just extra parameters. GauDP-full has 750.505M parameters with 2 agents, 784.140M with 3, and 817.686M with 4; GauDP-policy alone is 129.959M, 163.594M, and 197.140M respectively. A scaled-up LargeDP reaches 721.286M, 956.335M, and 1.191G parameters and averages 17.50% (60% Lift Barrier, 12% Place Food, 4% Stack Cube, 29% Align Camera, 0% Stack Cube 3-arm, 0% Take Photo), still below GauDP's 19.67%. Most GauDP parameters are frozen during policy learning, whereas all LargeDP parameters are learnable.

Methodology in Plain English

The system takes synchronized RGB images from each robot arm's camera. A feed-forward network called Noposplat — which can reconstruct 3D Gaussians from sparse, unposed views — is fine-tuned on the same multi-arm manipulation data used to train the policy. Each image is encoded by a shared-weight ViT encoder, the per-view features are fused by a cross-view ViT decoder using cross-attention in every transformer block, and a prediction head outputs a set of 3D Gaussians per pixel. During fine-tuning only, ground-truth depth is rendered and supervised alongside RGB, with the total loss combining an RGB term and a depth term scaled by a balancing weight.

The key step is allocation. Because each predicted Gaussian stays aligned with the image pixel it came from, the system can send each agent only the Gaussians derived from its own view. Those Gaussians have already absorbed information from the other views during reconstruction, so they act as a distilled global summary without flooding the agent with irrelevant scene content. These selected Gaussians are reshaped into a 2D grid matching the image, concatenated with the agent's local image features, and merged by a lightweight convolutional module for pixel-level alignment. A single global diffusion policy then predicts the future actions for all agents jointly, using DDPM with 100 denoising steps, an action prediction horizon of 8, 3 observation steps, and 6 action execution steps. At deployment the system is pose-free and depth-free, relying only on multi-view RGB.

Why This Matters

Impact on research. The paper argues the benefit comes from how visual representations are designed rather than from the input modality itself: an RGB-only pipeline can match or beat point-cloud-based policies on coordination-heavy tasks by fusing local detail with global spatial context. It also targets a gap the authors identify — data-driven policy learning for multi-agent robotic manipulation remains largely underexplored compared with single-agent settings.

Real-world applications (as listed or demonstrated in the paper):

• Industrial assembly
• Surgical robotics
• Assistive household tasks
• Concrete manipulation scenarios demonstrated in the real-robot experiments: card box stacking, card box handover, and grabbing a roller

Industry relevance. Multi-arm coordination is central to factory automation platforms, logistics and packing, and service robotics, where adding an extra arm should not require redesigning the policy architecture. GauDP's claim that the design scales to more agents without architectural changes, plus its use of only RGB cameras rather than depth sensors or centralized point clouds, speaks directly to cost and deployability on existing hardware.

Future Directions

• Gaussian representations for vision-language-action models. The authors propose designing Gaussian representations that are better suited as inputs to a VLA model to strengthen multi-agent collaboration.
• Dynamic scenes and world models. They want to use Gaussians to represent dynamic scenes so they can contribute to world models for multi-agent environments.
• Scaling beyond the reported agent counts. The paper shows scaling from 2 to 4 arms, but several 3-arm and 4-arm tasks still show 0% or 3% success, so how far the approach scales and where coordination breaks down remains open.
• Reducing reliance on depth during fine-tuning. Depth maps and camera poses are used only in the fine-tuning stage, which raises the question of how much performance depends on that geometric supervision when it is unavailable.

Target Audience

Robotics and embodied-AI researchers working on multi-agent manipulation, imitation learning, and diffusion policies; 3D vision researchers interested in Gaussian Splatting as a robot perception representation rather than a pure rendering tool; and engineers evaluating whether an RGB-only, scalable coordination stack can replace point-cloud pipelines in multi-arm industrial or service deployments.

Authors’ abstract

Recently, effective coordination in embodied multi-agent systems has remained a fundamental challenge, particularly in scenarios where agents must balance individual perspectives with global environmental awareness. Existing approaches often struggle to balance fine-grained local control with comprehensive scene understanding, resulting in limited scalability and compromised collaboration quality. In this paper, we present GauDP, a novel Gaussian-image synergistic representation that facilitates scalable, perception-aware imitation learning in multi-agent collaborative systems. Specifically, GauDP constructs a globally consistent 3D Gaussian field from decentralized RGB observations, then dynamically redistributes 3D Gaussian attributes to each agent's local perspective. This enables all agents to adaptively query task-critical features from the shared scene representation while maintaining their individual viewpoints. This design facilitates both fine-grained control and globally coherent behavior without requiring additional sensing modalities (e.g., 3D point cloud). We evaluate GauDP on the RoboFactory benchmark, which includes diverse multi-arm manipulation tasks. Our method achieves superior performance over existing image-based methods and approaches the effectiveness of point-cloud-driven methods, while maintaining strong scalability as the number of agents increases.