3EED: Ground Everything Everywhere in 3D

AI summary

Overview

• Research area: 3D visual grounding and language-driven embodied perception (computer vision and robotics).
• Technical level: Intermediate. The paper centers on dataset construction, benchmark protocols, and a scale-adaptive baseline built from PointNet++ and a Transformer decoder, so some familiarity with point clouds and grounding metrics helps, but the core ideas are accessible.
• Scope in one sentence: 3EED is a large-scale, multi-platform, multi-modal benchmark of synchronized LiDAR and RGB data from vehicle, drone, and quadruped platforms for grounding natural-language expressions to 3D objects in outdoor scenes.

What This Paper Is About

Grounding free-form language to objects in 3D space lets robots interpret human instructions, but existing benchmarks are mostly indoor, limited to a single sensor platform, and small in scale. The authors build 3EED to capture outdoor grounding across three embodied platforms (Vehicle, Drone, Quadruped) with both LiDAR and RGB sensing, and they define benchmark protocols plus a baseline model to measure how well methods transfer across platforms.

Key Contributions

1. A new multi-platform, multi-modal 3D grounding benchmark. 3EED spans Vehicle, Drone, and Quadruped platforms with synchronized LiDAR and RGB, covering over 128,000 objects and 22,000 human-verified referring expressions. The authors state this is 10x larger than existing outdoor datasets.
2. A scalable annotation pipeline. 3D boxes are produced through multi-detector fusion, tracking, filtering, and manual verification, while referring expressions come from prompting a vision-language model with structured cues followed by rule-based rewriting and human refinement.
3. Platform-aware normalization and cross-modal alignment techniques. These standardize geometric and sensory data across platforms while preserving platform-specific characteristics, so spatial relations such as "above," "below," and "behind" are encoded in comparable coordinates.
4. A benchmark suite and baseline. The paper defines in-domain, cross-platform (zero-shot), multi-object, and multi-platform grounding settings, and presents a scale-adaptive, agent-invariant baseline with a Cross-Platform Alignment (CPA), Multi-Scale Sampling (MSS), and Scale-Aware Fusion (SAF) design.

Main Findings

• Cross-platform gaps are severe for existing methods. Trained on Vehicle data, BUTD-DETR reaches Acc@25 of 52.38 on the vehicle test split but drops to 1.54 on drone and 10.18 on quadruped. EDA shows a similar pattern, with 53.54 on vehicle, 3.33 on drone, and 11.40 on quadruped.
• The proposed baseline narrows those gaps. Trained on Vehicle, the authors' model reaches Acc@25/Acc@50 of 78.37/45.72 on vehicle, 18.16/2.78 on drone, and 36.04/20.59 on quadruped, improving over the strongest baseline by +25.99, +16.62, and +25.86 Acc@25 respectively.
• Training on all three platforms gives the most balanced results. Under Union training, the model reaches Acc@25 of 80.86 on vehicle, 53.45 on drone, and 53.31 on quadruped, with Acc@50 of 50.11, 9.75, and 24.08. The paper reports an average gain of +12.29 over the best method in the unified multi-platform setting.
• Multi-object grounding is notably hard. On the Vehicle platform multi-object setting, BUTD-DETR achieves average Acc@25 of 25.40 and mIoU of 47.88, while the authors' model reaches average Acc@25 of 32.32, Acc@50 of 29.89, and mIoU of 56.40.
• Every proposed module contributes. Removing CPA drops Vehicle Acc@25 from 80.86 to 71.76 and Drone Acc@25 from 53.45 to 51.84; removing MSS drops Vehicle Acc@25 to 75.65 and Drone Acc@25 to 46.85; removing SAF drops Quadruped Acc@25 from 53.31 to 51.98.
• Performance falls as scenes get more crowded. On the Quadruped platform, Acc@25 falls from 71.23 in scenes with 1-3 objects to 30.75 in scenes with 7-9 objects.
• Drone data is the sparsest and hardest. Drone scenes average 102.24 LiDAR points per object versus 462.89 for Vehicle and 112.17 for Quadruped, and average 8.05 objects per scene — the highest density — which the authors cite as the reason for the pronounced aerial performance gap.
• Dataset statistics differ sharply by platform. The average number of annotated objects per scene is 4.77 for Vehicle, 8.05 for Drone, and 5.83 for Quadruped. The paper reports the train/val split contains 2.7k/2.7k vehicle, 4.1k/2.9k drone, and 4.9k/2.9k quadruped scenes.

Methodology in Plain English

The team assembled data from two existing sources: Vehicle sequences come from the Waymo Open Dataset, and Drone and Quadruped sequences come from M3ED. For the non-vehicle platforms, they used a three-stage annotation pipeline. First, off-the-shelf 3D detectors trained on other driving datasets produce pseudo-labels for every frame. Second, kernel-density estimation merges detector votes, a 3D multi-object tracker enforces temporal coherence and fills missed detections, and Tokenize-Anything projects each box onto the RGB image to confirm its class. Third, human annotators polish flagged boxes, with manual effort limited to roughly 100 seconds per frame.

For language, each 3D box is projected onto its RGB view alongside a knowledge base with five template slots (category, status, absolute location, egocentric position, relation) and fed to a vision-language model. Platform-specific terms are normalized by rewriting rules so wording stays consistent across platforms, and annotators then check semantic correctness, spatial fidelity, ambiguity, and platform consistency, discarding unsatisfactory cases.

The baseline model encodes LiDAR with a scale-adaptive PointNet++ backbone (raw scans uniformly down-sampled to 16,384 points, producing 1,024 visual tokens, with the top 256 tokens entering a six-layer Transformer decoder), encodes language with a frozen RoBERTa, and predicts each referenced 3D box in one shot. Training blends box-regression, token-alignment, and contrastive multimodal losses, with Hungarian matching for multi-object cases. Three modules target the outdoor and cross-platform challenges: CPA rotates each scan so gravity aligns with the global z-axis (with an extra altitude-normalizing height offset for drones), MSS queries PointNet++ neighborhoods at multiple radii from 0.6 m to 4.8 m, and SAF uses a lightweight MLP to produce per-point weights that fuse the scales into a single embedding. Training runs for 100 epochs on two NVIDIA RTX 4090 GPUs, with a learning rate of 1×10⁻³ for the visual encoder and 1×10⁻⁴ for other layers.

Why This Matters

The paper argues that real-world embodied agents need grounding that works outdoors, at long range, under sparse sensing, and across different embodiments — not just in small indoor rooms with household furniture. By releasing data from vehicle, drone, and quadruped platforms in one benchmark, it provides a common yardstick for measuring whether a grounding model actually generalizes across viewpoints and sensor heights.

Real-world applications include:

• Autonomous driving and mobile robotics: localizing language-referred vehicles and pedestrians (the two safety-critical classes annotated) in urban scenes.
• Aerial inspection and delivery: grounding instructions from drone viewpoints, where top-down geometry and sparse returns make reasoning difficult.
• Legged robots in outdoor environments: interpreting references from a ground-level, varying-pitch perspective.
• Human-robot interaction and situational awareness: following high-level instructions for navigation, interaction, and scene understanding.

For industry, the released dataset, toolkit, and baseline model offer a ready-made testbed for teams building language-conditioned perception on robot platforms, and the paper's finding that zero-shot cross-platform transfer is weak signals that deployment across differing hardware remains an open engineering problem.

Future Directions

• Closing the drone gap: drone Acc@50 stays low (for example 9.75 under Union training), which the authors attribute to sparse returns averaging 102.24 points per object and high scene density; better handling of range-dependent sparsity is a clear next step.
• Improving multi-object and crowded-scene reasoning: accuracy degrades as object count rises, and joint correctness across all referred objects is the strict metric used, leaving room for methods that reason about object sets rather than isolated targets.
• Stronger cross-platform generalization: zero-shot transfer remains far behind in-domain performance, so architectures that go beyond the gravity-alignment and scale-adaptive design presented here may be needed.
• Broadening the benchmark: the dataset focuses on the two classes Vehicle and Pedestrian and is built on Waymo and M3ED sequences, raising the question of whether the protocols extend to more categories, more platforms, or new sensing configurations.

Target Audience

Researchers and engineers working on 3D visual grounding, embodied AI, autonomous driving, and outdoor robot perception who need a multi-platform benchmark and baseline for language-driven 3D localization. It is also relevant to practitioners evaluating how well perception models transfer across different robot embodiments and sensor setups.

Authors’ abstract

Visual grounding in 3D is the key for embodied agents to localize language-referred objects in open-world environments. However, existing benchmarks are limited to indoor focus, single-platform constraints, and small scale. We introduce 3EED, a multi-platform, multi-modal 3D grounding benchmark featuring RGB and LiDAR data from vehicle, drone, and quadruped platforms. We provide over 128,000 objects and 22,000 validated referring expressions across diverse outdoor scenes -- 10x larger than existing datasets. We develop a scalable annotation pipeline combining vision-language model prompting with human verification to ensure high-quality spatial grounding. To support cross-platform learning, we propose platform-aware normalization and cross-modal alignment techniques, and establish benchmark protocols for in-domain and cross-platform evaluations. Our findings reveal significant performance gaps, highlighting the challenges and opportunities of generalizable 3D grounding. The 3EED dataset and benchmark toolkit are released to advance future research in language-driven 3D embodied perception.