Data Pyramid for Embodied Manipulation

AI summary

arXiv: 2607.24744v2 [cs.RO] · Published: 2026-07-27 (v2 dated 08 Aug 2026) · License: CC BY 4.0

Overview

• Research area: Robotics and embodied AI, specifically the data ecosystem used to train embodied foundation models for manipulation.
• Technical level: Advanced. The paper assumes familiarity with robot learning, teleoperation, vision-language-action models, and world models, though its organizing taxonomy is presented in accessible terms.
• Scope (one sentence): A large-scale survey that organizes embodied manipulation data into a five-layer "pyramid," characterizes each layer by six properties, analyzes how recent embodied foundation models mix these sources during pretraining, and lays out open challenges for embodied data.

What This Paper Is About

Multimodal foundation models learned broad capabilities from internet-scale visual and linguistic data, but embodied agents cannot take that shortcut because they need data that pairs observations with physical states and actions. This survey asks two questions: how heterogeneous embodied data sources can be collected, organized, compared, and integrated despite differing in scalability, robot alignment, physical fidelity, and transferability; and how embodied foundation models actually select, combine, and use them. The goal is to give the field a category-level, data-centric organization of the embodied data ecosystem, rather than the model-centric or architecture-centric views that existing surveys emphasize.

Key Contributions

1. A data-pyramid taxonomy guided by scalability and robot alignment. The paper organizes embodied AI data into five complementary categories ordered from apex to base: real-robot data, UMI-style data, egocentric and exocentric data, simulation data, and general data. Each category is characterized along six dimensions: scalability, robot alignment, quality, diversity, reusability, and physical fidelity. The authors also release and maintain an open-source repository curating representative datasets and related resources under this taxonomy.
2. A data-centric analysis of embodied foundation models. The survey examines how heterogeneous sources support embodied vision-language models, vision-language-action (VLA) models, and world-action models, connecting dataset properties (semantic coverage, temporal structure, action supervision, embodiment alignment, diversity, physical fidelity) to capabilities such as perception, reasoning, planning, action generation, and world prediction.
3. A framework for cross-source alignment. The paper analyzes how models accommodate heterogeneous datasets through action-space alignment (reconciling action dimensionality, control interfaces, and action semantics across embodiments) and geometric alignment (mapping observations and demonstrations from different viewpoints, coordinate systems, and embodiments into robot-compatible representations).
4. A discussion of open challenges and future directions, framed around three questions: what to collect, how to collect it, and how to use it.

Main Findings

• Two organizing axes are in tension. Scalability describes how efficiently a source can be expanded given hardware dependence, human labor, environment resets, safety supervision, and marginal generation cost. Robot alignment describes how directly its observations, representations, and supervision support learning and execution on physical robots. Data closely aligned with real-robot execution is expensive and hard to scale; highly scalable data provides only indirect supervision for physical interaction.

• Descending the pyramid is a systematic trade. Real-robot data sits at the apex because recorded actions are directly executable on the platform that produced them. UMI-style rigs drop the robot from the collection loop while retaining end-effector supervision. Human egocentric/exocentric recordings drop the gripper but keep real physics and everyday diversity. Simulation restores executable actions and privileged labels but approximates the physics itself. General data abandons robot grounding for web-scale semantic and reasoning coverage, saying nothing about contact or consequence.

• The layer ordering is a synthesis, not a monotonic progression. The authors state explicitly that the ordering reflects an overall synthesis of the six dimensions rather than a strictly monotonic progression along every individual property.

• Real-robot data has shifted from task-specific collections to general data ecosystems. Early efforts used scripted controllers for large numbers of trials on a few behaviors (Pinto and Gupta: 50K trajectories / 700 hours, 1 task; QT-Opt: 580k / 800 hours, 1 task). Later work expanded task breadth and scale, and cross-institution aggregation. Reported examples include MT-Opt at approximately 800K episodes across 12 tasks, RT-1 at 130K trajectories covering 700+ tasks, AgiBot World Beta at 1M trajectories and 2976.4 hours across 217 tasks, and RoboMIND 2.0 at 310K trajectories across 739 tasks and over 1,000 hours of execution. Open X-Embodiment aggregates more than two million trajectories spanning 22 embodiment types and 527 tasks.

• Embodiment coverage has broadened substantially. Collection moved from fixed-base single-arm manipulators with parallel grippers toward dual-arm systems, mobile manipulators, humanoid platforms, dexterous hands, and heterogeneous robot fleets, enabling study of whole-body coordination, contact-rich manipulation, and cross-embodiment learning.

• Sensing modalities have become richer and more heterogeneous. Beyond the standard vision–proprioception–action interface, datasets increasingly add tactile sensing (contact location, pressure distribution, incipient slip, grasp stability), force and torque at the wrist, gripper, or joints, audio cues about collisions and friction, and IMU, LiDAR, and odometry for whole-body platforms.

• Action and state representations differ across embodiments. Fixed-base manipulators typically use joint positions (qpos) or end-effector poses with gripper states, with actions in joint space or Cartesian end-effector space. Dexterous hands use finger joint positions, sometimes with fingertip poses to align with human data. Mobile and humanoid platforms add base velocity, torso configuration, and whole-body joint states. Proprioceptive states and actions are commonly normalized using MinMax, Q01–Q99, or MeanStd schemes.

• Collection paradigms fall into three families. Scripted collection (rule-based execution, trajectory playback, autonomous policy rollouts), teleoperation (kinesthetic teaching, leader-follower, integrated leader-follower, device-mediated, vision-based, wearable/motion-capture retargeting), and human-in-the-loop enhancement (intervention-based corrective data collection, following the dataset-aggregation principle of DAgger and extended by methods such as Intervention Weighted Regression, ThriftyDAgger, Sirius, Fleet-DAgger, DexCap, IntervenGen, and CR-DAgger).

• Teleoperation design varies along interface and feedback. Feedback channels include visual feedback, immersive stereoscopic and head-tracked viewpoints, force feedback through bilateral leaders, haptic controllers or exoskeletons, and tactile feedback via vibration, pressure, or fingertip stimulation. Integrated leader-follower teleoperation (for example the ARX AC-One) eliminates a dedicated leader device but creates a visual distribution shift when the operator's hand appears in training images but is absent at deployment.

• Model data recipes have shifted toward heterogeneous mixtures. Analysis of representative embodied foundation models shows a transition from early recipes dominated by real-robot trajectories toward co-training with real-robot, egocentric, simulation, general, and, less frequently, UMI-style data, and an architectural expansion from action-centric VLA systems toward world-model-augmented and unified world-action models. Prior work such as Motus and GR00T introduced hierarchical or pyramid-like views of training data, but those formulations were designed around a particular model's recipe and offered limited category-level analysis.

• The paper reports six open challenges in its abstract: building large-scale tactile datasets, collecting failure and recovery data, developing scalable data-collection pipelines, aligning actions across embodiments, leveraging egocentric data for dexterous manipulation, and designing principled data recipes for robot learning.

• Coverage note: the provided paper content is truncated in Section 2.4, so the detailed treatments of UMI-style data, egocentric/exocentric data, simulation data, general data, the model analyses, and the challenges section are summarized here only as they appear in the abstract and introduction. Specific dataset statistics, tables, and figures for those layers are not reported in the available excerpt.

Methodology in Plain English

This is a survey, not an experimental study, so its "method" is an organizing framework plus a literature synthesis. The authors first define two primary axes — scalability and robot alignment — and four additional category-level dimensions (quality, diversity, reusability, physical fidelity). They then sort the embodied data ecosystem into five layers along those axes, from real-robot data at the apex down to general web-scale data at the base. For each layer they review datasets, collection pipelines, embodiments, sensing configurations, supervision signals, scale, diversity, and transferability, and they chart how dataset scale has grown over time (demonstrations for simulation, UMI, and robot datasets; collected hours for ego datasets; question-answer pairs for general datasets). Separately, they examine representative embodied foundation models through the lens of their pretraining data recipes, grouping them into embodied brain models, VLA models, and world-action models, and analyzing cross-embodiment differences in terms of action-space alignment and geometric alignment. The paper closes with a structured discussion of challenges organized around what to collect, how to collect it, and how to use it. The authors also maintain a public repository that curates datasets and resources under the taxonomy.

Why This Matters

Impact on research. The paper reframes embodied AI as a data problem rather than only an architecture problem, giving researchers a shared vocabulary for comparing datasets that differ wildly in scale, embodiment, sensing, and supervision. By connecting data composition to capabilities in perception, reasoning, planning, action generation, and world prediction, it offers a basis for deciding which data layers a new model should draw on, and for reasoning about cross-embodiment transfer and sim-to-real gaps.

Real-world applications.

• General-purpose robot manipulation in unstructured settings, where policies must generalize across tasks, objects, scenes, and instructions rather than replaying scripted motions.
• Data-efficient industrial and warehouse deployment, where autonomous rollouts, trajectory playback, and human-in-the-loop correction can reduce the need for exhaustive manual teleoperation.
• Dexterous and contact-rich manipulation such as assembly and insertion, where tactile, force, and torque signals — and failure/recovery trajectories — are essential rather than optional.
• Humanoid and mobile manipulation, where whole-body state, locomotion, and navigation signals add dimensions that most existing datasets do not cover.

Industry relevance. The survey documents million-scale trajectories and thousands of hours of interaction from efforts spanning many institutions and robot platforms, and notes datasets collected across dozens of tasks and embodiment types. For companies building robot foundation models, the pyramid provides a practical framework for allocating collection budget across expensive real-robot teleoperation and cheaper scalable sources, and the accompanying repository serves as a catalogue of candidate training corpora. The emphasis on failure data, tactile data, and cross-embodiment action alignment maps directly onto practical bottlenecks in deploying learned policies.

Future Directions

1. What to collect. Build large-scale tactile datasets capturing contact states, forces, material properties, and fine-grained interaction dynamics, and collect failure and recovery trajectories that remain scarce but are needed for robust recovery behavior.
2. How to collect it. Develop scalable pipelines that reduce dependence on costly manual teleoperation, broaden task and environmental coverage, and adaptively prioritize underrepresented or highly informative interactions.
3. How to use heterogeneous data. Align action representations across embodiments, determine effective mixing ratios for data drawn from different pyramid layers, and transfer human interaction experience to robot manipulation, especially from egocentric data to dexterous hands.
4. Systematic assessment and evaluation. Establish methods for quality and diversity assessment, multimodal standardization, and principled evaluation of integrated heterogeneous data sources, so that data recipes can be compared rather than described. The paper notes that increasing data volume provides limited benefit when trajectories are repetitive, annotations are inaccurate, signals are poorly synchronized, or collection conditions are systematically biased.

Target Audience

Robot learning and embodied AI researchers, especially those designing pretraining data mixtures for manipulation policies; dataset builders deciding what to collect and how; engineers at robotics and foundation-model companies evaluating collection strategies and cross-embodiment transfer; and graduate students or newcomers who want a single structured map of the embodied data landscape and the characteristics, costs, and trade-offs of each data source. Readers looking for a specific new algorithm or benchmark result will find a taxonomy and literature synthesis instead.

Authors’ abstract

Multimodal foundation models learned to see and to speak by consuming the whole internet. Embodied agents admit no such shortcut, since they require data that couple observations with physical states and actions. These signals can be provided, to varying degrees, by multiple data sources. In this work, we organize the embodied data ecosystem as a "pyramid" spanning five complementary sources: real-robot data, UMI-style data, egocentric and exocentric data, simulation data, and general vision-language data. We organize the pyramid around the tension between scalability and robot alignment, and further characterize each source in terms of data quality, diversity, reusability, and physical fidelity. We then analyze recent embodied foundation models through the lens of their data recipes, examining how different sources are selected, aligned, and mixed during pretraining. For embodied brain models, vision-language-action models, and world-action models alike, we relate data composition to capabilities in perception, reasoning, planning, action generation, and world prediction. We close by discussing six open challenges: building large-scale tactile datasets, collecting failure and recovery data, developing scalable data-collection pipelines, aligning actions across embodiments, leveraging egocentric data for dexterous manipulation, and designing principled data recipes for robot learning. We hope this work paves the foundation for the design of next-generation embodied systems.