Learning a Thousand Tasks in a Day

AI summary

Overview

Research area: Robot manipulation, imitation learning, and few-shot learning for robotics.

Technical level: Intermediate. The paper assumes familiarity with behavioural cloning, point-cloud perception, and imitation learning benchmarks, but its core ideas — breaking a trajectory into two phases and retrieving demonstrations instead of training a policy — are explained conceptually.

Scope: A systematic empirical study of trajectory decomposition and retrieval-based generalisation for multi-task robot manipulation, culminating in a method (MT3) that learns 1,000 everyday tasks from a single demonstration each.

What This Paper Is About

Most imitation learning systems for robot manipulation need hundreds or thousands of demonstrations per task — for example, the paper reports that BC-Z required roughly 26K demonstrations for 100 tasks, RT-1 needed about 130K demonstrations across 744 tasks, and MT-ACT collected 7.5K demonstrations for 38 tasks, averaging 175–250 demonstrations per task. This makes scaling to thousands of tasks impractical.

The authors ask whether two priors — decomposing manipulation trajectories into an alignment phase and an interaction phase, and using retrieval-based generalisation instead of behavioural cloning — can deliver strong performance in the few-demonstrations-per-task regime (fewer than 10 demonstrations per task). They then test whether the resulting method can scale to an extremely large number of tasks from minimal data.

Key Contributions

1. A systematic evaluation of multi-task imitation learning in the few-demonstrations-per-task regime, comparing four decomposition-based methods (BC-BC, BC-Ret, Ret-BC, Ret-Ret) against a monolithic behavioural cloning baseline (MT-ACT+), across 3,450 real-world rollouts and 70 objects.

2. Multi-Task Trajectory Transfer (MT3), a fully retrieval-based decomposition method that uses retrieval for both alignment and interaction, learns tasks from as little as a single demonstration each, and generalises to novel object instances. The paper describes MT3 as an extension of Trajectory Transfer to the multi-task setting.

3. A large-scale validation: teaching a robot 1,000 distinct everyday manipulation tasks involving interactions with over 400 objects, from single demonstrations, collected in under 24 hours of demonstrator time on a single robot (17 hours of demonstration collection are reported in the methods).

4. A failure-mode analysis from 2,200 additional real-world rollouts, characterising where retrieval-based decomposition succeeds and where it breaks down.

Main Findings

• Decomposition beats a single monolithic policy when data is scarce. In the under-10-demonstrations-per-task regime, decomposition achieved an order of magnitude improvement in data efficiency over single-phase learning. Decomposition-based methods consistently outperformed the monolithic MT-ACT+ baseline across all tested data regimes, with the largest gap when demonstrations per task were most limited.

• Retrieval beats behavioural cloning for both phases. Averaged across interaction methods, methods using retrieval for alignment had higher success rates than those using BC for alignment; likewise for interaction. MT3 (fully retrieval-based) consistently performed best across all considered data regimes.

• Three demonstrations with MT3 beat 50 demonstrations with other methods. For both seen and unseen tasks, MT3 with just three demonstrations per task outperformed all other methods, even when those methods were given 50 demonstrations per task.

• Monolithic BC scales better as data grows. Decomposition-based methods gained rapidly with 1–10 demonstrations per task but plateaued near 50, whereas the monolithic baseline accelerated in the 10–50 range, narrowing the gap. The authors expect MT-ACT+ to eventually overtake decomposition given sufficient data, though the crossover point depends on the number of tasks and their similarity.

• Retrieval has a geometry-versus-pose trade-off under increasing object diversity. When a fixed budget of 150 demonstrations was spread across 10 to 50 tasks, unseen-task performance improved (more object instances gave closer matches) but seen-task performance degraded, because retrieval had to balance selecting demonstrations with similar object pose against similar object geometry.

• Diversity can substitute for data. Learning 50 tasks across 10 micro skills with 150 demonstrations achieved comparable performance to learning 12 tasks across four micro skills with 600 demonstrations. MT-ACT+ benefited from this diversity; BC-BC did not, despite identical BC implementations, which the authors attribute to decomposition disrupting shared structural patterns.

• MT3 scaled to 1,000 tasks. The evaluation spanned 31 macro skills, 534 micro skills, and 402 different objects, plus 100 unseen tasks. MT3 achieved a 78.25% average success rate on seen tasks and 68% on unseen tasks, under challenging conditions including 5–20 distractor objects, varied lighting, and randomised placement with up to ±45° rotation.

• Failure modes on seen tasks were attributed as follows: retrieval 22.3%, pose estimation 23.9%, segmentation 19.5%, with the remaining 29% of failures predominantly from tasks with grasped objects (20.2%). Retrieval failures occurred most with partially occluded objects or with small geometric variations that global matching could not identify.

• High-tolerance tasks worked well. For tasks permitting small deviations in approach angle and contact position — wiping, stirring, placing, grasping — success rates consistently exceeded 80%.

• The method struggled with small asymmetric features, high precision, and deformables. A kettle's spout being small relative to the body sometimes caused a 180° orientation error; inserting a plug into a socket or hanging small keys requires millimetre-level precision that open-loop interaction cannot compensate for; and deformable objects failed because stiffness and elasticity are not inferable from vision alone.

• Retrieval cannot interpolate. When a required trajectory lies between two demonstrated trajectories, retrieval selects one of the demonstrated ones rather than generating an intermediate solution.

Methodology in Plain English

The robot is a Sawyer arm with a 2F-85 Robotiq gripper and a single RealSense D415 RGB-D camera on its head. Every task involves a single interaction between the end-effector (or a grasped object) and a target object, and the authors assume the grasped object sits in the same pose in the gripper during demonstration and deployment.

The robot receives two inputs: a segmented point cloud of the target object and a language description of the task. Each demonstration is recorded at 30 Hz as RGB-D images plus end-effector state — a 6D pose in SE(3) and a binary gripper state — paired with a language description.

Decomposition. Instead of one policy handling an entire trajectory, the trajectory is split into an alignment phase (moving the end-effector to a suitable pose, where only the final position matters) and an interaction phase (the actual manipulation, where the precise trajectory matters). Each phase gets its own specialised policy, and each policy can be either BC-based or retrieval-based, giving four combinations: BC-BC, BC-Ret, Ret-BC, and Ret-Ret (MT3).

Behavioural cloning. The BC implementation adapts the MT-ACT transformer architecture with variational inference to accept point clouds and language descriptions, trained separately on alignment and interaction demonstrations.

Retrieval. All demonstrations are stored in memory. Before execution, a single retrieval step uses language processing of task descriptions combined with geometry similarity in a learned latent space, extracted from an RGB-D image of the scene. The alignment policy then uses pose estimation to map the demonstrated alignment pose onto the test scene and reaches it via motion planning. The interaction policy replays the demonstrated end-effector velocities in the end-effector frame. For novel objects, retrieval picks the closest demonstration object and behaves as if the novel object were identical to it.

Two controlled experiments. The first fixed the task set (four micro skills, 12 seen and 8 unseen tasks) and scaled demonstrations per task from 1 to 50. The second fixed the total at 150 demonstrations and spread them across 10, 30, and 50 tasks (15, 5, and 3 demonstrations each). Evaluations randomised object position within an 80 × 45 cm task space and orientation within ±180° of the demonstration pose around the vertical axis, with three evaluations per task.

The authors define a macro skill as a broad manipulation primitive ("open", "insert", "fold"), a micro skill as a macro skill specialised for an object category requiring a distinct motion profile, and a task as a micro skill executed on a specific object instance.

Why This Matters

Impact on research. The paper challenges the assumption that complex neural policies are necessary for large-scale robot learning. It provides quantitative evidence that analytical inductive biases (pose estimation, motion planning, trajectory replay) can outperform learned policies when per-task data is scarce, and it maps out where that advantage disappears as data grows — a trade-off that reframes how the field should think about data budgets versus model capacity.

Real-world applications:

• Domestic service robots that must acquire hundreds of household tasks (wiping, stirring, placing, grasping) with minimal teleoperation effort.
• Warehouse and logistics picking and placing, where new object categories appear frequently and retraining is impractical.
• Assembly and insertion tasks in manufacturing, where the paper identifies high-precision alignment as a key remaining limitation.
• Rapid task-library expansion in lab automation or food preparation, where appending a new demonstration to a dataset replaces fine-tuning.

Industry relevance. The paper reports collecting 1,000 demonstrations in under 24 hours of demonstrator time, and notes that adding new tasks requires only appending demonstrations — no fine-tuning or retraining. That directly addresses the dominant cost of deploying manipulation systems: human teleoperation time and repeated model retraining cycles.

Future Directions

• Closed-loop and reactive interaction. Open-loop replay cannot detect errors or adjust mid-execution, which the authors identify as a fundamental limitation for deformable objects, contact-based reorientation, and multi-step pushing.
• Tactile or other non-visual sensing. Vision alone cannot infer the stiffness and elasticity of deformable objects, which the paper shows causes systematic failures despite visual similarity.
• Better pose estimation for small, asymmetric features. Small features that break object symmetry (for example a kettle spout) are frequently misregistered, causing large orientation errors and task failure.
• Retrieval that can interpolate. Because retrieval selects a single stored demonstration, it cannot synthesise a trajectory between two demonstrations — a limitation the authors highlight for object instances requiring trajectories that were never demonstrated.
• Extensions to bimanual and multi-stage tasks. The authors note that the alignment-interaction decomposition has already been demonstrated for bimanual tasks elsewhere, and suggest single-interaction primitives could be chained using high-level planning and skill chaining.

Target Audience

Robotics and imitation learning researchers, particularly those working on few-shot manipulation, retrieval-based methods, or data-efficient multi-task learning. Also relevant for robot engineers and industry practitioners evaluating whether to invest in large-scale teleoperation datasets versus decomposition-and-retrieval pipelines, and for graduate students looking for a rigorous empirical comparison of monolithic versus decomposed policy architectures.

Authors’ abstract

Humans are remarkably efficient at learning tasks from demonstrations, but today's imitation learning methods for robot manipulation often require hundreds or thousands of demonstrations per task. We investigate two fundamental priors for improving learning efficiency: decomposing manipulation trajectories into sequential alignment and interaction phases, and retrieval-based generalisation. Through 3,450 real-world rollouts, we systematically study this decomposition. We compare different design choices for the alignment and interaction phases, and examine generalisation and scaling trends relative to today's dominant paradigm of behavioural cloning with a single-phase monolithic policy. In the few-demonstrations-per-task regime (<10 demonstrations), decomposition achieves an order of magnitude improvement in data efficiency over single-phase learning, with retrieval consistently outperforming behavioural cloning for both alignment and interaction. Building on these insights, we develop Multi-Task Trajectory Transfer (MT3), an imitation learning method based on decomposition and retrieval. MT3 learns everyday manipulation tasks from as little as a single demonstration each, whilst also generalising to novel object instances. This efficiency enables us to teach a robot 1,000 distinct everyday tasks in under 24 hours of human demonstrator time. Through 2,200 additional real-world rollouts, we reveal MT3's capabilities and limitations across different task families. Videos of our experiments can be found on at https://www.robot-learning.uk/learning-1000-tasks.