LeRobot: An Open-Source Library for End-to-End Robot Learning

AI summary

Overview

Research area: Robotics / robot learning infrastructure — open-source software tooling for real-world robot learning, spanning hardware middleware, datasets, policy implementations, and inference.

Technical level: Intermediate. The paper is a systems-and-tooling paper rather than a theory paper; it assumes familiarity with imitation learning, reinforcement learning, and standard deep learning workflows, but its arguments are largely about engineering integration and accessibility.

One-sentence scope: The paper presents lerobot, an open-source, vertically integrated library that unifies low-level motor middleware, multimodal dataset collection/storage/streaming, state-of-the-art policy implementations, and a decoupled inference stack for end-to-end real-world robot learning.

What This Paper Is About

Robot learning has advanced rapidly thanks to cheaper teleoperation hardware, larger openly shared datasets, and scalable learned policies, but the ecosystem remains fragmented: middleware is often tailored to a single robot, datasets are released in incompatible formats (TensorFlow Datasets, ROS bags, bespoke JSON), and algorithm implementations vary in ways that hurt reproducibility. The authors' goal is to remove this fragmentation by building one open-source library that covers the whole stack — from reading motor positions on a low-cost arm to streaming a million-episode dataset to running a large pretrained policy remotely.

Key Contributions

1. Unified robot integration. A consistent, Python-based middleware API for real-world motor control across a range of platforms — from low-end manipulators to humanoid arms and hands — built to interface directly with the low-level SDKs of low-cost actuator producers (FeeTech and Dynamixel), and designed to be extensible and composable.

2. Standardized datasets (LeRobotDataset). A self-contained, multimodal schema for recording, storing, and streaming high-frame-rate sensorimotor and image data, with metadata covering task descriptions, robot embodiment, FPS, and sensor types. It is accompanied by StreamingLeRobotDataset, which fetches frames on demand from remote storage instead of preloading a corpus.

3. Optimized, decoupled inference. An inference stack that decouples action prediction from action execution both physically (inference can run on a remote machine over the network, with more compute than is available onboard) and logically (an asynchronous producer-consumer scheme that predicts action chunks in parallel with control, merging overlapping predictions through a user-specifiable aggregation function f).

4. Efficient, reusable algorithms. Pure-PyTorch implementations of state-of-the-art methods across paradigms — HIL-SERL and TD-MPC for RL; ACT, Diffusion Policy, and VQ-BET for single-task behavioral cloning; π₀ and SmolVLA for multi-task models — supporting both training from scratch and reuse of openly available pretrained models.

Main Findings

• Real-world hardware is cheap and widely supported. lerobot integrates SO-100 and SO-101 (jointly SO-10X) arms in single and bimanual setups, Koch-v1.1, ALOHA-2, the Hope-JR humanoid arm, Stretch-3, LeKiwi, and the Reachy-2 humanoid. Reported costs for platforms with an openly available Bill of Materials are approximately €225 for SO-100/101 (€550 bimanual), €670 for Koch-v1.1 (€1346 bimanual), €21k for ALOHA, €500 for HopeJR-Arm, and €230 for LeKiwi.

• The dataset format has achieved broad community adoption. As of September 2025, 16K+ datasets from 2.2K+ individual contributors are openly shared in the LeRobotDataset format, including data for robots not natively integrated in the library (Franka Emika Panda, xArm, R1Pro).

• Datasets for non-integrated industrial arms dominate downloads, while low-cost arms dominate contributions. By downloads, Panda leads with 1'878'395, followed by xArm (1'107'329), WidowX (832'177), KUKA (662'550), SO-101 (319'586), SO-100 (278'697), and Koch-v1.1 (43'561). By dataset count, SO-100 (5'161) and SO-101 (3'965) lead, versus 588 for Panda and 3 for KUKA. 50%+ of contributed datasets were collected on SO-10X platforms.

• Episode tracking began in October 2024. Panda datasets account for 926'776 episodes, xArm 450'329, KUKA 419784, WidowX 214'117, SO-100 78'510, SO-101 58'299, and Koch-v1.1 20'959.

• ACT is the most popular policy. It dominates the number of model uploads, and consistently ranks among the most-downloaded and most-uploaded policies. The authors attribute this to its small size (52M parameters), fast inference, and its ability to yield well-performing policies from as little as 50 real-world trajectories. Its limitation is being single-task, requiring retraining when conditions change. TD-MPC, HIL-SERL, and VQ-BET are absent from all upload/download visualizations as they are not typically uploaded by users.

• Memory footprints largely match theoretical estimates from parameter count and precision. All models were run in full fp32 with a 5000ms inference timeout on a MacBook Pro M1 (2021, 16GB, CPU only), the same machine with the MPS backend, an NVIDIA RTX 4090, and an NVIDIA A100. Peak memory: ACT (52M params) 817.4MB CPU / 462MB MPS / 211.24 MB on both the RTX 4090 and A100; Diffusion Policy (263M) 1.22GB / 224MB / 1.12 GB on both GPUs; π₀ (3.5B) 4.13GB / 97MB / 13.32 GB on both GPUs; SmolVLA (450M) 1.69GB / 555MB / 1.75 GB on both GPUs. The exceptions are the CPU and MPS backends, where unified memory and swap offloading introduce variability.

• Small task-specific models reach high control rates; large foundation models do not. Average inference latency over 100 forward passes: ACT 182.313 ± 40.82 ms (CPU), 42.667 ± 10.085 ms (MPS), 5.013 ± 0.061 ms (RTX 4090), 13.77 ± 0.445 ms (A100) — roughly 100–200Hz on the high-end GPUs. Diffusion Policy: 100% timeouts on CPU, 3453.838 ± 39.271 ms (MPS), 369.788 ± 0.193 ms (RTX 4090), 613.893 ± 10.173 ms (A100). π₀: 100% timeouts on both CPU and MPS, 209.381 ± 2.762 ms (RTX 4090), 568.978 ± 2.937 ms (A100). SmolVLA: 2028.461 ± 302.59 ms with 2% timeouts (CPU), 721.826 ± 57.748 ms (MPS), 99.244 ± 1.195 ms (RTX 4090), 278.833 ± 1.886 ms (A100). The authors state π₀ fails to complete inference within the 5s limit on lower-tier devices, underscoring the challenge of deploying robotics foundation models in practice.

• Streaming performance is comparable to preloading in steady state. StreamingLeRobotDataset partitions data into .parquet tabular records and .mp4 videos with lightweight metadata downloaded in full; it uses an IterableDataset interface and torchcodec for on-the-fly decoding. Given good network connectivity, timing performance is comparable to a pre-loaded LeRobotDataset after initialization.

• Simulation is used primarily for benchmarking, not training. lerobot provides evaluation support for LIBERO (four fixed task suites of 10 tasks each: LIBERO-Spatial, LIBERO-Object, LIBERO-Goal, plus LIBERO-90 and LIBERO-Long) and Meta-World (50 distinct manipulation tasks structured into multi-task MT10/MT50 and meta-learning ML1/ML10/ML45 regimes).

• Low code overhead. The library provides composable recipes to train a model from scratch in less than 100 lines of code and to serve models in less than 40 lines of code.

Methodology in Plain English

The authors take an engineering-integration approach rather than proposing a new learning algorithm. They first diagnose why robot learning research is slow: middleware locked to specific hardware, datasets in incompatible formats, and implementations whose small differences produce large reproducibility problems. They then build one library that spans the whole pipeline.

Concretely, they write a shared Python middleware layer that sits on top of low-cost actuator SDKs and exposes one consistent API across many robots, so that configuration can be read from a "leader" robot and written to a "follower" robot for teleoperation, or the follower can be driven directly by a learned policy. They define LeRobotDataset, a schema that stores sensorimotor streams, multiple camera feeds, teleoperation signals, robot embodiment details, and task text, plus a streaming variant that pulls frames on demand from remote storage so users do not need to download whole corpora. They re-implement several published policies in pure PyTorch and measure their peak memory and inference latency across four hardware targets under fp32 with a fixed 5-second timeout. Finally, they replace the conventional synchronous inference loop with an asynchronous producer-consumer design in which a policy predicts chunks of future actions while the robot consumes them at a fixed control rate, merging overlapping chunks with a configurable function.

Why This Matters

Impact on research. The paper argues that ecosystem fragmentation imposes "incidental complexity" on researchers, diverting effort from scientific questions to systems integration. By providing one end-to-end stack with shared middleware, a standardized dataset schema, and reproducible algorithm implementations, lerobot aims to make results comparable across labs and to make hardware variability less of a confound. Its reported adoption — 16K+ datasets from 2.2K+ contributors, and the fact that the most-downloaded datasets are academic benchmarks released by the research community rather than natively supported robots — is offered as evidence that a common format changes how the field shares data.

Real-world applications:

• Low-cost manipulation research and education, using platforms like SO-100/SO-101 at roughly €225 (€550 bimanual) or LeKiwi at roughly €230.
• Bimanual manipulation and humanoid research on ALOHA-2, Hope-JR, Stretch-3, and Reachy-2.
• Teleoperation-based data collection for training task-specific policies, where ACT can yield usable policies from as few as 50 real-world trajectories.
• Language-conditioned control on real robots via SmolVLA, enabling broader applicability than single-task models.

Industry relevance. The physical decoupling of inference means computationally expensive policies can run on remote servers rather than onboard hardware, which matters for deploying large models on cheap robots. The latency and memory benchmarks provide concrete deployment guidance: small models hit roughly 100–200Hz on RTX 4090 and A100 GPUs, whereas π₀ at 3.5B parameters times out entirely on CPU and MPS, indicating that robotics foundation models still require careful hardware provisioning.

Future Directions

• Broaden robot coverage. The authors state the supported set of arms, grippers, sensors, and controllers is "far from exhaustive," and note that lerobot went from 3 manipulation setups (Koch-v1.1, SO-100, ALOHA) in early 2025 to 8 regular, humanoid, and mobile manipulators, arguing that a similar rate of progress is needed.

• Extend algorithm coverage. The implementations span key paradigms but are explicitly described as non-exhaustive; adding more algorithms is left as future work.

• Add low-level inference optimization. The library currently disregards quantization, graph compilation, and similar techniques, which the authors identify as necessary for strong practical inference performance.

• Improve dataset hygiene and streaming ergonomics. Faulty datasets that do not record the robot platform are tagged unknown and dominate the "Other" category for dataset count (2370), suggesting metadata completeness is an open community problem.

• Open questions the paper raises: How should overlapping action chunks be aggregated for different tasks, given that f is left user-specifiable? And can foundation-scale policies be made practical for lower-tier hardware without sacrificing capability?

Target Audience

Robotics researchers and graduate students who need hardware they can afford and pipelines they can reproduce; ML practitioners moving into embodied control who want working implementations of ACT, Diffusion Policy, π₀, or SmolVLA without re-deriving them; hobbyists and small labs building on low-cost open hardware with an openly available Bill of Materials; and engineering teams evaluating the deployment cost, memory footprint, and inference latency of robot policies across CPU, MPS, RTX 4090, and A100 hardware.

Authors’ abstract

Robotics is undergoing a significant transformation powered by advances in high-level control techniques based on machine learning, giving rise to the field of robot learning. Recent progress in robot learning has been accelerated by the increasing availability of affordable teleoperation systems, large-scale openly available datasets, and scalable learning-based methods. However, development in the field of robot learning is often slowed by fragmented, closed-source tools designed to only address specific sub-components within the robotics stack. In this paper, we present \texttt{lerobot}, an open-source library that integrates across the entire robot learning stack, from low-level middleware communication for motor controls to large-scale dataset collection, storage and streaming. The library is designed with a strong focus on real-world robotics, supporting accessible hardware platforms while remaining extensible to new embodiments. It also supports efficient implementations for various state-of-the-art robot learning algorithms from multiple prominent paradigms, as well as a generalized asynchronous inference stack. Unlike traditional pipelines which heavily rely on hand-crafted techniques, \texttt{lerobot} emphasizes scalable learning approaches that improve directly with more data and compute. Designed for accessibility, scalability, and openness, \texttt{lerobot} lowers the barrier to entry for researchers and practitioners to robotics while providing a platform for reproducible, state-of-the-art robot learning.