Learning from Fleet Experience: A Collective Learning Paradigm for Robots

AI summary

Overview

Research area: Robot learning and robotics systems — specifically, how fleets of deployed robots can improve a shared vision-language-action policy from their own execution experience, and the distributed infrastructure needed to make that loop work.

Technical level: Advanced. The paper combines a learning algorithm (retrieval-based fine-tuning of a flow-matching VLA policy), a formal local performance analysis, and a fault-tolerant distributed training system.

Scope: The paper introduces collective experience learning (CEL) as a paradigm, instantiates it as the FERT algorithm and FATS system, and evaluates the result on five real-world manipulation tasks involving fleets of up to 96 robots.

What This Paper Is About

Robots deployed in the real world generate useful experience, but the standard way to learn from it — having teleoperators watch executions and supply corrections when policies fail — becomes impossible to sustain as fleets grow. The paper asks how a fleet of robots can pool the experience from their own executions to improve one common policy, using only success/failure outcome labels rather than new expert demonstrations or corrective actions. It proposes a learning paradigm for this, an algorithm that decides which successful experience to learn from, and a system that keeps the loop running when individual robots drop out.

Key Contributions

1. Collective experience learning (CEL) as a paradigm. A formulation in which robots share trajectories generated during deployment to improve a shared policy, with updated policies returned to the fleet to generate further experience — closing the loop between distributed execution and centralized policy improvement.

2. Failure-guided experience retrieval fine-tuning (FERT). An algorithm that keeps successful fleet trajectories in a collective experience buffer, uses the observation sequence preceding a failure as a retrieval query, finds temporally aligned segments in successful trajectories via slope-constrained subsequence dynamic time warping (S-DTW), and raises the sampling priority of those segments during supervised fine-tuning. Failures decide where to focus learning; successes supply the action targets.

3. Fault-tolerant asynchronous training system (FATS). A distributed system with a data plane (SwarmLink) that pools cloud inference and assembles streamed trajectory fields for training, and a control plane (HotSwarm) that restarts failed robot workers in place while other robots keep collecting experience.

4. A local theoretical account of why retrieval helps. Lemmas on successful-replay alignment and retrieval alignment, combined into a theorem giving a local performance guarantee: under stated conditions a sufficiently small step size yields FERT improvement greater than uniform successful replay, which in turn exceeds the starting policy, with an explicit lower bound separating the replay term, the retrieval term, and a finite-step penalty.

5. Real-world fleet evaluation. Five manipulation tasks spanning deformable objects, precise alignment, tool use and multi-stage execution, evaluated over 120 trials per task on two physical robots excluded from training data collection, plus a fleet-size study and a retrieval ablation.

Main Findings

• High success on unseen robot instances: At the largest evaluated fleet size for each task, policies reached success rates of 93.3–99.2%, measured over 120 trials per task split equally between two physical robots excluded from training data collection (testing transfer to robot instances of the same platform that did not contribute data).

• More robots, more experience, generally better results: With collection time per robot held fixed, larger fleets supplied more experience and FERT success rates generally increased across the evaluated fleet sizes.

• Large gain on multi-stage assembly: On multi-stage phone packaging, learning with 96 robots raised success from 79.2% at the start of FERT to 98.3%.

• Retrieval is doing real work: In a 16-robot experiment on the same phone-packaging task, removing failure-guided retrieval dropped success from 86.7% to 64.2%, supporting the contribution of targeted experience selection over plain successful replay.

• Fleet interruptions are frequent but survivable: In the 96-robot run, fault incidents affected 22 robots, and some robot was interrupted every 22 min on average; environment resets pause robots far more often. FATS was designed so that transient faults (network interruption, low battery, triggered joint limit) do not stall the whole fleet.

• Interaction and experience have very different data profiles: An inference request carries 0.45 MB of images per model input and must be answered before the robot's next interaction, whereas a complete phone-packaging episode is 1.58 GB, with episodes completing about every 4 min. SwarmLink separates these paths and reuses inference observations for training records.

• Theory matches the design: The analysis states that under current-policy successful replay with the retrieval-alignment conditions, fitting successful experience gives an update aligned with improving expected return, and retrieval adds a positive first-order contribution — expressed as the μ (replay alignment) and κ (retrieval contribution) terms in the bound.

• Not reported in the provided excerpt: The detailed per-task breakdowns, the full experimental section (Section 4), and the complete system description after Section 3.2.3 are truncated in the supplied content; the figures above are those stated in the abstract and introduction.

Methodology in Plain English

The approach starts from an existing policy that can already attempt the task. In this work that starting point is π₀, a flow-based vision-language-action model, first trained with supervised fine-tuning and multi-robot HG-DAgger (described in the paper's Appendix C).

From there, robots run autonomously and every episode ends with a simple binary label: success or failure. Humans provide those labels and reset the environments — but crucially, humans do not provide corrective actions.

Successful episodes go into a shared buffer; failed episodes do not contribute actions at all. Instead, each failure is turned into a search query: the paper takes the last stretch of observations before the failure (up to 90 frames, ending 10 frames — about one-third of a second at 30 Hz — before the recorded endpoint, to avoid terminal frames that may reflect a delayed failure signal). Each observation is encoded as a frozen vision-model feature map plus normalized joint state, so the representation stays consistent as the policy changes.

That query is then matched against every cached successful trajectory using dynamic time warping with a slope constraint, which allows the query to be stretched by at most a factor of two in time while preserving order. The best-matching intervals are selected (K = 10 candidates), each expanded by 30 frames before and after, and every action chunk overlapping a retrieved interval gets its sampling priority increased, with exponential decay (ρ) so old retrieval events fade. Training then simply fits the successful action chunks, sampled in proportion to those priorities — for the flow-based policy this is done with the base model's conditional flow-matching loss as a stand-in for likelihood.

To keep retrieval cheap, both query and candidate sequences are downsampled by a factor of three, giving a theoretical 9× speedup and about 6× in practice.

On the systems side, robots, inference workers and training workers are all scheduled as resources. Inference is a shared cloud pool rather than fixed robot-to-server pairings, so a paused robot frees capacity for others. Each inference worker dispatches when it either fills a batch or hits a batching timeout, so nobody waits indefinitely for a slow robot. Experience is assembled in the cloud from streamed fields, and an episode is only released to training once all required fields and its outcome arrive. The control plane gives every process a unique identity attached to its data, so stale replies from a restarted worker can be rejected, and repair of connections flows back down after a failure.

Why This Matters

Impact on research. The paper reframes robot learning during deployment as a collective rather than individual process, and shows that a fleet's own successes can serve as the action-supervision source once the policy has been initialized. This decouples the growth of learning data from the growth of teleoperation capacity — the binding constraint in prior fleet-learning systems — and provides both a local theoretical rationale and a real 96-robot deployment.

Real-world applications:

• Multi-stage assembly and packaging: the phone-packaging task, where errors compound across stages, is exactly the setting where the 79.2% → 98.3% improvement matters.
• Deformable-object manipulation: named as one of the evaluated task categories, relevant to food handling, textile handling and cable routing.
• Precision insertion and alignment tasks: named in the task set; relevant to electronics assembly and connector mating.
• Tool use: also named, relevant to maintenance and service robotics.
• Fleet operations generally: warehouses, labs or factories running many identical robot platforms where individual units intermittently drop out for resets, charging or faults.

Industry relevance. The system results are directly operational: a single 30 Gbps edge-cloud link serving 96 robots and six cloud servers, with measured fault behavior and modeled comparison against restarting the whole system after each fault (restart times of 2–10 min in the model). This is the kind of evidence an operator needs before trusting a shared-policy loop in production.

Future Directions

• Reducing the remaining human role. Operators still label outcomes and reset environments in these experiments. The paper notes that outcome labels could come from a learned reward model instead — an obvious next step toward a fully self-sustaining loop.

• Removing the expert-supervised initialization. FERT depends on a policy obtained via SFT and multi-robot HG-DAgger. Whether collective experience alone can build capability, rather than only refine it, is left open.

• Strengthening the theory. Theorem 1 is explicitly local: it compares a single sufficiently small mean-gradient update against uniform successful replay under stated alignment and stage-model assumptions. Extending guarantees to repeated updates with historical experience (the practical setting) remains open.

• Broadening beyond the evaluated tasks and platforms. The results cover five manipulation tasks and same-platform transfer to robots excluded from data collection. Transfer across different robot instances of differing hardware, and to tasks outside the named categories, is not established in the reported results.

Target Audience

Robotics and embodied-AI researchers working on real-world policy improvement, imitation learning and VLA models; ML systems engineers interested in asynchronous, fault-tolerant training infrastructure for physical fleets; and industrial practitioners running or planning multi-robot deployments (warehouse, manufacturing, lab automation) who need to know whether operational experience alone can raise reliability without proportionally more teleoperation. Readers should be comfortable with imitation learning and distributed systems concepts; the algorithmic and theoretical sections are dense, but the abstract-level results are accessible to a technical product or operations audience.

Authors’ abstract

Experience generated during deployment offers an opportunity for robots to improve policy reliability through learning from their own executions. However, learning through human corrections requires teleoperators to monitor execution and intervene when policies fail, a demand that is difficult to meet as robot fleets grow. We introduce collective experience learning (CEL), a learning paradigm in which deployed robots share their execution experience to improve a common policy. We realize CEL through failure-guided experience retrieval fine-tuning (FERT), which uses failure contexts from robot executions to retrieve relevant trajectory segments from the fleet's shared successful experience and uses their actions as supervision for policy updates, without collecting additional expert demonstrations or corrective actions. A fault-tolerant asynchronous training system (FATS) coordinates experience collection, policy optimization and policy distribution despite local interruptions within the robot fleet. Across five real-world manipulation tasks, the resulting policies achieve success rates of 93.3-99.2%, evaluated over 120 trials per task on robots excluded from training data collection. With collection time per robot held fixed, larger fleets provide more experience and FERT generally achieves higher success rates across the evaluated sizes. On multi-stage phone packaging, learning with 96 robots increases success from 79.2% to 98.3%. These results support collective experience as a practical basis for scalable robot learning and a path towards improving policy reliability during deployment without collecting additional expert corrective actions.