LIBERO-Industry: Benchmarking Robotic Manipulation under Industrial Process Constraints

AI summary

Overview

Research area: Robot learning and embodied AI — specifically, benchmarking generalist robot policies (vision-language-action models and world-action models) for industrial manipulation.

Technical level: Advanced. The paper assumes familiarity with policy learning, simulation benchmarks, behaviour-based task specification, and manipulation metrics, though its central argument is stated plainly.

Scope: The paper introduces LIBERO-Industry, a simulation benchmark built on LIBERO that turns four industrial process constraints into controllable task variables, and reports simulation results for five policies plus a physical robot validation study.

What This Paper Is About

Factories that make many different products in small batches, such as high-mix, low-volume electronics assembly, constantly change what is built and under what conditions. Generalist robot policies are appealing here because one model can be conditioned on language and images rather than reprogrammed per product, but the benchmarks used to compare them mostly test household-style tasks and ignore production realities like moving conveyor belts, deadlines, and tight placement tolerances. The paper's goal is to build a benchmark where these industrial conditions are knobs that researchers can turn on individually and in combination, and to measure how current generalist policies behave when they are turned on.

Key Contributions

1. An industrial-process benchmark. LIBERO-Industry extends LIBERO with four constraints made explicit and measurable: product variants, moving material flows, finite execution windows, and tight geometric tolerances. The authors' comparison table marks LIBERO-Industry as the only listed dataset with explicit evaluation across language-conditioned policies, product variants, material flow, execution time windows, geometric tolerance, and real-world validation.

2. Three process-constrained task suites with a controlled factor design. Static-to-Dynamic (S2D) loading, Dynamic-to-Static (D2S) unloading, and Dynamic-to-Dynamic (D2D) transfer are instantiated as 216 tasks (180 train, 36 test). Each suite is a full grid of 4 objects × 3 conveyor speeds × 2 directions × 3 difficulty levels = 72 combinations, with 60 train and 12 test.

3. A procedural task-generation pipeline. Layouts are serialized as BDDL specifications with initial predicates, transport conditions, instructions and goals; infeasible layouts are resampled, and a task-specific scripted expert produces demonstrations that pass completion, kinematic-safety and replay checks before being stored as HDF5 files with provenance.

4. Empirical evaluation and failure diagnosis. Five generalist policies are benchmarked in simulation, a cumulative stress progression from D0 to D6 is traced to localize where performance breaks, and a physical D2S study compares co-training on LIBERO-Industry data against co-training on original LIBERO data under matched conditions.

Main Findings

• No policy completed a single task. Across 36 held-out tasks and 360 rollouts per policy, all five checkpoints (π0.5, OpenVLA-OFT, UniVLA, X-VLA, Cosmos Policy) recorded a success rate of 0.0% (0/360). Suite-level breakdowns also show zero success in S2D, D2S and D2D.

• π0.5 shows the strongest partial progress. It is the only policy with nonzero target-transfer completion (PCR_T = 0.4% overall, including 1.1% in S2D) and the highest terminal-slot occupancy (PCR_S = 12.5%, versus 11.2–12.2% for the other four).

• The slot-occupancy metric is dominated by a baseline. Held-out BDDL specifications give a mean initial-state slotting baseline of PCR_S^init = 12.31%, so PCR_S values near that figure mostly reflect pre-existing destination objects that stayed valid rather than new transfers; PCR_T is what isolates new target transfers.

• Stress accumulates into collapse. For π0.5, success falls from 97.0% at D0 to 43.8% at D4, then to 2.1% at D5 and 0.0% at D6 as the task expands from one object to multiple objects. Partial transfer persists at D5 and D6 (PCR_T = 18.1% and 11.5%), so the collapse is about finishing, not about reaching objects at all.

• Reach-stage failures localize the bottleneck. Reach-stage failures account for 132/240 episodes at D5 and 202/240 at D6, pointing to longer multi-object sequences as the principal difficulty. The authors note that because each stress stage retains the preceding modifications, the progression is diagnostic rather than causal.

• Industry co-training outperforms LIBERO co-training in the physical study. On an AUBOi5H–CTAG2F90D workcell, S2 (1,412 LIBERO-Industry simulation demonstrations plus 95 physical demonstrations) beat S1 (1,724 original-LIBERO demonstrations plus the same 95 physical demonstrations). At 0.03 m/s, PCR_T was 34.7% (25/72) for S2 versus 26.4% (19/72) for S1; at 0.05 m/s, 23.6% (17/72) versus 4.2% (3/72).

• Later placement steps separate the conditions more sharply. The fraction of trials completing the third placement was 16.7% for S2 versus 4.2% for S1 at 0.03 m/s, and 20.8% versus 0.0% at 0.05 m/s.

• The advantage holds under operating-condition variation. Across five physical variations in illumination, object appearance, camera configuration and table texture, S2 exceeded S1 by 12.5 to 34.7 percentage points, with the largest gap under the camera variation (34.7% versus 0.0%).

Methodology in Plain English

The authors start from LIBERO, a widely used robot-learning benchmark, and keep its language-conditioned observation-and-action interface so that existing models can be dropped in unchanged. On top of that they add a notion of process: each task specifies whether the source of parts and the destination are static, continuously moving, or intermittently moving, along with a conveyor velocity. From velocity and the spacing between parts, they derive how often a part arrives and how long a robot has to act on it, and they turn that schedule into an explicit episode time limit rather than a generic timeout. Requirements get stricter across three levels: Level 1 uses broad sector trays, Level 2 uses object-specific grooves with prescribed orientations, and Level 3 uses form-fitting inlays cut as negative-space profiles — all built around a strict 1 cm clearance rule, with Level 3 meshes processed through V-HACD decomposition.

Tasks are generated procedurally rather than hand-placed. Given an operation anchor, the generator samples fixtures and back-projects moving sources from transport direction, velocity, arrival interval and offset; it sets slot spacing from geometry and service intervals from velocity and dwell. Each layout is written as a BDDL file with initial conditions, transport conditions, instruction and goal, and layouts that fail a feasibility gate are resampled. A scripted expert then attempts each task from a fixed initial state and seed, and only trajectories passing completion, kinematic-safety and replay checks are kept as demonstrations; failed attempts can trigger refinement of the collector.

Instructions are authored offline with an LLM-assisted workflow and perturbed by level: Level 1 paraphrases, Level 2 replaces object names with visual descriptions, Level 3 uses industrial routing terminology. Evaluation reports three numbers — full task success, target-transfer completion, and terminal-slot occupancy including pre-existing destination objects. Finally, the authors run a matched physical experiment: two co-training setups initialized from the same π0.5 checkpoint, identical physical demonstrations, identical 9:1 simulation-to-real sampling ratio and identical 30,000 training steps, differing only in which simulation corpus was used.

Why This Matters

For research, the paper supplies a diagnostic instrument rather than a leaderboard. The finding that all five policies score 0.0% success while still showing measurable partial transfer, and that failures cluster at the reach stage during longer multi-object sequences, gives the field a specific target rather than a vague "policies need to improve" conclusion. The matched co-training design also offers a template for isolating the effect of simulation data source from the effect of data volume and physical supervision.

Real-world applications suggested by the paper's framing and asset domains:

• High-mix, low-volume electronics assembly, the motivating setting, where products and production conditions change frequently.
• Consumer electronics handling, represented by workpieces such as mice, a headset and a keyboard.
• Personal care and cosmetics packing, represented by cream bottles and jars, pump bottles and serum bottles.
• Small-part logistics and kitting involving model figurines and vehicles moved between conveyors and slotted carriers.

Industry relevance centres on the cost of re-engineering. The paper argues that conventional automation works well when tasks and environments are stable but needs reprogramming, recalibration, robot re-teaching or workcell changes when they are not. A policy that adapts through language conditioning could reduce that burden — but only if it can be shown to operate under the timing and tolerance conditions that production actually imposes, which is precisely what this benchmark measures.

Future Directions

• Determining whether generalist policies can keep flexibility without sacrificing production rates. The authors name this as the key open question, with the benchmark's finite opportunity windows standing in for heterogeneous factory needs.

• Moving to production-calibrated settings. The paper is explicit that its time limits are benchmark time limits, not universal factory takt times, and calls for evaluation that relates cycle times to empirical operating rates.

• Testing repeated material arrivals, recovery events and long-horizon throughput, extending evaluation beyond single-episode completion toward sustained operation.

• Attacking the identified bottleneck. Since reach-stage failures dominate at D5 and D6, closing the gap on longer multi-object sequences is the concrete technical problem the results expose.

Target Audience

Robotics and embodied-AI researchers who develop or evaluate generalist policies, especially vision-language-action and world-action models; benchmark designers who need a template for encoding process constraints as controlled variables; and industrial automation engineers or manufacturing researchers assessing whether learned policies are ready for flexible production. Readers focused on household manipulation benchmarks will find the framing familiar but the constraints new, while readers from manufacturing will find the terminology — takt time, dwell, arrival intervals, ISO 9283 pose accuracy — treated with care.

Authors’ abstract

Industrial manipulation in flexible manufacturing, e.g., high-mix, low-volume (HMLV) electronics assembly, requires robot policies to adapt to changing products and production conditions. Such settings impose industrial process constraints, including product variants, moving material flows, finite execution windows, and tight geometric tolerances. However, existing evaluation frameworks do not systematically capture these constraints when assessing generalist robot policies. To address this gap, we introduce LIBERO-Industry, a novel benchmark built on the widely used robot learning benchmark LIBERO, that translates the aforementioned industrial constraints into controllable task conditions and measurable success criteria. Specifically, LIBERO-Industry comprises three classic task scenarios: Static-to-Dynamic (S2D) loading, Dynamic-to-Static (D2S) unloading, and Dynamic-to-Dynamic (D2D) transfer. The suites contain 216 tasks in total that systematically vary object geometry, conveyor speed, interaction-window duration, and geometric acceptance region size. Extensive experiments across five generalist robot policies (including vision-language-action models and world-action models) show that performance progressively degrades as industrial constraints are introduced and combined. Under a shared fine-tuning and evaluation protocol, the LIBERO-Industry co-training condition achieved higher observed target-transfer completion than the original-LIBERO condition.