Does Imitation Learning Preserve Temporal Robustness in Dexterous Manipulation? An Expert-Learner Comparison Across Task Execution Speeds

AI summary

Overview

Research area: Robotics — imitation learning for dexterous manipulation, with a focus on evaluation methodology (temporal robustness). Technical level: Intermediate. The paper assumes familiarity with imitation learning, policy evaluation, and architectures such as Action Chunking with Transformers (ACT), but the central question is conceptual rather than deeply mathematical. Scope in one sentence: A direct, matched-conditions comparison of a scripted expert and imitation-learned ACT policies on a single contact-rich parcel manipulation task, evaluated across a range of task execution speeds.

What This Paper Is About

Imitation-learned manipulation policies are usually stress-tested against changes in scenes, objects, or instructions, but rarely against changes in how fast the task is executed. That gap leaves unclear how much of the expert's temporal robustness actually survives the imitation process. This paper holds task conditions, initial-condition draws, and speedup factors fixed and compares the expert against the learner, asking whether matching the expert at normal speed means matching it when the task is sped up.

Key Contributions

1. Reframes temporal robustness as an explicit expert-versus-learner comparison, rather than evaluating a learned policy in isolation across speeds.
2. Introduces ParcelStow, a contact-rich task in which a robot acquires, reorients, and inserts a parcel, with demonstrations spanning the speedup range for the manipulation phases that follow parcel acquisition.
3. Provides a speed-swept evaluation of a scripted expert and multiple ACT policies (trained on the expert's demonstrations, with different parameter initializations), including stage-level failure attribution and an analysis of acquisition under a relative-motion handoff.
4. Releases code, data, and evaluation scripts at the linked GitHub repository.

Main Findings

• Nominal speed hides the difference: Both the scripted expert and the ACT policy achieve 100 percent task success at nominal speed.
• Divergence appears within the demonstrated range: At the maximum demonstrated speed, expert success is 84 percent while ACT success is 53 percent — the gap opens without leaving the range the learner was trained on.
• Degradation is not a single-initialization artifact: Two ACT policies with different parameter initializations degrade similarly, dropping by 34 and 48 percentage points from nominal speed to the maximum demonstrated speed, compared with 16 points for the expert.
• Failures concentrate at insertion: Stage-level analysis attributes 35 of ACT's 47 failures at the maximum demonstrated speed to insertion misalignments.
• Acquisition transfer is not the whole story: Under the relative-motion handoff, every ACT acquisition retains the parcel through reorientation and transfer in free space, yet only 64 percent complete the overall task, versus 95 percent after expert acquisition.
• A hard boundary on force closure: Across all evaluated policies and speeds, none of the 414 acquisitions without force closure completes the task.
• Headline implication: Equal nominal task success does not imply preservation of expert performance across execution speeds.

Methodology in Plain English

The researchers built one task, ParcelStow, and ran two kinds of policy on it: a scripted expert, and an ACT policy trained from that expert's demonstrations. To make the comparison fair, both were tested under identical task conditions, identical initial-condition draws, and identical speedup factors — so any difference in outcome is attributable to the policy, not to easier or harder setups. The demonstrations cover the speedup range for the phases after the parcel is acquired, meaning the learner has seen sped-up behavior in training. The team then measured success at each speed, broke failures down by task stage to see where things go wrong, and separately examined the handoff after acquisition, distinguishing acquisitions that did and did not establish force closure. The abstract does not report further architecture, training, or speed-sampling details.

Why This Matters

Research impact: The result challenges a common evaluation shortcut — reporting nominal-speed parity and calling a learned policy robust. Temporal robustness becomes a dimension that claims about imitation quality should be required to address, and stage-level attribution offers a template for locating where robustness is lost.

Real-world applications:

• Warehouse and fulfillment robotics, where parcel stowing and sortation run at production rates rather than demonstration rates.
• Assembly and insertion tasks, where cycle-time pressure is routine and misalignment failures are costly.
• Packaging and palletizing systems that must maintain throughput without dropping or misplacing objects.
• Service and logistics robots expected to operate at speeds dictated by human workflows rather than by their training data.

Industry relevance: Deployment decisions often trade throughput against reliability. This work suggests that a policy matching its teacher at slow speed can lose a substantial fraction of its reliability when sped up, particularly at the insertion stage, and that acquisition quality (specifically force closure) can bound overall success regardless of how well the rest of the motion is imitated.

Future Directions

• Repeat the expert-learner comparison on other contact-rich tasks and other imitation-learning architectures to test how general the degradation pattern is.
• Try interventions that might close the speed gap, such as speed-augmented demonstrations, speed-conditioned policies, or explicit force-closure-aware acquisition behavior.
• Determine whether the force-closure requirement observed here is a general property of contact-rich insertion or specific to ParcelStow.
• Establish standard metrics and benchmarks for temporal robustness, and probe whether the degradation trend continues beyond the demonstrated speed range — something the abstract does not address.

Target Audience

Robotics researchers working on manipulation and imitation learning; evaluation-focused machine learning practitioners; engineers deploying learned policies in production settings where speed varies; and graduate students looking for a concrete example of how evaluation design shapes conclusions about policy robustness.

Authors’ abstract

Dexterous manipulation policies learned by imitation are typically evaluated for robustness to variation in scenes, objects, or instructions, but their performance across task execution speeds is less often examined. This leaves open how much temporal robustness a learner retains relative to the expert it imitates. We compare an expert and learner under the same task conditions, initial-condition draws, and speedup factors. We instantiate the evaluation in ParcelStow, a contact-rich task in which the robot acquires, reorients, and inserts a parcel. The demonstrations span the speedup range for the manipulation phases after parcel acquisition. A scripted expert and an Action Chunking with Transformers (ACT) policy trained from the expert's demonstrations both achieve 100 percent task success at nominal speed. Their success rates diverge within the demonstrated range: at its maximum, expert success is 84 percent and ACT success is 53 percent. Two ACT policies with different parameter initializations show similar degradation, decreasing by 34 and 48 percentage points from nominal speed to the maximum demonstrated speed, compared with 16 points for the expert. Stage-level analysis shows that 35 of ACT's 47 failures at the maximum demonstrated speed are insertion misalignments. Under the relative-motion handoff, every ACT acquisition retains the parcel through reorientation and transfer in free space, but only 64 percent complete the overall task, compared with 95 percent after expert acquisition. Across all evaluated policies and speeds, none of the 414 acquisitions without force closure completes the task. Equal nominal task success therefore does not imply preservation of expert performance across execution speeds. Code, data, and evaluation scripts are available at https://github.com/coenwerem/parcelstow.