BiKC+: Bimanual Hierarchical Imitation with Keypose-Conditioned Coordination-Aware Consistency Policies

AI summary

Overview

• Research area: Robotics — imitation learning for bimanual (two-arm) robotic manipulation, combining hierarchical learning, keypose (sub-goal) prediction, generative models (Consistency Models), and vision-language models for contact reasoning.
• Technical level: Advanced. The paper assumes familiarity with diffusion/consistency generative models, score-based modeling, ordinary differential equations, and prior imitation-learning methods such as ACT and Diffusion Policy. The intuition, however, is explained here in plain language.
• Scope: One sentence: The paper proposes BiKC+, a hierarchical imitation-learning framework that identifies bimanual keyposes through a three-stage pipeline, predicts them with a consistency-model-based keypose head, and executes actions with a consistency-model trajectory generator that needs only one-step inference.

Note on completeness: the provided paper content is truncated inside Section IV-D. Sections V (experiments), VI (conclusion), and VII (limitations and future work) are not included, along with Appendix A-A. Therefore no quantitative results, benchmark names, dataset sizes, or success-rate numbers can be reported here — those are simply not present in the supplied text.

What This Paper Is About

Bimanual multi-stage tasks — such as cloth folding, battery slotting, and tool assembling — require two robot arms to coordinate across several sequential stages, and a failure in any one stage propagates to later stages. Existing imitation-learning methods (ACT, Diffusion Policy) either fix a single behavior style at evaluation time or require slow iterative inference, and neither explicitly models the sub-goals needed to complete a whole multi-stage task. The paper's goal is a hierarchical framework that learns from distributionally multi-modal demonstrations, predicts the next bimanual keypose as an explicit sub-goal, and generates action sequences in a single inference step for reliable and efficient execution.

Key Contributions

1. A three-stage pipeline for bimanual keypose identification. It integrates robot-centric and task-centric features by extracting unimanual keyposes with heuristic rules, identifying coordination modes with a VLM-assisted contact-aware solution, and constructing bimanual keyposes through a coordination-driven merging strategy — ensuring synchronization during coordination while preserving flexibility in non-coordination phases.
2. The BiKC+ hierarchical imitation-learning framework. A high-level keypose predictor forecasts the next target keypose, which acts both as guidance for low-level actions and as an indicator that a sub-stage is complete, thereby enhancing per-stage reliability.
3. A scratch-trained Consistency Model as the trajectory generator. It enables one-step inference of high-quality action sequences, balancing efficiency against sample quality — in contrast to concurrent work that distills from pretrained diffusion models and is limited to unimanual tasks.

Main Findings

• Keypose guidance improves per-stage reliability (claimed): The paper states that the keypose acts as both guidance for low-level actions and an indicator for sub-stage completion, and that this "enhances per-stage reliability and thereby improves the overall success rate." No numerical success rates appear in the provided content.
• One-step inference improves efficiency (claimed): The low-level trajectory generator is a Consistency Model that generates action sequences in a single inference step, which the authors state increases inference speed and operational efficiency. No latency or speed measurements are reported in the provided text.
• Demonstrations are distributionally multi-modal in bimanual settings: The paper attributes this to increased degrees of freedom, yielding diverse behavior styles and stage orders — a challenge it addresses with generative modeling rather than by collapsing to a single style as ACT does.
• Coordination modes can be detected automatically from contact relationships: Using Qwen2.5-VL-72B-instruct to produce sequences of contact graphs, the framework applies three principles (static shared-object contact; contact transferring from one arm to the other; two separately held objects forming a new contact between them) to mark coordination-mode durations, which then drive the merging strategy.
• Three bimanual operation categories are distinguished: Coordination mode (both arms toward a shared goal requiring spatial and temporal synchronization), parallel mode (distinct goals concurrently, no synchronization), and sequential mode (goals completed in a specific order without synchronization). Only coordination mode demands strict spatio-temporal synchronization.
• Keyposes are represented in joint space (configuration space), not SE(3): The choice is made to align with the representation of states and actions.
• A dual-head keypose predictor: A shared observation encoder branches into a consistency-model-based bimanual keypose head and an MLP-based mode head for binary classification; training combines the consistency training objective with a binary cross-entropy loss, weighted by β with a default value of 1.
• Mode-dependent sub-stage completion during deployment: If the predicted mode is coordination, a sub-task completes only when both arms simultaneously reach their keyposes; if non-coordination, completion occurs when either arm reaches its keypose. Algorithm 1 details this loop, including resetting the keypose only after the condition is satisfied and padding actions once the action horizon passes the target keypose.
• Experiment scope (stated but not detailed): The introduction states that experiments were conducted in both simulated and real-world environments and that BiKC+ can enhance overall success rates of multi-stage tasks while significantly improving operational efficiency. The actual experimental section, benchmark names, dataset sizes, and figures are not included in the provided content.

Methodology in Plain English

The problem setup. The method assumes a set of demonstrated trajectories, each a sequence of observation-action pairs. Observations combine RGB images with proprioceptive information such as joint positions; actions are desired proprioceptive states. The framework learns two things: a high-level keypose predictor g_ω that outputs the next target keypose and its coordination indicator, and a low-level trajectory generator π_θ that outputs an action sequence conditioned on observations and the predicted keypose.

Step 1 — Find keyposes for each arm separately. The authors scan each arm's action trajectory and flag timesteps using heuristic rules: changes in robot-object contact mode (grasping or releasing), stalling of motion before interaction, and case-specific spatial-temporal rules such as the distance between the two grippers or their heights relative to the tabletop. In implementation terms, a timestep is labeled a keypose when the gripper begins to open or stops closing, the gripper's velocity drops below a threshold, or the relative distance/height is below a threshold. The paper notes that copying unimanual heuristics directly into bimanual settings is difficult, because two arms must interact and stay synchronized rather than each proceeding immediately to its next keypose.

Step 2 — Decide when the arms must be synchronized. A pretrained vision-language model, Qwen2.5-VL-72B-instruct, takes sequences of image frames from demonstrations and outputs sequences of contact graphs whose edges represent contacts. From those graphs, three principles identify coordination: (1) both arms contacting the same object simultaneously; (2) an object losing contact with one arm and establishing contact with the other in the next graph; (3) two objects each contacted by one arm subsequently forming a contact with each other. Durations satisfying these principles define coordination mode.

Step 3 — Merge into bimanual keyposes. In coordination mode, when one arm reaches a keypose, the other arm's joint positions at that timestep are designated as a keypose as well, enforcing synchronization. In non-coordination mode, each arm keeps its own keyposes and proceeds freely, preserving flexibility.

Step 4 — Enrich the keypose dataset. Because keyposes are far fewer than total steps, the authors expand the keypose dataset into tuples of observation plus the preceding and succeeding keyposes, so the predictor learns to forecast a target keypose at every timestep rather than only at identified keypose steps.

Step 5 — Train the high-level predictor. A shared encoder feeds a consistency-model keypose head that predicts the next keypose in joint space and an MLP mode head that predicts the binary coordination indicator. The keypose head is trained with the consistency training objective, which enforces that outputs for two adjacent noise levels on the same trajectory match, using an exponential moving average of the parameters for stability. The mode head is trained with binary cross-entropy, and the combined loss weights the two with β (default 1).

Step 6 — Train the low-level generator from scratch. The trajectory generator is also a Consistency Model, trained via consistency training rather than distilled from a pretrained diffusion model. It is conditioned on a history of observations and on the next keypose, and outputs a short-horizon action sequence. If the action horizon extends past the target keypose step, actions are padded with the value just before that keypose so both arms stay at the keypose.

Step 7 — Deploy in a closed loop. At each step, the policy generates an action sequence in one inference. The keypose is refreshed only when the sub-task completion condition for the current mode is met, after which the predictor produces the next keypose and mode.

Why This Matters

• For research: The work connects three threads that are usually separate — keypose-based hierarchical imitation learning, bimanual coordination modeling, and consistency-model generative policies — and argues that coordination style should inform how keyposes are defined, not just how they are predicted. It also distinguishes itself from Bi-KVIL, which uses learned keyposes as terminal control targets tracked with keypoint-based admittance control, whereas BiKC+ uses keyposes as generative trajectory guides. It extends the authors' earlier BiKC, whose merging of two unimanual keypose sets ignored bimanual coordination and over-restricted individual arm movement.
• Real-world applications (as named in the paper):
  • Cloth folding.
  • Battery slotting.
  • Tool assembling.
  • Transferring a cube from one arm to the other, and two-arm insertion / jointly lifting a box.
  • The authors frame the general target as real-world manufacturing and industrial settings.
• Industry relevance: Bimanual multi-stage manipulation is exactly the class of task where hard-coded leader/follower coordination rules are labor-intensive and must be re-customized per task. The paper positions a learned, automatically detected coordination structure plus single-step inference as a path toward deployable industrial systems; the involvement of Huawei Technologies' Advanced Manufacturing Lab is consistent with that framing.

Future Directions

• Integrate force and torque sensing. The Note to Practitioners states that future research will incorporate additional sensory information such as forces and torques to reproduce subtle motions arising from contact and force interactions, strengthening the approach for fine-grained manipulation.
• Extend coordination principles to new scenarios. The authors claim the three coordination identification principles cover the robotic tasks studied in this paper while remaining extensible to novel scenarios; testing that claim more broadly is a natural next step.
• Reduce reliance on the vision-language model. The pipeline depends on Qwen2.5-VL-72B-instruct to produce contact graphs from image sequences; whether a smaller or on-robot model can do this reliably is an open question the provided text does not address.
• Validate heuristics and thresholds under multi-modal demonstrations. The paper notes that direct transfer of unimanual heuristics is challenging in bimanual settings, and that demonstrations vary in behavior style and stage order; how robust the fixed thresholds (gripper velocity, relative distance, height) are across tasks and operators is an open question, and Section VII, which was to discuss limitations, is not included in the provided content.

Target Audience

Robotics researchers and graduate students working on imitation learning, hierarchical task decomposition, generative policies, and dual-arm manipulation, particularly those interested in replacing hard-coded coordination rules with learned coordination structure. It is also relevant to applied engineers in manufacturing and industrial automation evaluating whether learned bimanual policies can meet real-time efficiency requirements, and to readers tracking the use of vision-language models for contact and interaction reasoning in robot learning. Beginners will find the consistency-model preliminaries mathematically demanding without background in diffusion or score-based generative modeling.

Authors’ abstract

Robots are essential in industrial manufacturing due to their reliability and efficiency. They excel in performing simple and repetitive unimanual tasks but still face challenges with bimanual manipulation. This difficulty arises from the complexities of coordinating dual arms and handling multi-stage processes. Recent integration of generative models into imitation learning (IL) has made progress in tackling specific challenges. However, few approaches explicitly consider the multi-stage nature of bimanual tasks while also emphasizing the importance of inference speed. In multi-stage tasks, failures or delays at any stage can cascade over time, impacting the success and efficiency of subsequent sub-stages and ultimately hindering overall task performance. In this paper, we propose a novel keypose-conditioned coordination-aware consistency policy tailored for bimanual manipulation. Our framework instantiates hierarchical imitation learning with a high-level keypose predictor and a low-level trajectory generator. The predicted keyposes serve as sub-goals for trajectory generation, indicating targets for individual sub-stages. The trajectory generator is formulated as a consistency model, generating action sequences based on historical observations and predicted keyposes in a single inference step. In particular, we devise an innovative approach for identifying bimanual keyposes, considering both robot-centric action features and task-centric operation styles. Simulation and real-world experiments illustrate that our approach significantly outperforms baseline methods in terms of success rates and operational efficiency. Implementation codes can be found at https://github.com/JoanaHXU/BiKC-plus.