GeCo-SRT: Geometry-aware Continual Adaptation for Robotic Cross-Task Sim-to-Real Transfer

AI summary

Overview

Research area: Robotics — sim-to-real transfer, continual learning, and manipulation policy learning using 3D point-cloud representations.

Technical level: Advanced. The paper assumes familiarity with diffusion policies, mixture-of-experts architectures, experience replay, and sim-to-real transfer literature.

Scope: The paper proposes GeCo-SRT, a method that accumulates transferable geometric knowledge across a sequence of sim-to-real robotic manipulation tasks so that each new task is adapted faster and with less real-world data.

What This Paper Is About

Robots trained in simulation lose performance when deployed in the real world because simulated visuals and physics do not match reality (the "sim-to-real gap"). Existing methods treat every new task's transfer as an isolated problem, requiring costly repeated tuning and discarding experience gained from earlier transfers. This paper builds a continual cross-task sim-to-real paradigm, where knowledge from prior transfers (for example, "pick cube") is retained and reused to adapt quickly to novel tasks (for example, "stack cube").

Key Contributions

1. A continual cross-task sim-to-real transfer paradigm. Instead of treating each sim-to-real transfer as isolated, the paper formulates transfer as a sequence of tasks (Task 1 … N) in which knowledge is accumulated iteratively and reused for new tasks.

2. A geometry-aware mixture-of-experts perception residual, Geo-MoE. The module extracts local geometric features (planarity, linearity, saliency) from point clouds and uses them to dynamically route inputs to specialized experts, forming a corrective residual concatenated with the frozen base encoder's output. Local geometry is chosen for its claimed dual invariance: domain invariance (geometry is consistent between simulation and reality, unlike textures) and task invariance (primitives such as edges and corners are shared across manipulation tasks).

3. A geometry-expert-guided prioritized experience replay strategy, Geo-PER. It shifts the sampling priority metric from task loss to expert utilization, so historical samples that strongly activate underused experts are replayed more often, refreshing idle experts and countering catastrophic forgetting.

4. An empirical benchmark and evaluation protocol. A sequence of four simulated and real manipulation tasks (Pick Cube, Stack Cube, Pick Banana, Plug Insert) with a human-in-the-loop correction pipeline, evaluated with Success Rate (SR) and a newly proposed Normalized Negative Backward Transfer (N-NBT) metric for measuring relative forgetting.

Main Findings

• Single-task transfer: Geo-MoE achieves a 50.0% average success rate, versus 38.3% for Transic, 9.2% for Action Residual, and 3.1% for Direct Deploy (which confirms a substantial sim-to-real gap). Per-task Geo-MoE results were 80.0% (Pick Cube), 43.3% (Stack Cube), 40.0% (Pick Banana), and 36.7% (Plug Insert).

• Continual transfer: GeCo-SRT achieves the best average success rate at 63.3% and the lowest average N-NBT at 26.5% across the task sequence Pick Cube → Stack Cube → Pick Banana → Plug Insert. The final-task success rate is 53.3%, compared with 43.3% (Geo-MoE + PER), 33.3% (Transic + PER), 16.7% (Geo-MoE + EWC), 3.3% (Naive Fine-tuning), and 0.0% (Direct Deploy).

• Reported headline gains: The abstract and conclusion report a 52% average performance improvement over the baseline, and the ability to match the baseline's success rate using only 16.7% (1/6) of the data.

• Architecture matters more than the replay variant alone: Replacing Transic with Geo-MoE under standard PER raises average success from 40.0% to 55.7% and lowers average N-NBT from 55.0% to 29.6%, suggesting the geometry-routed expert structure is inherently better suited to continual settings.

• Forgetting is severe without continual mechanisms: Naive fine-tuning reaches 9.2% average success with 75.0% average N-NBT, indicating catastrophic forgetting as new task data overwrites prior knowledge.

• Ablation shows complementary components: With neither observation residual nor MoE, average success is 3.3% (N-NBT 75.0%). Adding MoE alone gives 9.2% (N-NBT 65.5%) — MoE without informative features is ineffective. Adding the observation residual alone gives 45.8% (N-NBT 37.0%). Combining both gives 55.8% average success and 29.6% average N-NBT, which the paper reports as the best configuration.

• Improved sample efficiency: In the data-efficiency study using 20, 40, and 60 correction trajectories, continual learning on Pick Cube reaches 76.6% success with only 20 trajectories, nearly matching the from-scratch model trained with 60 trajectories. For Plug Insert, the method with 20 trajectories reaches a level the from-scratch model requires more than 60 trajectories to reach.

• Task similarity governs transfer direction: With only 10 correction trajectories, learning Stack Cube transfers positively from Pick Cube (40.0% SR) versus from scratch (26.7%), but negatively from PlugInsert (16.7%). For PlugInsert, transfer from StackCube yields 40.0% and from PickBanana yields 30.0%, compared with 23.3% from scratch; the paper attributes the PickBanana benefit to the shared challenge of grasping non-cuboid objects.

Methodology in Plain English

The pipeline works in stages. For each task, the researchers train a base diffusion policy in simulation using behavior cloning on expert trajectories, with 3D point clouds (not RGB) as the observation modality. Point clouds from two cameras are transformed into the robot's base frame, merged, cropped to remove background, and downsampled with Farthest Point Sampling. This mirrors the real-robot sensing setup to reduce the gap from the start.

The trained base policy is then deployed on real hardware under a shared-autonomy scheme. When a human operator anticipates failure, they intervene with a teleoperated action via a 3Dconnexion SpaceMouse and an intervention indicator is recorded; otherwise the policy acts on its own. These correction trajectories are merged with simulation trajectories into a replay buffer.

While the base policy's parameters stay frozen, a shared perception residual module is updated across all tasks. This module, Geo-MoE, groups nearby points with k-nearest neighbors, estimates local geometry via local PCA, and passes each group through a softmax gating network over M parallel experts. The weighted expert outputs form a residual vector that is concatenated with the frozen encoder's features before the diffusion policy head predicts the final action. Training minimizes an action MSE loss plus a balance loss that prevents gating collapse.

Because sequential training can erase expert knowledge, Geo-PER changes what gets replayed. Each stored sample keeps its expert activation vector. For a new task, the system computes average expert utilization and assigns sampling priority inversely proportional to that utilization — samples that fired an idle expert are replayed more. This keeps all experts in the gradient updates even when the current task does not use them. Evaluation uses Success Rate for forward transfer and Normalized Negative Backward Transfer for forgetting, with 30 trials per task.

Why This Matters

Impact on research. The paper reframes sim-to-real transfer as a continual process rather than a series of one-off engineering efforts, and proposes local geometric features as the shared medium that makes low-cost simulation data reusable across tasks. It also introduces N-NBT as a replacement for standard Negative Backward Transfer, which the authors argue misreports forgetting when initial success rates are low — as is common in sim-to-real settings.

Real-world applications (drawn from the tasks and hardware used in the paper):

• Industrial pick-and-place and stacking of objects on a robotic arm line.
• Electronic or mechanical assembly tasks involving insertion, such as the Plug Insert task.
• Food handling and grasping of non-cuboid, irregular objects, such as the Pick Banana task.
• Servicing of deployed low-cost robot arms that must be adapted to new objects without large new data-collection campaigns.

Industry relevance. The reported data efficiency (adapting with 16.7% of the data, or as few as 20 correction trajectories) targets the main cost driver in real deployments: the labor and time required to collect demonstration and correction data on physical hardware. A method that reuses prior deployments' experience can shorten commissioning for each new task on the same robot fleet.

Future Directions

• Non-geometric gaps. The authors state that the method focuses primarily on bridging the observation gap through local geometric features, which may limit applicability to sim-to-real gaps that are not geometric, such as complex dynamics.
• Additional modalities. The paper suggests exploring multiple modalities, including semantic relationships between tasks, as a route to more effective cross-task sim-to-real transfer.
• Broader validation. The evaluation covers four tasks in one sequence order with a fixed sim-to-real pipeline, leaving open whether the geometry-expert and replay mechanisms generalize to other robots, sensing setups, and longer task sequences.
• Scaling the knowledge base. The replay buffer grows with each task; how Geo-PER's expert-utilization prioritization behaves as the number of tasks and experts increases is not addressed.

Target Audience

Robotics researchers and graduate students working on sim-to-real transfer, imitation learning, and continual learning for manipulation; engineers deploying learned policies on real robotic arms who need to reduce per-task data collection; and practitioners interested in 3D point-cloud representations or mixture-of-experts architectures for control. Readers without background in diffusion policies or continual learning will need to consult the cited prior work for context.

Authors’ abstract

Bridging the sim-to-real gap is important for applying low-cost simulation data to real-world robotic systems. However, previous methods are severely limited by treating each transfer as an isolated endeavor, demanding repeated, costly tuning and wasting prior transfer experience.To move beyond isolated sim-to-real, we build a continual cross-task sim-to-real transfer paradigm centered on knowledge accumulation across iterative transfers, thereby enabling effective and efficient adaptation to novel tasks. Thus, we propose GeCo-SRT, a geometry-aware continual adaptation method. It utilizes domain-invariant and task-invariant knowledge from local geometric features as a transferable foundation to accelerate adaptation during subsequent sim-to-real transfers. This method starts with a geometry-aware mixture-of-experts module, which dynamically activates experts to specialize in distinct geometric knowledge to bridge observation sim-to-real gap. Further, the geometry-expert-guided prioritized experience replay module preferentially samples from underutilized experts, refreshing specialized knowledge to combat forgetting and maintain robust cross-task performance. Leveraging knowledge accumulated during iterative transfer, GeCo-SRT method not only achieves 52% average performance improvement over the baseline, but also demonstrates significant data efficiency for new task adaptation with only 1/6 data.We hope this work inspires approaches for efficient, low-cost cross-task sim-to-real transfer.