Supervised Mixture-of-Experts for Surgical Grasping and Retraction

AI summary

Overview

• Research area: Surgical robotics, imitation learning, and robot policy architecture design for minimally-invasive surgery (MIS).
• Technical level: Intermediate to Advanced — assumes familiarity with imitation learning, action chunking (ACT), Vision-Language-Action models, and mixture-of-experts gating.
• Scope: The paper introduces a supervised Mixture-of-Experts (MoE) extension to action transformer policies and evaluates it on a laparoscopic bowel grasping and retraction task against ACT, SmolVLA, and π0.5 baselines.

What This Paper Is About

Learning autonomous surgical manipulation is hard because demonstration data is scarce, the workspace is cramped, tissue deforms, and policies must be safe and predictable. The authors add a supervised Mixture-of-Experts layer on top of a lightweight Action Chunking Transformer (ACT) so that each task phase (idle, grasp, hold, retract, maintain tension) is handled by a dedicated expert, then test whether this lets a vision-only policy learn long-horizon surgical manipulation from fewer than 150 demonstrations.

Key Contributions

1. A supervised MoE architecture for phase-structured surgical tasks that can be added on top of any action transformer policy. Applied to ACT, it learns from far fewer demonstrations than prior work (SRT-H's gallbladder clipping and cutting required approximately 16,000 demonstrations) while using only stereo endoscopic images — no wrist cameras or multi-view setups.
2. A new surgeon-robot collaboration task: laparoscopic bowel retraction, where a human surgeon gives high-level visual cues with a laparoscopic instrument while the robot performs precise grasping, pulling, and sustained retraction.
3. An empirical demonstration that state-of-the-art VLAs fail in this domain: both π0.5 (4 B parameters) and SmolVLA (0.24 B) achieve 0/20 end-to-end success in-distribution, while ACT+MoE (53.3 M parameters) reaches 17/20.
4. Two prerequisites for in vivo translation: viewpoint invariance (82% success, 18/22, on unseen camera angles after training on randomized viewpoints) and zero-shot transfer to ex vivo porcine tissue (80% success, 12/15) despite training only on phantom data. Preliminary qualitative in vivo porcine roll-outs are also shown.

Main Findings

• In-distribution performance: ACT+MoE reached 20/20 reaching, 20/20 grasping, 19/20 retracting, and 17/20 end-to-end, versus standard ACT at 16/20, 13/20, 12/20, and 10/20. Grasping improved from 60% to 85% and end-to-end from 50% to 85%, described as a 70% relative improvement over ACT.
• VLA models fail entirely: SmolVLA scored 5/20 reaching, 2/20 grasping, 0/20 retracting, 0/20 end-to-end. π0.5 scored 2/20, 4/20, 0/20, 0/20. SmolVLA produced "erratic and dangerous actions," and π0.5 violated task phase constraints (retracting before securing a grasp).
• Statistical significance: ACT+MoE over ACT: grasping p=0.008, retracting p=0.020, end-to-end p=0.041 (two-sided Fisher's exact test). Over both π0.5 and SmolVLA: p<10⁻⁷ on all reported metrics.
• Inference speed: ACT+MoE runs at 27 Hz, the same as standard ACT, with negligible overhead. π0.5 runs at 10 Hz (compiled), SmolVLA at 3.3 Hz. Training took 3 hours for ACT and ACT+MoE, 8 hours for π0.5, and 14 hours for SmolVLA.
• Out-of-distribution robustness: ACT+MoE reached 19/20 reaching, 16/20 grasping, 13/20 retracting, 13/20 end-to-end; ACT reached 16/20, 13/20, 6/20, 6/20. p-values were Reaching p=0.342, Grasping p=0.480, Retracting p=0.056, End-To-End p=0.056.
• Zero-shot ex vivo transfer: The MoE-ACT policy achieved 80% (12/15) on ex vivo porcine bowel. Of the three failures, two grasped two bowels simultaneously but still completed retraction; only one was a complete failure.
• Unseen viewpoints: Retrained on the combined fixed and random-viewpoint datasets, the policy achieved 82% (18/22) on unseen testing viewpoints.
• Data efficiency ablation: Tested at 100% (120 episodes), 50% (60 episodes), and 25% (30 episodes) with a 10% validation split. At 25%, both policies performed identically at 45% success, suggesting 30 demonstrations is a lower bound. At higher data availability, ACT plateaued at 50% while MoE scaled from 60% to 85%.
• Gating network quality: The phase classifier confusion matrix showed high classification accuracy on validation data, and expert utilization rates closely mirrored the phase frequency of the training dataset — evidence against mode collapse or expert underutilization.
• Saliency behavior: AblationCAM heatmaps showed phase-dependent attention: the robot instrument early in episodes, the surgeon-indicated target during cueing, and the bowel segment between grasp points during retraction.

Methodology in Plain English

The team built a setup using an OpenHELP open-body phantom with two UR5e robotic arms: one holds a static stereo TIPCAM1 S 3D endoscope (Karl Storz SE & Co. KG) for vision, the other holds a laparoscopic bowel grasper through a mechatronic interface and moves under a remote-center-of-motion constraint, controlled by joystick.

They recorded 120 episodes from a fixed endoscopic viewpoint ("fixed-viewpoint dataset") and 50 episodes with randomized camera angles ("random-viewpoint dataset"). The task is split into H=5 phases — Idle, Approach & Grasp, Hold, Retract, and Maintain Tension — labeled automatically using gripper state and instrument movement magnitude. The policy sees only the stereo image pair at the current time; proprioceptive data is deliberately excluded, and the action space is a chunk of 3D delta movements plus binary gripper actions.

Architecturally, they keep ACT's variational backbone but replace the single action head with H action experts, H gripper experts, and a gating network. Because training MoEs end-to-end is unstable, they supervise the gating network directly with phase labels. Final actions are phase-weighted mixtures of expert outputs. The training objective combines four terms: an L1 action reconstruction loss, a cross-entropy phase loss on the gating network, a binary cross-entropy gripper loss, and a KL term regularizing the posterior encoder.

Evaluation used two trained medical students and one surgical resident in a single-blinded process, labeling the final frame as success or failure by majority vote. A success means tissue grasped by two graspers and retracted with sufficient tension. In-distribution testing divided the field of view into 12 angular sectors with 20 starting positions paired with one of eight grasping points. Out-of-distribution tests covered novel grasp locations, reduced illumination, partial occlusion from phantom fat, and a different camera angle, with five trials per perturbation.

Why This Matters

• Research impact: The results reinforce recent findings that generalist foundation models fail on high-precision surgical tasks in data-scarce regimes, and show that explicit phase supervision can stabilize MoE training — a known weak point of end-to-end MoE optimization.
• Real-world applications:
  • Autonomous surgical assistants that hold and retract tissue so the operating surgeon can focus on critical steps like suturing or bowel anastomosis.
  • Bowel manipulation during minimally-invasive gastrointestinal cancer surgery, where assistants must expose jejunum and ileum for anastomosis.
  • Deployment on resource-constrained surgical hardware, since ACT+MoE runs at 27 Hz on a single RTXA5000 GPU.
  • Zero-shot transfer to real tissue and varying camera angles, relevant for dynamic clinical environments.
• Industry relevance: Staff shortages relative to growing surgical treatment needs create demand for autonomous intraoperative assistance. The finding that a 53.3 M-parameter policy beats 4 B and 0.24 B VLAs on this task has direct implications for surgical robot manufacturers weighing model size against real-time latency.

Future Directions

• Replacing manual phase supervision with unsupervised methods that discover latent task skills implicitly from demonstration data, removing the current reliance on phase labels.
• Adding real-time depth vision from the stereoscopic endoscope feed to improve 3D spatial understanding, motivated by in vivo failures attributed to the lack of depth during the grasping phase.
• Extending quantitative evaluation to in vivo porcine surgery; the current in vivo evidence is limited to qualitative preliminary roll-outs of two successful rollouts.
• Scaling the demonstrations and testing whether the architecture maintains its advantage as dataset size grows beyond 120 episodes, given that the 25% data regime showed no difference between ACT and ACT+MoE.

Target Audience

Robotics and surgical-robotics researchers working on imitation learning, action chunking, and mixture-of-experts architectures; clinical translation teams evaluating autonomous surgical assistance; and machine learning engineers interested in data-efficient policy design for constrained, safety-critical domains. Readers without background in imitation learning or transformer policies will find the architecture section dense, but the benchmark comparisons and ex vivo/in vivo results are accessible.

Code and dataset are stated to be released upon acceptance, with a project page at https://surgical-moe-project.github.io/rss-paper/.

Authors’ abstract

Imitation learning has achieved remarkable success in robotic manipulation, yet its application to surgical robotics remains challenging due to data scarcity, constrained workspaces, and the need for an exceptional level of safety and predictability. We present a supervised Mixture-of-Experts (MoE) architecture designed for phase-structured surgical manipulation tasks, which can be added on top of any autonomous policy. Unlike prior surgical robot learning approaches that rely on multi-camera setups or thousands of demonstrations, we show that a lightweight action decoder policy like Action Chunking Transformer (ACT) can learn complex, long-horizon manipulation from less than 150 demonstrations using solely stereo endoscopic images, when equipped with our architecture. We evaluate our approach on the collaborative surgical task of bowel grasping and retraction, where a robot assistant interprets visual cues from a human surgeon, executes targeted grasping on deformable tissue, and performs sustained retraction. Our results show that generalist Vision Language Action models fail to acquire the task entirely, even under standard in-distribution conditions. Furthermore, while standard ACT achieves moderate success in-distribution, adopting a supervised MoE architecture significantly boosts its performance, yielding higher success rates in-distribution and demonstrating superior robustness in out-of-distribution scenarios, including novel grasp locations, reduced illumination, and partial occlusions. Notably, it generalizes to unseen testing viewpoints and also transfers zero-shot to ex vivo porcine tissue without additional training, offering a promising pathway toward in vivo deployment. To support this statement, we present qualitative preliminary results of policy roll-outs during in vivo porcine surgery.