Bridging Embodiment Gaps: Deploying Vision-Language-Action Models on Soft Robots

AI summary

Overview

Research area: Robotics, specifically Vision-Language-Action (VLA) models, soft continuum manipulators, and cross-embodiment policy transfer for human-robot interaction.

Technical level: Advanced. The paper assumes familiarity with VLA architectures (OpenVLA-OFT, π₀), robot kinematics, teleoperation, and large-model finetuning workflows.

Scope: A single study that finetunes two state-of-the-art VLA models on a custom soft continuum robot (Embuddy) and a rigid UR5 baseline across three manipulation tasks, showing that targeted finetuning closes the rigid-to-soft embodiment gap.

What This Paper Is About

VLA models have been proposed as general, language-guided control frameworks for real robots, but their deployment has so far been limited to conventional rigid serial-link manipulators. Soft continuum robots offer intrinsic safety and compliance, which matters for human-shared environments, yet no existing dataset or benchmark covers VLA deployment on these platforms. This paper asks whether VLA policies can be transferred from rigid arms to a soft continuum manipulator, and demonstrates a finetuning pipeline that makes such transfer work.

Key Contributions

1. The authors introduce what they describe as the first open-source dataset of soft robot demonstrations, released on HuggingFace (HCSuMoss/soft_orange and HCSuMoss/soft_feed), enabling reproducible research on compliant embodiments.
2. They benchmark OpenVLA-OFT on both a rigid UR5 and the soft robot Embuddy, showing that finetuning closes the rigid-to-soft domain gap and yields comparable task success rates.
3. They compare OpenVLA-OFT and π₀ on the soft robot, finding that while π₀ shows stronger generalization on rigid embodiments, OpenVLA-OFT achieves superior performance on the compliant platform after finetuning.
4. They provide a structured finetuning and deployment pipeline spanning task design, data capture, preprocessing, model adaptation, and inference, including real-time off-board control-loop frequency measurements.

Main Findings

• Out-of-the-box policies fail on the soft robot: Neither vanilla OpenVLA-OFT nor π₀ succeeded without finetuning. The stated cause is the discrepancy between soft- and rigid-robot dynamics, specifically the mapping from end-effector pose to internal configurations; Embuddy consistently got stuck mid-execution when the model generated motions suitable for rigid manipulators but incompatible with its kinematics.
• Finetuning closes the rigid-to-soft gap: After finetuning, OpenVLA-OFT achieved exactly the same success rate on UR5 and Embuddy for Task 1 and Task 2. The specific percentage values are shown in Figure 2; the text reports a 70% success rate for OpenVLA-OFT on the "Put X in the plate" task in the language-grounding analysis.
• Both models work on the soft robot after finetuning: π₀ also reached a high success rate on soft hardware, though slightly lower than OpenVLA-OFT on the soft robot in Task 1 and Task 3.
• Model ranking changes with embodiment: π₀ has better generalization on rigid embodiments, but OpenVLA-OFT outperforms π₀ when transferring to a completely new platform with different dynamics after proper finetuning.
• Control-loop frequency: Even with connection delay, the soft robot achieved at least 25 Hz in the control loop. Table 1 reports 32.3 Hz for UR5 with OpenVLA-OFT on an A100 (Azure VM), 25.1 Hz for Embuddy with OpenVLA-OFT on an H100 (remote cluster), and 38.0 Hz for Embuddy with π₀ on an H100 (remote cluster). All inference used action chunks of size 8.
• Robustness to humans in the scene: Human movement and appearance had zero influence on performance for both UR5 and soft robot experiments, as long as the workspace was not covered or interrupted.
• Sensitivity to object novelty and workspace: With unseen objects the model was confused once in 10 trials. When the tasked object was placed outside the workspace, the model failed all the time, even when the object was only 10 cm away from the region, indicating that the workspace seen in training is a deterministic factor.
• Language grounding: Enabling the FiLM module in Task 2 directed attention to the object named in the instruction rather than selecting arbitrarily. In Task 3, when an orange was placed in the plate instead of marshmallows, both models refrained from executing the pick-and-place action and terminated mid-execution.
• Physical robustness of the soft arm: During Task 3 inference, a trial lasted around 2 to 3 minutes and Embuddy was manually stopped or pushed away twice, each for around 5 seconds, with no observed performance degradation; the robot recovered its pose, continued the correct trajectory, and finished the task.

Methodology in Plain English

The researchers built a pipeline in five stages. First, they defined three tasks: a simple pick-and-place ("Put the orange in the plate"), a choice-based pick-and-place ("Put the X in the plate", where X is orange or milk), and a close human-interactive feeding task ("Feed the person with marshmallow"). Second, they teleoperated both the UR5 and the soft robot Embuddy with a 3dconnexion space mouse joystick, using a Piecewise Constant Curvature model for inverse kinematics on the soft arm, and recorded third-person and wrist camera views, end-effector pose, and language instructions. Third, they cropped and down-sampled images from 640x480 to 256x256, flipped the wrist view, filtered out episodes with almost zero motion, and padded state and action representations, exporting to RLDS format for OpenVLA-OFT and LeRobot format for π₀. Fourth, they finetuned both models: OpenVLA-OFT with LoRA (rank 32, Llama 2 7B backbone, A100 on Azure for UR5 and H100 on a remote HPC cluster for the soft robot) and π₀ with full finetuning (PaliGemma 3B backbone, H100). Fifth, they ran inference with the local PC capturing observations and sending them to a remote GPU, which returned an action chunk for local execution, repeating until the task finished or maximum steps were reached. Performance was measured as success rate over 10 trials.

Dataset sizes were 50 demonstrations for Task 1, 100 for Task 2 (50 for orange, 50 for milk), and 20 for Task 3. Teleoperation capture ran at 5 Hz because of small transformation and rotation gains.

Why This Matters

This work is the first to systematically evaluate VLA models on a soft continuum robot, addressing the embodiment gap that has kept compliant robots out of the VLA literature. It shows that the advanced reasoning of VLA models can be combined with the intrinsic safety of soft robotics, and that a small custom dataset plus targeted finetuning is enough to make the transfer work. For research, it argues that embodiment mismatch — not just data scarcity — is a central obstacle in cross-platform policy transfer, and it supplies an open dataset to study it.

Real-world applications suggested by the paper's framing:

• Human-shared environments: Soft arms that are inherently deformable and stoppable by people, which the authors demonstrate by manually pushing Embuddy mid-task while the policy continued and completed the task.
• Assistive feeding: Task 3 specifically targets feeding a person a marshmallow, a scenario where compliance around a human body reduces injury risk.
• Food handling and pick-and-place: Tasks 1 and 2 involve real food objects (orange, milk, yogurt, baguette), where a compliant arm may be safer around soft or fragile items.
• Collision-tolerant operation: Because the underactuated sections remain deformable regardless of motor positions and the arm weighs 5 kg total, inertial forces are limited and collisions are more forgiving.

Industry relevance centers on combining pretrained generalist VLA policies with inexpensive, lightweight, 3D-printed compliant hardware. The reported off-board control-loop frequencies (32.3 Hz, 25.1 Hz, and 38.0 Hz) show that remote inference over a network is fast enough for a compliant platform to operate, which matters for architectures where the heavy model runs off-robot.

Future Directions

• Expanding the investigation to a wider range of tasks and compliant platforms, which the authors state as their explicit next step.
• Investigating embodiment transfer more broadly: whether the finetuning recipe generalizes beyond this single soft continuum design and its one-plane bending sections.
• Understanding and mitigating the workspace dependence found in the perturbation study, since objects only 10 cm outside the trained region caused total failure.
• Determining what data volume is needed to achieve reliable soft-robot policies, given that the smallest dataset here (20 demonstrations for Task 3) still produced workable finetuning and the largest was 100.

Target Audience

Robotics researchers working on VLA models, cross-embodiment transfer, or soft robotics; engineers deploying learned policies on compliant or non-standard manipulators; and graduate students interested in the practical details of finetuning large models for real hardware, including data formats, hyperparameters, and inference latency. Practitioners evaluating whether soft robots can serve as a safe platform for language-conditioned autonomy will find the benchmark and the open dataset most directly useful.

Authors’ abstract

Robotic systems are increasingly expected to operate in human-centered, unstructured environments where safety, adaptability, and generalization are essential. Vision-Language-Action (VLA) models have been proposed as a language guided generalized control framework for real robots. However, their deployment has been limited to conventional serial link manipulators. Coupled by their rigidity and unpredictability of learning based control, the ability to safely interact with the environment is missing yet critical. In this work, we present the deployment of a VLA model on a soft continuum manipulator to demonstrate autonomous safe human-robot interaction. We present a structured finetuning and deployment pipeline evaluating two state-of-the-art VLA models (OpenVLA-OFT and $\pi_0$) across representative manipulation tasks, and show while out-of-the-box policies fail due to embodiment mismatch, through targeted finetuning the soft robot performs equally to the rigid counterpart. Our findings highlight the necessity of finetuning for bridging embodiment gaps, and demonstrate that coupling VLA models with soft robots enables safe and flexible embodied AI in human-shared environments.