When would Vision-Proprioception Policies Fail in Robotic Manipulation?

AI summary

Overview

• Research area: Robot learning for manipulation — specifically multimodal behavior-cloning policies that combine vision with proprioception (robot internal state), and the optimization dynamics that govern how those two modalities are used.
• Technical level: Advanced. The paper moves between an optimization-level analysis of gradient flow through separate modality encoders and a practical training-time algorithm, and assumes familiarity with behavior cloning, gradient descent, and VLA models.
• Scope in one sentence: The paper identifies when vision-proprioception manipulation policies fail (during "motion-transition" phases where the robot must relocate a target), explains the failure as proprioception dominating the gradient signal, and proposes a gradient-adjustment algorithm, GAP, to fix it.

What This Paper Is About

Prior work disagrees about whether adding proprioception to a vision-based manipulation policy helps or hurts — HPT reported clear improvements, while Octo observed that policies trained with additional proprioception seemed generally worse than vision-only policies. This paper asks when vision-proprioception policies fail, and finds through temporally controlled intervention experiments that the visual modality of such a policy plays a limited role precisely during the sub-phases where the robot's motion transitions and target localization is required. The goal is to remove that suppression so the two modalities collaborate dynamically rather than one crowding out the other.

Key Contributions

1. A temporally controlled diagnosis of the failure mode. The authors run an intervention experiment in simulation in which the "assembly" task is executed by a vision-only policy, but for a specific 10-timestep period the executed actions are replaced with those predicted by a vision-proprioception policy under the same observations. The intervention has minimal impact during motion-consistent phases such as "move forward" but produces noticeable degradation during motion-transition phases such as "locate base" and "assemble them," indicating the vision modality fails to take effect there.
2. An optimization-level explanation. The paper formalizes the vision-proprioception policy under behavior cloning with separate vision and proprioception feature chunks whose features are concatenated, and shows that because visual cues during transitions can differ only at the pixel level while proprioceptive signals directly encode gripper state, the policy gravitates toward proprioception for faster loss reduction, dominating optimization and suppressing learning of the vision chunk.
3. The GAP algorithm. Gradient Adjustment with Phase-guidance defines a motion representation from proprioception (gripper position, orientation, opening degree), segments trajectories into motion-consistent phases using Change Point Detection, then uses a temporal network (LSTM) on proprioceptive temporal differences to predict a per-timestep probability of belonging to a motion-transition phase. During policy learning, that probability modulates the magnitude of proprioception's backward gradient.
4. A broad empirical validation. GAP is tested in Meta-World and RoboSuite, on real one-arm and dual-arm setups, against three baseline approaches (MS-Bot, auxiliary loss, masking), across three fusion approaches (concatenation, summation, FiLM), and on the Vision-Language-Action model Octo, reporting consistent gains.

Main Findings

• Vision-proprioception policies underperform vision-only policies overall. Figure 1 reports vision-proprioception policies performing 15.8% worse than vision-only policies.
• The failure is phase-specific, not global. Switching a vision-only rollout to vision-proprioception actions had minimal impact during motion-consistent phases and noticeable degradation during motion-transition phases.
• Proprioception dominates the optimization. Because visual cues during transitions can be subtle (pixel-level) while proprioceptive signals concisely encode gripper opening degree and pose, the policy reduces training loss faster via proprioception, which suppresses learning of the vision chunk.
• GAP outperforms vision-only and all three baselines in simulation. Example results (success rate, averaged over 5 seeds): Meta-World "pick-place" — Vision-only 91.8, Concatenation 78.4, MS-Bot 90.8, Aux 88.2, Mask 86.4, GAP 94.2; Meta-World "assembly" — 82.6 / 74.6 / 91.0 / 90.4 / 89.2 / 94.2; Meta-World "push-wall" — 63.2 / 54.4 / 66.2 / 51.0 / 79.2 / 73.6; RoboSuite "threading" — 43.6 / 33.2 / 51.2 / 45.4 / 47.2 / 53.0.
• GAP also leads in real-world rollouts (out of 20). Real one-arm "press button" — Vision-only 18/20, Concatenation 12/20, MS-Bot 20/20, Aux 19/20, Mask 18/20, GAP 20/20. Real dual-arm "lift lid and pour" — 9/20 / 5/20 / 10/20 / 8/20 / 7/20 / 15/20.
• GAP improves out-of-distribution generalization. In scenarios where the initial distribution of object positions differs from the training data: Meta-World "assembly" — Vision-only 78%, Concatenation 62%, GAP 88%; RoboSuite "threading" — 32% / 28% / 49%; Real one-arm "cube" — 12/20 / 7/20 / 15/20; Real dual-arm "handover" — 12/20 / 9/20 / 15/20.
• GAP transfers to a Vision-Language-Action model. Fine-tuned Octo with vision only (Octo-V) versus vision plus proprioception (Octo-VP): "disassemble" 95% vs 82%; "push-wall" 77% vs 65%; "put hammer into drawer" 92% vs 88%; "threading" 69% vs 57%. With GAP applied (Octo-VP†): 100%, 85%, 97%, 78% — an average improvement of 17% over Octo-VP.
• GAP works across fusion approaches. Applied to Concatenation, Summation and FiLM, GAP-equipped variants outperform the vision-only policy in all three cases (e.g. Meta-World "assembly": Vision-only 82%, Concatenation† 96%, Summation† 97%, FiLM† 94%).
• The estimated transition probability aligns with visual uncertainty. Using R3M features and local entropy over a sliding window, the authors observe that increases in the predicted transition indicator ρ are consistently accompanied by decreases in scaled visual uncertainty, indicating visual input changes are relatively subtle during transitions.
• Loss curves behave as predicted. GAP causes loss to decrease more slowly during the early and middle stages of training, but the policy ultimately converges to a lower loss.

Methodology in Plain English

The authors begin with a diagnosis. They run a vision-only policy and, for a short window of 10 timesteps, swap in actions from a vision-proprioception policy fed the same observations. If the swap does no harm, that period did not need vision; if it hurts, vision was silently being relied upon and the proprioceptive policy had failed to learn it. This isolates when the problem occurs.

They then explain the phenomenon. The policy has two separate feature extractors — one for images, one for proprioceptive state — whose outputs are concatenated before the action head. Because gradient descent simply follows whatever reduces the loss fastest, and proprioception is a low-dimensional, direct readout of gripper pose and opening, the proprioceptive pathway absorbs most of the learning signal, leaving the visual pathway under-trained.

The fix, GAP, works in two stages. First, it defines a compact motion representation for each timestep from proprioception — change in gripper 3D position, change in orientation, and change in opening degree — and uses Change Point Detection with a custom distance to cut each demonstration trajectory into "motion-consistent" phases. The gaps between those phases are where motion transitions happen. Because transitions are continuous rather than discrete, the authors feed the temporal differences of proprioceptive signals into an LSTM to predict a per-timestep probability ρ of being in a transition, supervising it with the change-point labels while softening the penalty near transitions. Second, during training, the proprioceptive encoder's gradient is scaled down by (1 − ρ) with a coefficient λ, so the more likely a timestep is to be a transition, the more the proprioceptive pathway is held back and the more relative weight the visual pathway receives. The adjustment is applied only in the early stage of learning (for example, the first 50 epochs), and only to the proprioception feature extractor for transformer-based Octo.

Experimental setup: Meta-World tasks use a 4-dimensional action space (gripper position and opening degree) with 100 hard-coded expert demonstrations; RoboSuite tasks are longer-horizon with richer physics and an action space that also includes gripper orientation, using 500 synthesized trajectories. Policies use a ResNet-18 vision backbone projecting to a 512-dimensional representation, a 4-layer temporal transformer with hidden dimension 256, a 3-layer MLP for proprioception, concatenation fusion, and a 3-layer MLP head predicting an action sequence of length 9; observation history window is 5, learning rate 3e-4 with Adam, batch size 128, trained for 100 epochs on a single NVIDIA RTX 3090 GPU. Simulation tasks are evaluated with 100 rollouts and real-world tasks with 20 rollouts.

Why This Matters

The paper reframes an inconsistency in the literature — some works finding proprioception helps, others finding it hurts — as a temporal and optimization phenomenon rather than a data or architecture problem. That gives the field a concrete, measurable target: rather than debating whether to include proprioception, practitioners can ask when during a trajectory it should be allowed to dominate learning. Because GAP is a training-time gradient modification and not a new architecture, it is compatible with MLP-based, diffusion-based and transformer-based policies and with VLA models such as Octo, which lowers the cost of adoption.

Real-world applications:

• Industrial assembly and disassembly, where the initial position of a part varies and the arm must relocate it before inserting — exactly the motion-transition regime the paper targets.
• Bin picking and logistics, where clutter and pose variation demand visual localization between grasp attempts.
• Dual-arm service or household robotics, such as handover and putting objects into containers, evaluated here on the Cobot Magic platform.
• Precision tasks like threading, where the paper notes vision-only policies fail drastically and proprioception is needed for precision — so a policy must use both well rather than swinging to one extreme.

Industry relevance: the algorithm is a drop-in training change, validated on a 6-DoF xArm 6 with a Robotiq gripper and the open-source Cobot Magic dual-arm platform, and on open-source VLA models, which makes it directly usable by teams already training behavior-cloning policies from teleoperated demonstrations.

Future Directions

• Cross-embodiment proprioception. The stated limitation is that all vision-proprioception policies in this work are trained on a single embodiment; exploring the role of proprioception in cross-embodiment datasets is flagged as promising future research.
• Replacing hand-crafted motion representations. The motion distance in Equation 4 uses weighting factors α and β to balance orientation and opening-degree components; a learned motion representation could remove that tuning.
• Going beyond binary phase structure. The ablations compare human labeling, HDBSCAN clustering and CoTPC, and find automated CPD-based estimation generally outperforms human annotation; how finely transitions should be resolved, and whether more than two phase types are useful, remains open.
• Extending beyond early-stage adjustment. GAP is applied only during the early stage of learning (for example the first 50 epochs) and, for Octo, only to the pre-transformer proprioception feature extractor — whether other schedules or more deeply integrated transformer layers would benefit is untested.

Target Audience

Robotics and embodied-AI researchers working on multimodal manipulation policies, especially those training behavior-cloning or Vision-Language-Action models from demonstrations and wrestling with whether and how to fuse proprioception. It is also useful for practitioners deploying manipulation arms in industrial or service settings where object positions vary between runs, and for graduate students looking for a worked example of diagnosing a multimodal learning failure through controlled intervention rather than architectural speculation.

Authors’ abstract

Proprioceptive information is critical for precise servo control by providing real-time robotic states. Its collaboration with vision is highly expected to enhance performances of the manipulation policy in complex tasks. However, recent studies have reported inconsistent observations on the generalization of vision-proprioception policies. In this work, we investigate this by conducting temporally controlled experiments. We found that during task sub-phases that robot's motion transitions, which require target localization, the vision modality of the vision-proprioception policy plays a limited role. Further analysis reveals that the policy naturally gravitates toward concise proprioceptive signals that offer faster loss reduction when training, thereby dominating the optimization and suppressing the learning of the visual modality during motion-transition phases. To alleviate this, we propose the Gradient Adjustment with Phase-guidance (GAP) algorithm that adaptively modulates the optimization of proprioception, enabling dynamic collaboration within the vision-proprioception policy. Specifically, we leverage proprioception to capture robotic states and estimate the probability of each timestep in the trajectory belonging to motion-transition phases. During policy learning, we apply fine-grained adjustment that reduces the magnitude of proprioception's gradient based on estimated probabilities, leading to robust and generalizable vision-proprioception policies. The comprehensive experiments demonstrate GAP is applicable in both simulated and real-world environments, across one-arm and dual-arm setups, and compatible with both conventional and Vision-Language-Action models. We believe this work can offer valuable insights into the development of vision-proprioception policies in robotic manipulation.