The Pulse
SB Intuitions Explores Visual Priors for Robot Action Learning
SB Intuitions is exploring whether visual priors and visual prediction can improve action learning, with robotic manipulation as a downstream testbed. Its Latent-WAM experiments examine attention designs, visual memory and learning from int
AI.info Team · Written with AI · fact-checked ·
SB Intuitions is exploring whether visual priors and prediction can improve action learning, using robotic manipulation as a downstream testbed. The research asks whether scene dynamics and future visual representations can provide useful signals for generating better actions. Its experimental Latent-WAM framework brings visual prediction and action generation together in a shared latent space.
Core contributors are Chen Zhu and Gang Yan, with Shiqi Yang as project lead. The team began by exploring Vision-Language-Action (VLA) and World-Action Model (WAM) approaches, including π₀.₅, GWP and LingBot-VA. It evaluated those approaches on the SBInt/OpenArm platform using internally collected robot demonstrations of a multi-step task: placing objects into a box and closing its lid.
Latent-WAM predicts future visual representations, not future pixels
The model uses a frozen V-JEPA 2 encoder to turn observations into visual tokens, while trainable visual and action experts share self-attention. Instead of reconstructing future frames, Latent-WAM predicts visual representations at 12, 24, 36 and 48 frames ahead. Its predictor and action components contain approximately 145 million parameters.
For pretraining, the team combined AgiBot Alpha, RDT-ALOHA and RoboMIND agilex, for an initial total of 608.8 hours. It later expanded the AgiBot data with AgiBotWorld-Beta, bringing the expanded total to about 2,716.5 hours. AgiBotWorld-Beta already includes Alpha, so the expanded total counts it once. The team also maps robot actions into a shared 16-dimensional representation, with embodiment-specific mappings and validity masks for platform differences.
Bidirectional attention achieves the highest average success rate on three LIBERO benchmarks
Experiments on three LIBERO benchmarks showed that bidirectional attention achieved the highest average success rate, 94.7%, compared with 92.0% for independent branches. The comparison also reported 92.3% for Video → Action and 89.3% for Action → Video. Each suite contained 10 tasks × 10 trials, with seed 0, checkpoint 50k, replan 10 and wait 30.
The difference was particularly noticeable on LIBERO Goal: bidirectional interaction achieved 93% success, compared with 86% when only actions could read visual predictions. In the Action → Video configuration, future visual queries read action tokens, while action generation does not depend on future visual queries. Although bidirectional attention achieved the highest success rate, the team adopted Action → Visual to prioritize inference efficiency; this allows it to skip the future-prediction branch during inference.
Exploring visual memory for action prediction
The team experimented with visual memory on RememberColor3, where a robot must retain an earlier color cue and later use it to select an object. It compared full-history memory, which can attend to earlier visual features, with compact memory, which retrieves relevant history and combines it with current features through a learned gate. The experiments also showed that recognizing the cue did not necessarily mean the policy selected the correct target.
With the updated failure-case expert-data and DiffusionNFT training recipe, RememberColor3 success improved from 68.5% to 86.0% on the same 200-episode seed block. Under that recipe, full-history memory achieved 86.0%, compared with 81.0% for compact memory. The improvement belongs to the updated failure-case expert-data and NFT retraining recipe.
In RoboTwin, click-bell training-rollout success increased from 68.8% to 96.9% within just a few training rounds. The turn-switch task struggled to improve consistently with 64 rollouts per round; after the rollout budget increased to 512 per round, success improved to 80.5%, with a peak of 84.0%. These were training-rollout results rather than independent held-out evaluations, and the scaled turn-switch run was still in progress.
Initial RoboTTT experiments have not yet demonstrated consistent improvements in task success
SB Intuitions developed an initial test-time training prototype called RoboTTT. After each action chunk, the system compares predicted future visual representations with observations that actually arrive and uses the difference as a self-supervised learning signal. Initial experiments have not yet demonstrated consistent improvements in task success.