# SB Intuitions Explores Visual Priors for Robot Action Learning

Published 2026-11-09 · AI.info Team · AI.info

> SB Intuitions is exploring whether visual priors and visual prediction can improve action learning, with robotic manipulation as a downstream testbed. Its Latent-WAM experiments examine attention designs, visual memory and learning from interaction.

[SB Intuitions](https://www.sbintuitions.co.jp/blog/entry/2026/11/09/170000) is exploring whether visual priors and prediction can improve action learning, using robotic manipulation as a downstream testbed. The research asks whether scene dynamics and future visual representations can provide useful signals for generating better actions. Its experimental Latent-WAM framework brings visual prediction and action generation together in a shared latent space.

Core contributors are Chen Zhu and Gang Yan, with Shiqi Yang as project lead. The team began by exploring Vision-Language-Action (VLA) and World-Action Model (WAM) approaches, including π₀.₅, GWP and LingBot-VA. It evaluated those approaches on the SBInt/OpenArm platform using internally collected robot demonstrations of a multi-step task: placing objects into a box and closing its lid.

## Latent-WAM predicts future visual representations, not future pixels

The model uses a frozen V-JEPA 2 encoder to turn observations into visual tokens, while trainable visual and action experts share self-attention. Instead of reconstructing future frames, Latent-WAM predicts visual representations at 12, 24, 36 and 48 frames ahead. Its predictor and action components contain approximately 145 million parameters.

For pretraining, the team combined AgiBot Alpha, RDT-ALOHA and RoboMIND agilex, for an initial total of 608.8 hours. It later expanded the AgiBot data with AgiBotWorld-Beta, bringing the expanded total to about 2,716.5 hours. AgiBotWorld-Beta already includes Alpha, so the expanded total counts it once. The team also maps robot actions into a shared 16-dimensional representation, with embodiment-specific mappings and validity masks for platform differences.

## Bidirectional attention achieves the highest average success rate on three LIBERO benchmarks

Experiments on three LIBERO benchmarks showed that bidirectional attention achieved the highest average success rate, 94.7%, compared with 92.0% for independent branches. The comparison also reported 92.3% for Video → Action and 89.3% for Action → Video. Each suite contained 10 tasks × 10 trials, with seed 0, checkpoint 50k, replan 10 and wait 30.

The difference was particularly noticeable on LIBERO Goal: bidirectional interaction achieved 93% success, compared with 86% when only actions could read visual predictions. In the Action → Video configuration, future visual queries read action tokens, while action generation does not depend on future visual queries. Although bidirectional attention achieved the highest success rate, the team adopted Action → Visual to prioritize inference efficiency; this allows it to skip the future-prediction branch during inference.

## Exploring visual memory for action prediction

The team experimented with visual memory on RememberColor3, where a robot must retain an earlier color cue and later use it to select an object. It compared full-history memory, which can attend to earlier visual features, with compact memory, which retrieves relevant history and combines it with current features through a learned gate. The experiments also showed that recognizing the cue did not necessarily mean the policy selected the correct target.

With the updated failure-case expert-data and DiffusionNFT training recipe, RememberColor3 success improved from 68.5% to 86.0% on the same 200-episode seed block. Under that recipe, full-history memory achieved 86.0%, compared with 81.0% for compact memory. The improvement belongs to the updated failure-case expert-data and NFT retraining recipe.

In RoboTwin, click-bell training-rollout success increased from 68.8% to 96.9% within just a few training rounds. The turn-switch task struggled to improve consistently with 64 rollouts per round; after the rollout budget increased to 512 per round, success improved to 80.5%, with a peak of 84.0%. These were training-rollout results rather than independent held-out evaluations, and the scaled turn-switch run was still in progress.

## Initial RoboTTT experiments have not yet demonstrated consistent improvements in task success

SB Intuitions developed an initial test-time training prototype called RoboTTT. After each action chunk, the system compares predicted future visual representations with observations that actually arrive and uses the difference as a self-supervised learning signal. Initial experiments have not yet demonstrated consistent improvements in task success.

## Sources

1. Original source · Blog post [SB Intuitions TECH BLOG](https://www.sbintuitions.co.jp/blog/entry/2026/11/09/170000) sbintuitions.co.jp

## Related stories

- [Kyutai Releases REKI, a 2.4B Robot-Action Model](https://ai.info/news/kyutai-releases-reki-robot-action-model)
- [Artificial Analysis Puts Frontier Models on a Real Robot Arm](https://ai.info/news/artificial-analysis-aa-robotics-real-robot-arm-benchmark)
- [Reka Puts Text, Video and Robot Control in 19B Rho-1](https://ai.info/news/reka-rho-1-19b-omni-model)
- [GPT-6 Astra Tops 40 Robot Policies, but Precision Still Fails](https://ai.info/news/gpt-6-astra-tops-40-robot-policies-but-precision-still-fails)
- [Black Forest Labs Releases 7B FLUX 3 Action Robotics Models](https://ai.info/news/black-forest-labs-flux-3-action-robotics-models)

## Explore

[More articles](https://ai.info/news)

---

Canonical: https://ai.info/news/sb-intuitions-visual-prediction-robot-actions
