ALPS: Visual State Encoders for World Models

AI summary

Overview

• Research area: Computer vision and representation learning, specifically visual pretraining for world models and robot learning.
• Technical level: Advanced (assumes familiarity with vision transformers, contrastive learning, and latent world models, though the core ideas are described plainly).
• Scope: The paper introduces ALPS, a ViT-B/16 visual encoder pretrained on still images and text that unifies physical-state information, language alignment, and pixel reconstruction in one frozen representation, and evaluates it against fifteen public encoders across state probing, retrieval, reconstruction, control, planning, and prediction.

What This Paper Is About

World models predict how a scene will change in response to actions, and many of them operate on the frozen tokens of a pretrained visual encoder — so that encoder determines what the world model can effectively "see." The authors argue a good visual foundation for such models needs three things at once: tokens that capture physical state (where objects are and how they move), tokens aligned with language so instructions can be grounded, and tokens that can be decoded back into pixels for inspection. Existing encoders typically deliver only one or two of these, so the paper's goal is to pretrain a single encoder that provides all three.

Key Contributions

1. Formulates visual state pretraining for world models and establishes a unified evaluation protocol spanning physical-state information, native language alignment, pixel reconstruction, and downstream control, planning, and prediction.
2. Introduces ALPS (Language-aligned, Pixel-renderable State encoder), a ViT-B/16 trained from scratch on still images and text using joint language, reconstruction, and spatial objectives, followed by a cross-view self-distillation stage.
3. Demonstrates that one frozen representation can be competitive across all three capabilities, achieving the lowest aggregate LIBERO state-probe error among ViT-B encoders while retaining strong language alignment and accurate pixel reconstruction.
4. Shows cross-view refinement is architecture-dependent: applying the same Stage 2 recipe to SigLIP 2 and CLIP improves their state information but costs them 4–12 points on language metrics, whereas ALPS preserves its language alignment.

Main Findings

• Lowest state-probe error among ViT-B encoders: ALPS reaches an aggregate relative LIBERO position error of .829, ahead of C-RADIOv3 (.864), DINOv3 (.870), and PE-Spatial (.936). It matches the 3.5× larger, video-pretrained V-JEPA 2-L (.829), though the two differ in profile: V-JEPA 2-L reads out current state more accurately (.741 vs. .842), while ALPS predicts 3 seconds ahead more accurately (.813 vs. .904).
• Large margin over image-text encoders on state: ALPS reduces aggregate state-probe error by about 29% relative to SigLIP 2 (1.160) and PE-Core (1.170).
• Comparable object-level accuracy on moving objects: ALPS reaches a mean error of 3.22 cm across four prediction settings, within the 3.04–3.25 cm range of V-JEPA 2, DINOv3, and C-RADIOv3.
• Language alignment stays close to the best: 93.2/78.9 Recall@1 on Flickr30k and 70.2/51.9 on COCO (image-to-text/text-to-image), within 2 points of the best score on each measure. ImageNet zero-shot accuracy is 73.4 versus 78.4 for PE-Core, a 5.0-point gap the authors attribute to pretraining on descriptive synthetic captions.
• Reconstruction from frozen tokens: ALPS achieves ℓ1 error .120 and LPIPS .372, lower ℓ1 than every compared encoder not explicitly trained to reconstruct pixels or latents — C-RADIOv3 has 34% higher error (.161), DINOv3 63% higher (.196), and image-text encoders more than twice the error. Reconstruction-focused encoders (MAE .109, VC-1 .106, OpenVision 3 .104) are more accurate but incur 14% to 22% higher state-probe error. With a stronger decoder, ALPS reaches .107 ℓ1 versus C-RADIOv3's .141 and overtakes it on LPIPS (.302 vs. .311).
• Stage 2 drives the state gains: refinement reduces ALPS's aggregate state-probe error by 17% (14% to 19% across six settings) and raises DAVIS video object segmentation from .456 to .533, while retrieval shifts by at most 0.5 points and reconstruction stays near its first-stage level (.120 vs. .112 ℓ1; .372 vs. .365 LPIPS).
• The same recipe damages other encoders' language ability: SigLIP 2 improves on the state probe from 1.160 to .893 but loses 4.3 points of ImageNet accuracy and 7.5 and 11.4 points on Flickr30k and COCO image-to-text retrieval; CLIP improves from 1.194 to .898 but loses 10.2, 10.9, and 11.8 points. Lowering SigLIP 2's learning rate to 3×10⁻⁶ still produces a 17% state gain but retains a language loss (2.3 points on Flickr30k, 2.9 on COCO, versus 0.2 and 0.4 for ALPS).
• Best mean LIBERO policy success: with frozen tokens and BERT instructions, ALPS reaches 83.4% mean success versus 81.4% for SigLIP 2, 79.0% for DINOv3, 78.4% for DINOv2, 78.2% for C-RADIOv3, and 75.0% for CLIP. Switching to each encoder's native text tower raises ALPS to 85.5%, the highest in the comparison (SigLIP 2 84.8%, C-RADIOv3 81.8%, CLIP 79.2%). Stage 2 alone lifts LIBERO-Spatial from 76.8% to 86.5%.
• Leads or ties world-model planning: 90% success on PushT (tied with C-RADIOv3) and 92% on Wall (tied with DINOv2); all six encoders solve every PointMaze goal. On Granular, refinement lowers Chamfer distance from .324 to .176. C-RADIOv3 is best on Rope (.558 vs. .750) and DINOv3 is best on Reacher (86.0% vs. 78.7%).
• More accurate predicted tokens: Stage 2 reduces five-step prediction error by 28% on PushT, 17% on Reacher, 13% on Granular, and 11% on Rope.
• Predicted futures can be rendered: on PushT, five-step open-loop rendered rollouts give ALPS an LPIPS of .020, versus DINOv3 .024, Stage 1 .025, SigLIP 2 .031, and C-RADIOv3 .018.
• Objective trade-offs: at one tenth of the Stage 1 schedule, removing the reconstruction loss raises ℓ1 reconstruction error by 72% and 78% (.194 and .200 vs. .113), while removing reconstruction or captioning lowers state-probe error by 8.5% and 4.3% respectively — a capacity trade-off the authors accept to keep the pixel interface.
• Dense correspondence matters for cross-view matching, not global metrics: removing it reduces matching accuracy by 6.0 points after Stage 1 (37.4% to 31.4%), and the full recipe retains a 2.6-point advantage after Stage 2 (40.8% vs. 38.2%).

Methodology in Plain English

The encoder is trained in two stages, using only still images and text — no video and no robot interaction data.

Stage 1 trains a ViT-B/16 from scratch on DataComp-1B image-text pairs, pairing each image with its web caption and a synthetic caption generated by the LLaVA-LLaMA-3 captioner from Recap-DataComp-1B. Four objectives act on the same patch tokens: a contrastive image-text loss (each caption set gets its own softmax so the two captions for one image are never treated as negatives), a captioning loss where a decoder generates the synthetic caption from the image tokens, a reconstruction loss that rebuilds the image from both clean and noisy tokens, and a dense correspondence loss that matches each patch in a cropped/flipped view to its known location in the original view. Training processes 12.8 billion samples at 128×128, then 1.0 billion at 224×224. Notably, no external visual model supplies feature targets; the frozen CLIP ViT-L/14 used by the captioner influences the encoder only through text.

Stage 2 sharpens spatial and state information through cross-view self-distillation on 5 million single images from curated public image-editing datasets. The Stage 1 encoder is frozen as a teacher and receives a clean crop covering 40% to 100% of the image; a student initialized from the same weights receives a local crop (10% to 40%) or a global crop with photometric augmentation. A four-layer transformer predictor reads the student's tokens plus "geometry queries" that encode, for each teacher patch, where it sits in the student's coordinate frame and the log ratio of crop sizes. The student predicts the standardized teacher tokens at those locations, and only patches overlapping both views contribute to the loss. After training, the predictor is discarded and the refined student is ALPS, paired with the unchanged Stage 1 text encoder.

Evaluation keeps every encoder frozen and applies shared protocols: a 2.5M-parameter attentive probe over the last 16 frames for LIBERO state readout and 1- and 3-second prediction; each model's own text tower for ImageNet zero-shot and Flickr30k/COCO retrieval; a per-encoder lightweight adapter (trained on 19,000 ImageNet validation images, tested on 1,000 held-out images) mapping tokens toward SDXL autoencoder latents; the LIBERO transformer policy with 50 demonstrations per task; and the DINO-WM protocol on PushT, Wall, PointMaze, Granular, and Rope plus LeWorldModel on Reacher.

Why This Matters

Research impact. The paper reframes visual pretraining as a foundation for world models rather than only for recognition or generation, and its fifteen-encoder comparison is a reusable yardstick for asking what a frozen encoder actually provides. The finding that the same refinement recipe helps state accuracy but wrecks language alignment in SigLIP 2 and CLIP — and that ALPS's linear-projection readout may explain why it does not — is a concrete, testable hypothesis about encoder design.

Real-world applications:

• Robot manipulation: frozen ALPS tokens plus its native text tower support language-conditioned policies, reaching 85.5% mean success on LIBERO.
• Model-based planning: the tokens serve as the state space for planners that predict future tokens and search over actions in PushT, Wall, and Granular.
• Debuggable world models: because predicted future tokens decode into images, engineers can literally look at what a world model imagined and spot where it went wrong.
• Inspection and monitoring: the same tokens reconstruct observed scenes, so a single representation supports both perception and visual verification.

Industry relevance. Training uses only publicly available image-text data and a modest ViT-B/16 backbone, and all downstream systems keep the encoder frozen — an attractive property for teams that want one pretrained visual backbone shared across policy, planning, and visualization stacks instead of maintaining separate encoders for each.

Future Directions

• Scale and physical deployment: the authors explicitly leave larger model scales and real-robot deployment to future work, since all experiments use a ViT-B/16 backbone and simulation benchmarks.
• Single-stage training: folding cross-view refinement into one training stage instead of a two-stage curriculum remains an open question.
• Video and interaction data: ALPS is pretrained without video supervision and is described as less sensitive to fine-grained temporal actions, motivating an extension of Stage 2 to video and interaction data.
• A unified planner: downstream evaluations test language-conditioned control and world-model planning separately; integrating state, language, and pixel decoding into a single language-guided, pixel-verifiable planner is proposed as a natural next step.

Target Audience

Researchers and engineers working on robot learning, world models, and visual representation learning who need to choose or design a frozen visual encoder; practitioners building model-based control or planning systems; and students with some background in vision transformers and self-supervised learning who want a worked example of how pretraining objectives trade off against each other. Readers looking for a beginner tutorial on world models will find the evaluation protocol dense, but the two-stage recipe and the three-capability framing are described concretely enough to follow.

Authors’ abstract

World models turn visual representations into predictions and actions. We argue that their visual foundation should combine physical-state information, native language alignment, and pixel reconstruction, so that a single representation can support prediction, instruction, and inspection. However, existing encoders typically specialize in only one or two of these capabilities; we therefore introduce ALPS, a pretrained visual state encoder that unifies all three in a single set of patch tokens. Using only still images and text, we pretrain a ViT-B/16 from scratch with language, reconstruction, and spatial correspondence objectives, and then refine it via cross-view self-distillation. In a unified evaluation across fifteen public encoders, ALPS achieves the lowest aggregate LIBERO state-probe error among ViT-B models (matching the 3.5x larger V-JEPA 2-L) while maintaining strong language alignment and accurate pixel reconstruction. Downstream, frozen ALPS tokens reach the highest mean policy success on LIBERO (85.5%) when paired with our native text tower, match or exceed leading encoders on world-model planning across PushT, Wall, and Granular, and allow predicted future tokens to be decoded directly into pixels for visual inspection. We hope this work helps establish visual pretraining as a shared foundation for understanding, prediction, and action in world models.