PhysBrain 1.5: From Vision-Language Models to Physical Foundation Models

AI summary

Overview

Research area: Embodied AI / physical foundation models — specifically, extending vision-language models (VLMs) into unified systems for embodied understanding, robot action generation, and future-state prediction.

Technical level: Advanced. The paper assumes familiarity with autoregressive language modeling, VQ-VAE tokenization, vision-language-action (VLA) models, and robot end-effector control conventions.

Scope: A single 8B-parameter model, built on the Qwen3-VL Instruct family, that is trained to emit language, end-effector action tokens, and discrete future visual-state tokens through one shared autoregressive objective, and is evaluated on 28 embodied understanding benchmarks.

What This Paper Is About

Physical intelligence is framed by the authors as a recurring "physical loop": an agent observes a scene, reasons about it, plans and executes motion, and then observes how the world changed. Today these pieces are typically handled by separate specialized models — VLMs for understanding, VLAs for robot behavior, world models for prediction. This paper asks whether a single model can learn all three capabilities jointly, and shows that starting from a pretrained VLM, adding action and visual-state tokens to its vocabulary, and supervising everything with human interaction video plus a mixture of human, robot, and simulated data produces a model that reaches open-source state of the art on a broad embodied benchmark suite.

Key Contributions

1. A unified token vocabulary for physical generation. The language vocabulary is extended to 𝒱 = 𝒱_lang ∪ 𝒱_act ∪ 𝒱_vis, so that language answers, end-effector motion, and dense visual targets are all produced by the same backbone, token embedding, and output head. Task-specific loss masks select which tokens are supervised, but the prediction objective itself is unchanged.

2. Action generation as homogeneous trajectory continuation. Human motion (via the Human-as-Humanoid pipeline) and robot trajectories are mapped into a common 10-dimensional end-effector representation and discretized by an ActionPiece tokenizer into 32 tokens per wrist trajectory. Rather than forcing all data into one global coordinate frame, the model is conditioned on the preceding action chunk, which acts as an implicit system-identification signal for the local convention of each data source.

3. Future-state prediction as interleaved discrete tokens. The future physical state is defined as a spatially aligned triple — RGB image, depth map, and robot (or human arm) mask — tokenized by a shared VQ-VAE at 128×128 with a 16,384-entry codebook, producing 256 codes per modality. The three modalities are interleaved in RGB–depth–mask order at each spatial location, yielding 770 tokens per future-state payload.

4. A two-stage data and training recipe. Physical-aware pre-training draws all embodied supervision from human interaction video (roughly 30,000 hours after curation), followed by embodied SFT that mixes human demonstrations, real-robot trajectories from 17 sources, and simulated experience, with general instruction data retained in both stages.

Main Findings

• Overall embodied score 72.5 at 8B parameters. Across 28 embodied understanding benchmarks the model averages 72.5 on a 0–100 scale, which the authors describe as a new open-source state of the art and on par with proprietary systems Gemini 3.6 Flash (73.0) and GPT-6 Astra (73.3).

• Best open-source result on 14 benchmarks and second-best on 10. The remaining benchmarks are not claimed as first or second place among open-source models.

• Strong individual results include BLINK 87.9, CV-Bench 90.0, MindCube 86.2, VLABench 76.4, Part-Affordance 84.0, RoboRefit 89.6, ShareRobot-Traj. 84.9, VABench-Visual-Trace 89.8, and ViewSpatial-Bench 62.5.

• Weaker spots relative to the top proprietary systems include MMSI-Bench (41.0), ERQA (52.8), RefSpatial-Bench (50.9), PIOBench (68.3), and PixMo-Points (62.2).

• Comparison baseline. The base Qwen3-VL-Inst. 8B model scores 59.5 overall, and represents the starting point the paper builds from; other open-source comparators range from 57.4 (MiMo Embodied 7B) to 66.0 (Hy-Emb.-VLM-1.0 30A3B).

• Generation capabilities are shown qualitatively, not quantitatively. The paper reports qualitative examples of end-effector trajectory generation and future-scene prediction via spatially aligned RGB, depth, and robot-mask outputs. No numeric metrics for action generation or visual prediction appear in the provided content.

• General multimodal capabilities are claimed to be retained, but no general multimodal benchmark tables appear in the provided (truncated) content.

Methodology in Plain English

The authors start from an existing 8B vision-language model and give it three new "languages" of output.

Understanding stays as it was: the model answers questions in text or outputs spatial coordinates, points, boxes, and image-space waypoints, but it is fine-tuned on a large body of embodied annotations (counting, depth, spatial relations, viewpoints, planning steps, affordance, and motion traces).

Acting is converted into a token-generation problem. Each human or robot wrist motion is written as a 16-step chunk where every step has 3 numbers for relative position, 6 numbers for relative orientation (the first two columns of the rotation matrix), and 1 number for gripper closure. A dedicated tokenizer, trained on 28.7 million such chunks (459.2 million action timesteps), turns each chunk into 32 discrete tokens from a 512-token vocabulary. Crucially, the model is also fed the previous chunk as input, so it can infer the local motion habits of whichever robot or dataset it is looking at and simply continue the trajectory.

Predicting the future is handled the same way. The future frame's RGB, depth, and arm/robot mask are each compressed into a 16×16 grid of codes by a shared VQ-VAE. Those codes are interleaved position by position so that, at each grid cell, the model sees RGB then depth then mask, and it generates the whole interleaved sequence with the same output head. At inference, the decoder splits the sequence back into three grids and reconstructs each image separately.

Data. Pre-training supervision comes entirely from human interaction video — the paper cites Xperience-10M, Egocentric-10K, Ego4D, EgoVerse, EgoDex, EgoLife, Ego-Exo4D, and two in-house corpora (PhysBrain-Human and PhysBrain-Ego360). Perception data (24.3M samples), human action data (31.2M samples), and future-state data (26.8M samples) are all derived from the same episodes, alongside 14.9M general instruction samples. Embodied SFT then adds embodied understanding (6.61M), action (5.4M), and future-state (1.2M) samples from real robots and simulators, plus 1M general samples.

Training. Two stages, one epoch each, using ms-swift with the Megatron-Core backend. Adam with β₁=0.9, β₂=0.95, ε=10⁻⁸, weight decay 0.1, 3% warmup to a peak learning rate of 2×10⁻⁵ followed by cosine decay, maximum sequence length 32,768 tokens with sequence packing, roughly 2K samples per global batch, gradient norm clipped at 1.0, and bfloat16 precision. Data categories are mixed in proportion to sample counts. Both stages jointly optimize text, action-token, and visual-state generation.

Why This Matters

Impact on research. The paper argues that embodied understanding, action, and world modeling do not need separate architectures or separate heads — they can share a backbone, embedding table, and output head and be distinguished purely by task formatting and loss masks. It also argues that all embodied pre-training supervision can come from human video, with robot and simulation data reserved for a fine-tuning stage, which matters because robot data is far scarcer than human video. Achieving an average of 72.5 at 8B parameters, close to GPT-6 Astra (73.3) and Gemini 3.6 Flash (73.0), suggests that open-weight models of modest size can be competitive on embodied benchmarks.

Real-world applications:

• General-purpose robot manipulation assistants that must interpret an instruction, decide where to act, and emit a wrist trajectory in one pass.
• Household or warehouse robots needing to reason about spatial layout and viewpoint changes (the model is tested on RoboSpatial-Home, ViewSpatial-Bench, and 3DSRBench).
• Instruction-following systems for assistive or industrial robotics that require affordance and placement reasoning (Where2Place, RoboAfford, Part-Affordance).
• Predictive perception for safer automation, where anticipating the next visual state — including depth and a mask of the robot's own body — supports planning and collision avoidance.

Industry relevance. The model is built on an existing open model family and open training tooling, and the authors release a Hugging Face data collection and a GitHub evaluation kit, lowering the barrier for labs and companies to reproduce or build on the approach. Unifying three capabilities in one model also reduces the engineering cost of maintaining separate understanding, policy, and world-model stacks.

Future Directions

• Quantitative evaluation of action generation and future-state prediction. The paper reports only qualitative examples for trajectories and future frames; closed-loop robot success rates and image/depth prediction metrics are the obvious missing evidence.
• Global, canonical action frames. The authors deliberately avoid canonicalizing coordinates and rely on action history for implicit system identification. Whether explicit calibration or learned frame estimation can improve cross-embodiment transfer is left open.
• Closing the loop end-to-end. The physical loop is described as observe–act–re-observe, but the evaluation covers understanding plus offline generation. Running PhysBrain 1.5 as an actual closed-loop controller in real environments is a natural next step.
• Scaling and generalization. The future-state model predicts a single target frame rather than a rollout; extending prediction to longer horizons, and testing whether joint training measurably improves general multimodal ability (a claim made but not shown in the provided content), remain open.

Target Audience

Researchers and engineers working on embodied AI, vision-language-action models, and robot learning; teams building physical foundation models or world models; and practitioners who need a single model that handles perception, spatial reasoning, planning, action output, and future-state prediction. Readers without background in VLMs, tokenizer design, or robot control conventions will find the architecture sections dense, while those focused on benchmark comparisons will find the evaluation table directly useful.

Note on scope: the provided paper content is truncated after the description of comparison models in Section 5.1.2, so later evaluation sections — including general multimodal results, action-generation numbers, and ablations — are not available and are therefore not reported here.

Authors’ abstract

We present PhysBrain 1.5, a unified model for understanding physical environments, generating actions, and predicting future states. Motivated by the physical loop of observation, interaction, and environmental change, we bring these capabilities into a common learning framework. Starting from a general vision--language model, we encode language responses, end-effector motion, and dense visual targets as discrete sequences and jointly optimize them with autoregressive next-token prediction. Pre-training draws its embodied supervision entirely from human interaction videos, using task-centered episodes to pair semantic and spatial context with recovered motion and subsequent observations. We then adapt the model through supervised fine-tuning on a mixture of human demonstrations, robot trajectories, and simulated experience. Across 28 embodied understanding benchmarks, our 8B model achieves an average score of 72.5, setting a new open-source state of the art and performing on par with leading proprietary models such as GPT-6-Astra and Gemini 3.6 Flash. It achieves the best open-source results on 14 benchmarks while retaining general multimodal capabilities. Beyond these understanding evaluations, qualitative examples show the model's ability to produce end-effector trajectories and predict future scenes through spatially aligned RGB, depth, and robot-mask outputs.