World Embedding Benchmark

AI summary

Overview

Research area: Multimodal representation learning and evaluation — specifically, how video embeddings encode and align physical information, with a downstream link to video generation as world modeling.

Technical level: Intermediate. Readers need some familiarity with embedding models, contrastive learning, retrieval metrics (Recall@K), and regression metrics (nRMSE), but no deep physics or numerical-simulation background.

Scope: The paper introduces and applies the World Embedding Benchmark, an 8,000-case, 80-family simulation benchmark with three tasks for measuring physical alignment versus physical-property recoverability in video representations.

Note: the provided paper content is truncated partway through Appendix B.2.3, so details of the remaining appendices are not reported here.

What This Paper Is About

Video generative models are increasingly proposed as "world models," but it is unclear what physical information video representations actually contain. The authors build a controlled benchmark that pairs rendered physics simulations with their generating parameters, then use it to test two distinct capabilities: whether embeddings can match a video to a language description of its physics, and whether a trained probe can recover quantitative physical values from the same frozen embeddings. They then test whether improved physical retrieval actually helps a video generator produce more physically faithful output.

Key Contributions

1. The World Embedding Benchmark: 8,000 controlled simulation cases from 80 families spanning fluid mechanics, solid mechanics, dynamics, and optics & electromagnetism, each pairing a rendered video with simulation-derived physical annotations, supporting three complementary tasks (text-video retrieval, physical-property regression, and multiple-choice video-description pair classification).

2. A diagnosed gap between encoding and alignment: Pre-trained omnimodal embedding models show weak physics-aware retrieval and near-chance within-family pair classification, while lightweight probes still recover useful physical information from frozen video embeddings — evidence that physical signals exist in the representations but are not accessible through cross-modal similarity.

3. An alignment–recoverability trade-off: Continual contrastive training with physics-specific video-text pairs improves retrieval and pair classification but degrades physical-property regression, suggesting contrastive adaptation reorganizes and aligns existing signals rather than adding new physical capability.

4. A downstream demonstration: Embeddings used as retrievers for retrieval-augmented generation with MiniMax-H3 supply reference videos that improve the physical fidelity of generated videos, with stronger retrieval models yielding larger gains in the reported experiments.

Main Findings

• Pre-trained embeddings are weak at physics-aware retrieval. Recall@10 within each branch stays low across all evaluated embedding models: Video-CLIP-XL (3.3 Fluid, 1.4 Solid, 2.3 Dynamics, 0.6 Optics & EM), Qwen3-VL-Embedding-2B (3.9 / 1.6 / 7.3 / 2.3), Qwen3-VL-Embedding-8B (6.6 / 1.9 / 7.4 / 3.1), Omni-Embed-Nemotron-3B (3.1 / 1.4 / 3.1 / 0.7), LCO-Embedding-3B (3.9 / 1.5 / 5.1 / 0.7), LCO-Embedding-3B-2605 (4.6 / 1.5 / 4.5 / 1.2), and LCO-Embedding-7B (4.9 / 1.6 / 5.7 / 1.9).

• Within-family pair classification is near chance, but cross-family is easier. Embedding models score roughly 46.8 to 53.1 on within-family pairs versus 66.1 to 86.2 on cross-family pairs.

• Directly prompted MLLMs do much better on pair classification. Qwen3.5-27B (61.3 within-family, 92.0 cross-family), Qwen3.5-35B-A3B (58.8 / 91.2), Qwen3.6-27B (60.2 / 91.9), and Qwen3.6-35B-A3B (59.0 / 90.9) reach roughly 91–92% cross-family and 59–61% within-family accuracy, indicating the capability exists in generative models but is not exposed through embedding similarity.

• Probes recover physical quantities from frozen embeddings. Macro-average nRMSE across the 10 regression tasks ranges from 0.81 (Video-CLIP-XL) to 2.47 (LCO-Embedding-3B) among embedding models, with intermediate values of 1.76 (Qwen3-VL-Embedding-2B), 1.80 (Qwen3-VL-Embedding-8B), 1.41 (Omni-Embed-Nemotron-3B), 1.85 (LCO-Embedding-3B-2605), and 2.02 (LCO-Embedding-7B).

• V-JEPA2, trained without text alignment, retains quantitative physical information. V-JEPA2 ViT-L records nRMSE 0.94 and ViT-g records 1.02 on physical-property regression using frozen video features; the paper describes this as strong regression performance and reads it as evidence that predictive video representation learning can preserve physical information without language alignment.

• Physics adaptation boosts retrieval and pair classification. LCO-Embedding-3B + Physics Adaptation reaches Recall@10 of 18.6 / 8.0 / 16.5 / 8.9 across the four branches, 53.5 within-family accuracy, and 98.2 cross-family accuracy; LCO-Embedding-7B + Physics Adaptation reaches 17.9 / 9.4 / 21.0 / 9.4, 59.6 within-family, and 98.4 cross-family.

• Adaptation degrades regression. Physics-adapted models have higher (worse) nRMSE than the base models: 3.95 for the 3B variant and 2.89 for the 7B variant, versus 2.47 and 2.02 for the corresponding base LCO-Embedding models. The paper interprets this as a trade-off between cross-modal alignment and quantitative information recoverability.

• Physics retrieval keeps improving with more training. Retrieval performance continues to rise throughout training with no sign of convergence in the allocated budget; the physics-only setting with 2,000 examples requires 16 epochs, and mixing general-domain video-text data slows training — unlike general-domain MLLM-based embedding training, where up to 2 epochs is typically sufficient.

• Stronger backbones give stronger physics retrieval. After identical contrastive training, the 7B backbone outperforms the 3B backbone, which the paper reads as supporting a generation–representation scaling pattern in physics representation learning.

• Retrieved references improve generated video fidelity. With MiniMax-H3, the no-RAG baseline scores 0.67 (Optics), 0.53 (Thermal), and 0.60 (Average) on PhyGenBench. VideoCLIP-XL retrieval slightly degrades the average to 0.59. LCO-Omni-3B improves it to 0.63. The physics-adapted LCO variants reach 0.66 average (general + physics) and 0.65 average (physics-only), with the general+physics variant attaining the best average score.

Methodology in Plain English

The authors start by generating physics data rather than collecting it. They pick 80 "families" of physical systems across four branches — fluid mechanics, solid mechanics, dynamics, and optics & electromagnetism — and simulate 100 parameter variations per family, giving 8,000 cases. Different branches use different established simulators: OpenFOAM for fluids, FEniCSx/DOLFINx for solids, Project Chrono plus analytical or ODE integration for dynamics, and analytical relations with HCIPy, Mitsuba 3, and Meep (FDTD) for optics and electromagnetics. The resulting fields and trajectories are rendered into PNG frames with ParaView 6.0.1 and encoded into MP4 clips with FFmpeg, and automated checks screen for failed runs, NaN or infinite values, blank or flickering videos, and duplicate case IDs or parameters. Because the data is simulated, every video comes with its exact generating parameters.

Three tasks reuse this data. In retrieval, a template sentence describing the physical system is instantiated with each case's metadata, and models must match text to video and video to text within a branch. In regression, ten family-specific continuous targets (such as gravity, Reynolds number, wavelength, or elastic modulus) are predicted from frozen video embeddings with a linear probe, scored by nRMSE with Spearman's rank correlation as a complement. In pair classification, ten anchors per family are each paired with a within-family negative and a cross-family negative, and models either compare similarities (embeddings) or answer a prompted binary question (MLLMs).

For adaptation, the authors simulate 8,000 new text-video pairs with different numerical values, sample one hard negative per anchor from the same family, and run continual contrastive training from LCO-Embedding-Omni-3B and 7B. Two settings are compared: physics-only (nested subsets of 2k, 4k, 8k examples; batch size 64; 16, 8, and 4 epochs) and general training (the same physics subsets mixed with 271k general video-text pairs, giving 273k, 275k, 279k examples; batch size 512; 2 epochs). Finally, they use embedding models as retrievers to feed reference videos into MiniMax-H3's ref2va variant and score the generated videos on PhyGenBench.

Why This Matters

Impact on research. The paper separates two things that are often conflated: whether a representation contains physical information, and whether that information is reachable by similarity matching. Showing that these can move in opposite directions under contrastive training is a caution for anyone building or evaluating embeddings for scientific or physical domains, and it argues for reporting both alignment and recoverability metrics rather than one alone.

Real-world applications.

• Engineering and scientific design tools that retrieve prior simulation or experiment videos resembling a new case, then estimate parameters such as Reynolds number or elastic modulus.
• Retrieval-augmented video generation for physically plausible content, where retrieved reference clips guide a generator toward correct fluid, thermal, or optical behavior.
• Robotics and embodied agents that need representations sensitive to physical parameters, not just scene categories, to predict how a system will behave under changed conditions.
• Education and scientific communication, where retrieving a matching simulation for a described phenomenon supports accurate visualization.

Industry relevance. The results touch two commercial threads: video generation products that need physically credible output, and multimodal embedding infrastructure used for search, recommendation, and retrieval over video corpora. The finding that stronger generative backbones yield stronger retrieval after identical contrastive training also matters to teams deciding how much model capacity to invest in embedding quality.

Future Directions

• Closing the alignment–recoverability trade-off. The paper observes that physics-property regression may need objectives beyond contrastive learning, such as explicit representation regression losses, and leaves this as an open direction.

• Explaining why adaptation degrades regression. The authors offer two interpretations — that contrastive learning calibrates existing capabilities rather than enhancing them, and that regression needs different objectives — but do not settle the question experimentally.

• Extending the training budget. Retrieval performance had not converged within the allocated training resources, and general-domain mixing slowed learning, so the achievable ceiling under longer or differently scheduled training is unknown.

• Broadening the evaluation. The benchmark covers four physics branches and ten regression targets; whether the observed patterns hold for other physical domains, other simulator families, and other embedding or JEPA architectures remains to be tested.

• Strengthening the RAG link. Only top-1 retrieved references are reported with one generator and one benchmark; how retrieval quality translates into generation gains under other retrieval depths or generators is not established here.

Target Audience

Researchers and engineers working on multimodal representation learning, video-text retrieval, scientific machine learning benchmarks, and video generation as world modeling. It is also relevant to evaluators and benchmark designers who need to distinguish representational content from cross-modal usability, and to practitioners building retrieval-augmented generation pipelines over video.

Authors’ abstract

Physical fidelity has received increasing attention in world models and video generation, yet how video representations encode physical information remains less understood. We introduce the World Embedding Benchmark, comprising 8,000 controlled simulation cases from 80 families spanning fluid mechanics, solid mechanics, dynamics, and optics & electromagnetism. Each case pairs a rendered video with simulation-derived physical annotations, supporting three complementary tasks: text-video retrieval, physical-property regression, and multiple-choice video-description pair classification. We use these tasks to distinguish cross-modal physical alignment from the recoverability of quantitative physical information. Evaluated pre-trained omnimodal embedding models show weak retrieval and near-chance within-family pair classification, while lightweight probes recover useful physical information from frozen video embeddings. Continual contrastive training with physics-specific video-text pairs improves retrieval and pair classification but degrades physical-property regression, revealing a trade-off between alignment and quantitative information recoverability. Finally, we use the embeddings to retrieve reference videos for retrieval-augmented generation with MiniMax-H3. Retrieved references improve the physical fidelity of generated videos, with stronger retrieval models yielding larger gains in our experiments. Together, these findings highlight the need to evaluate physical alignment and property recoverability jointly, and demonstrate the utility of physical representations for improving video generation.