Order from Chaos: Physical World Understanding from Glitchy Gameplay Videos

AI summary

Overview

Research area: Multimodal large language models (MLLMs), video-language instruction tuning, and physical world understanding (object dynamics, material properties, causal interactions).

Technical level: Intermediate. The paper is readable without deep background, but it assumes familiarity with instruction tuning, multimodal benchmarks, and evaluation protocols for vision-language models.

Scope: The paper introduces PhysGame, an instruction-tuning dataset built from glitch-containing gameplay videos (140,057 question–answer pairs across five physical domains), plus GameBench, an expert-annotated 880-video evaluation benchmark, and demonstrates that tuning on glitches improves both real-world physical reasoning and general video understanding.

What This Paper Is About

Current MLLMs understand physics far worse than humans — on the PhysBench benchmark, GPT-4o reaches only 49.49% average accuracy versus human-level performance of 95.87%. Existing physical-reasoning datasets are either real-world videos (high annotation cost, hard to scale) or simulator-generated synthetic data (uniform backgrounds, simple primitives such as cubes and cylinders, large sim-to-real gap). This paper proposes a third source: visual glitches in gameplay videos — anomalies where rendering errors, physics engine limits, or unexpected object interactions violate the game's own physical rules — and uses them as a scalable, photo-realistic supervision signal.

Key Contributions

1. PhysGame, a glitch-centric instruction-tuning dataset. 140,057 question–answer pairs drawn from 38,957 unique videos, covering five physical domains (mechanics, optics, material properties, thermodynamics, electromagnetism) and sixteen fine-grained categories. Videos are harvested from the GamePhysics subreddit, which pairs glitch footage with user-written titles and descriptions.

2. A meta-information-guided prompting strategy. Rather than prompting GPT-4o on video alone, the pipeline injects video-associated metadata (titles, descriptions) into the prompt. Manual validation on 2,000 sampled instances reports 86% metadata accuracy and 91% QA accuracy with metadata versus 64% without.

3. GameBench, an expert-annotated evaluation benchmark. 880 glitch-identified gameplay videos (disjoint from PhysGame), each with a four-way multiple-choice question whose correct option describes the physical-commonsense violation, with distractors built to reference objects and actions genuinely present in the video.

4. Empirical evidence of dual transferability. Tuning on gameplay glitches improves real-world physical reasoning ("Game2Real") and general video understanding ("Game2General"), across three base models.

Main Findings

• GameBench is difficult for current models. Among proprietary models, GPT-4o scores 56.1% average accuracy and Gemini-1.5-pro 55.2%; Claude3.5-Sonnet scores 54.3%. Open-source video models score lower (for example, LLaVA-Next-Video 32.2%, Video-LLaVA 29.0%, Qwen2-VL-7B 37.5%). GPT-4o's score is measured after filtering out questions that could be answered from text alone during quality control.

• PhysGame tuning substantially raises gameplay physical reasoning. Absolute average gains on GameBench: Qwen2-VL-7B +6.3 (37.5 to 43.8), Qwen2.5-VL-7B +3.7 (44.4 to 48.1), InternVL2.5-8B +9.4 (38.6 to 48.0). The Qwen series shows smaller headroom, which the authors attribute to already-optimized vision-language alignment.

• Game2Real transfer to real-world physics benchmarks. On PhysBench, Qwen2-VL-7B improves +2.5 (46.6 to 49.1), Qwen2.5-VL-7B +1.7 (45.9 to 47.6), and InternVL2.5-8B +2.1 (43.9 to 46.0). On MMVU validation, gains are +6.5 (Qwen2-VL-7B), +1.7 (Qwen2.5-VL-7B), and +8.5 (InternVL2.5-8B). Note: the abstract attributes the 2.5% PhysBench gain to Qwen2.5-VL, while Table 4 shows +2.5 for Qwen2-VL-7B and +1.7 for Qwen2.5-VL-7B.

• Game2General transfer to general video understanding. Qwen2-VL-7B gains +1.9 on MVBench (64.5 to 66.4) and +1.3 on Video-MME without subtitles (58.0 to 59.3). Qwen2.5-VL-7B gains +1.3 on MVBench (66.0 to 67.3) and +2.3 on LongVideoBench (56.0 to 58.3). InternVL2.5-8B reaches 72.8 on MVBench (from 72.0), reported as a new state-of-the-art result, but shows small drops on the medium and long splits of Video-MME (-0.2 and -0.1), which the authors attribute to a short-duration bias from PhysGame's short videos (average 23.33 seconds).

• Meta-information guidance is measurably important. Training Qwen2.5-VL on PhysGame generated without metadata drops PhysBench from 47.6 to 42.9, a 4.7-point absolute difference across the full model comparison.

• Data mixing helps. With a fixed 160K budget and controlled resampling, PhysBench accuracy is 45.9% at 0K PhysGame / 160K LLaVA-Hound, 46.9% at 40K/120K, 47.3% at 80K/80K, 47.6% at 140K/20K, and 45.1% at 140K/0K. The 140K/20K mixture was selected as optimal; training on PhysGame alone falls below the zero-shot baseline, suggesting a gameplay-domain bias.

• Scaling has not saturated. Using Qwen2.5-VL-7B with 0K, 40K, 80K, and 140K PhysGame samples, gains are monotonic and do not plateau at 140K.

• Errors are dominated by missing knowledge, not perception. Manual analysis of 100 GameBench cases for Claude 3.5-Sonnet, GPT-4o, and Gemini 1.5-Pro finds "Lack of Knowledge" at roughly 79–81% of errors, Reasoning Errors at 7–12% (highest for GPT-4o at 12%), Perception Errors at 4–7%, and Refusal to Answer at 2% for Claude 3.5-Sonnet and 5–7% for GPT-4o and Gemini 1.5-Pro.

• Qualitative gains in causal reasoning. On MMVU cases, the fine-tuned model attributes a landslide to gravity where the baseline attributes it to dynamite, and correctly infers one degree of freedom in a linkage mechanism where the baseline says two.

Methodology in Plain English

The researchers collect gameplay clips showing physics glitches from the GamePhysics subreddit, a community where users post unusual in-game events along with descriptive titles and discussion threads. Instead of hand-writing supervision, they prompt GPT-4o to generate questions and answers about the clips, using a self-instruction paradigm with three question styles: directly asking about the glitch, asking whether any anomalies are present, and simply asking for a description of the video content. To reduce hallucination, the prompt includes the video's metadata (title/description) as a hint — the paper shows an example where the title mentioning "water physics" lets GPT-4o correctly note the absence of wave propagation, while a metadata-free prompt produces invented content.

Because per-sample human review of 140,057 pairs is infeasible, quality is assessed by manually checking a 2,000-instance sample on two dimensions: whether metadata matches video semantics, and whether the QA matches the observed anomaly.

For evaluation, they build GameBench separately, excluding any PhysGame videos to avoid leakage, checking for duplicate downloads and non-game content, and having expert annotators write four-way multiple-choice questions. Distractors must reference people or actions actually present in the video (so the task cannot be solved by object recognition alone), options must be similar in length, and the correct answer must be evenly distributed at 25% per option. A GPT-4o text-only filter removes questions answerable without watching the video, capping its question-only accuracy below 25%.

Training uses supervised fine-tuning on three open models — Qwen2-VL-7B, Qwen2.5-VL-7B, and InternVL2.5-8B. To avoid over-biasing toward gameplay, the authors add 20K samples from the general video instruction dataset LLaVA-Hound, giving a 160K training set. Qwen models train for one epoch with batch size 128 and learning rate 2e-7; InternVL2.5-8B uses 1e-6 with the same batch size. All models take 32-frame video inputs, and experiments run on 8 NVIDIA A100 GPUs. Evaluation spans GameBench, PhysBench, and MMVU (physical understanding) plus MVBench, Video-MME, and LongVideoBench (general video understanding), with selection accuracy for multiple-choice scoring.

Why This Matters

Impact on research. The paper reframes data scarcity in physical reasoning: instead of paying for real-world video curation or accepting the sim-to-real gap of physics engines, it exploits a data source that is abundant, visually complex, and already labeled by the fact that a glitch is visible. The demonstrated Game2Real and Game2General transferability suggests physics-oriented tuning is not a narrow task specialization but a route to broader visual reasoning. The error analysis also redirects attention: if 79–81% of failures are "lack of knowledge," better perception or reasoning architectures alone will not close the gap.

Real-world applications.

• Automated game quality assurance: detecting physics bugs, clipping, and collision errors in playtest footage at scale rather than through manual QA.
• Robotics and embodied AI: training agents to notice when the world behaves contrary to expectation, a prerequisite for safe planning and manipulation.
• Video generation and VFX: using glitch detectors as reward or filtering signals to suppress physically implausible outputs in generative video.
• Video surveillance and industrial inspection: transfer of anomaly-detection ability to footage where objects or materials behave unexpectedly.

Industry relevance. Game studios, MLLM and video-model developers, and robotics companies all stand to benefit. The dataset's reliance on publicly available user footage with metadata means the pipeline is cheap to reproduce and extend, which matters for organizations that cannot afford large-scale human annotation.

Future Directions

• Scale further. The data-scaling curve (0K to 140K) is still improving without plateauing, so the obvious next question is how much more data yields how much more gain.
• Fix the short-video bias. InternVL2.5-8B regressed slightly on Video-MME's medium and long splits after tuning; longer-form gameplay footage or length-balanced sampling could address this.
• Broaden taxonomy and sources. PhysGame covers five domains and sixteen categories from one subreddit community; extending to more games, genres, and languages may improve diversity.
• Attack the knowledge bottleneck. Since most errors are knowledge-related rather than perceptual, integrating retrieval, knowledge bases, or explicit physics simulation into inference is a natural follow-up.
• Move beyond glitch detection to prediction. The current formulation asks models to identify violations after the fact; predicting when a physical rule will be violated is a harder and more useful capability.

Target Audience

Researchers and graduate students working on multimodal LLMs, video-language understanding, and physical/causal reasoning; dataset and benchmark builders interested in low-cost annotation pipelines; practitioners building game QA, content moderation, or generative video systems; and anyone evaluating how far current models are from human-level physical intuition. Readers with a general ML background can follow the paper, though the benchmark tables and ablation design are aimed at specialists.

Authors’ abstract

Understanding the physical world, including object dynamics, material properties, and causal interactions, remains a core challenge in artificial intelligence. Although recent multi-modal large language models (MLLMs) have demonstrated impressive general reasoning capabilities, they still fall short of achieving human-level understanding of physical principles. Existing datasets for physical reasoning either rely on real-world videos, which incur high annotation costs, or on synthetic simulations, which suffer from limited realism and diversity. In this paper, we propose a novel paradigm that leverages glitches in gameplay videos, referring to visual anomalies that violate predefined physical laws, as a rich and scalable supervision source for physical world understanding. We introduce PhysGame, an meta information guided instruction-tuning dataset containing 140,057 glitch-centric question-answer pairs across five physical domains and sixteen fine-grained categories. To ensure data accuracy, we design a prompting strategy that utilizes gameplay metadata such as titles and descriptions to guide high-quality QA generation. Complementing PhysGame, we construct GameBench, an expert-annotated benchmark with 880 glitch-identified gameplay videos designed to evaluate physical reasoning capabilities. Extensive experiments show that PhysGame significantly enhances both Game2Real transferability, improving the real world physical reasoning performance of Qwen2.5VL by 2.5% on PhysBench, and Game2General transferability, yielding a 1.9% gain on the MVBench benchmark. Moreover, PhysGame-tuned models achieve a 3.7% absolute improvement on GameBench, demonstrating enhanced robustness in detecting physical implausibilities. These results indicate that learning from gameplay anomalies offers a scalable and effective pathway toward advancing physical world understanding in multimodal intelligence.