Lab2Sim: Turning Real-World Experiment Videos into Interactive Virtual Experiments

AI summary

Overview

Research area: Multimodal video understanding, front-end code generation, and AI-assisted science education — specifically the automated conversion of real-world experiment videos into interactive, browser-based virtual experiments.

Technical level: Intermediate. The framing is accessible, but the method involves multimodal model pipelines, explicit state machines, and browser-based execution verification.

Scope: The paper introduces Lab2Sim, a benchmark of 218 science experiment videos with 11,881 rubric items, and EduVideo2Code, a compile–verify–repair framework that turns such videos into interactive virtual experiments.

What This Paper Is About

Science experiment videos show learners how an experiment is performed, but they offer no way to actually carry it out. The paper asks what it would take to automatically convert such a video into a browser-based virtual experiment that both reproduces the apparatus and state changes shown in the footage, and enforces the preconditions that govern each operation. To answer this, the authors build an evaluation benchmark (Lab2Sim) with video-grounded specifications and rubric items, and propose a framework (EduVideo2Code) that extracts procedures from video, compiles them into a state machine, generates a page, and repairs it through browser execution.

Key Contributions

1. Lab2Sim benchmark. A dataset of 218 real-world chemistry, physics, and biology experiment videos (89 / 87 / 42), each paired with an evidence-grounded gold specification of apparatus, initial conditions, ordered steps, preconditions, and state transitions. It yields 11,881 rubric items (54.5 per instance on average).

2. Four-dimension rubric evaluation. Apparatus Fidelity (AF), Procedure Execution (PE), Procedural Constraint Compliance (PCC), and State-Transition Fidelity (STF), separating fidelity (AF, STF) from operability (PE, PCC), plus diagnostic metrics: Runnable Rate (RR) and Operation Failure Rate (OFR).

3. EduVideo2Code framework. A compile–verify–repair pipeline combining hierarchical procedure extraction (HPE), state-machine-driven generation (SMG), and execution-based verification (EBV) with targeted repair.

4. Evidence-grounded annotation protocol. Every annotated element is linked to source frames or a time interval and labeled video_visible, apparatus_inferred, or domain; fixed templates convert gold fields into rubric items, including automatically generated "violation probes" (skip, reorder, overdose, substitute).

Main Findings

• Runnable does not mean faithful. All eight evaluated systems reach a Runnable Rate above 93%, yet scores diverge widely: AF ranges from 20.62 to 61.32 and PE from 20.11 to 41.40. Among the six baselines, no model reaches 60% apparatus fidelity (the highest baseline is DeepSeek-V4.1-Flash at 59.09).

• Apparatus is easier than procedure. For every system AF is higher than PE, though for Seed-2.1-Pro the gap is only 0.51 points. The authors read this as drawing the right apparatus being easier than making every step work.

• Executing steps differs from enforcing preconditions. Kimi-K3 has the second-lowest OFR among baselines (8.04%) but a PCC of only 22.22%, while GPT-6-Astra has the highest baseline PCC (44.83%) despite an OFR of 22.28%.

• EduVideo2Code improves both backbones on all four dimensions. With GLM-5.3-Flash: AF 61.32 (+20.88), PE 37.17 (+14.59), PCC 38.33 (+22.00), STF 49.63 (+15.15), OFR 7.10, RR 100.00. With GPT-5.6-Sol: AF 60.76 (+24.15), PE 41.40 (+8.61), PCC 33.75 (+1.19), STF 45.45 (+3.77), OFR 12.38, RR 100.00.

• Gains are uneven across dimensions and backbones. PCC improves by 22.00 points with GLM-5.3-Flash but only 1.19 with GPT-5.6-Sol; the GPT-5.6-Sol variant still trails DeepSeek-V4.1-Flash (36.67) and GPT-6-Astra (44.83) in PCC, and its STF (45.45) is close to Kimi-K3's (44.86).

• Ablations show all three stages matter (GLM-5.3-Flash backbone). Removing HPE causes the largest AF (−13.08), PE (−13.64), and PCC (−16.30) drops. Removing SMG causes the largest STF drop (−12.13) but almost no AF change (−1.29). Removing EBV produces the highest OFR (14.84), worse than direct generation's 12.90.

• Stages are interdependent. For PE, PCC, and STF, the drops from removing single stages sum to more than the total gain over direct generation, and explicit preconditions alone do not make pages enforce them (removing SMG lowers PCC by 6.13 versus about 16 for HPE or EBV).

• Four recurring failure modes in case studies. Blank rendering (GPT-5.6-Sol shows only "[object Object]" for a candle-combustion page), experiment substitution (GPT-6-Astra generates a potassium permanganate experiment and Seed-2.1-Pro a fuchsine diffusion experiment instead of the ammonia and phenolphthalein setup), apparatus mismatch (GPT-6-Astra draws a balance-like stand and bent tube instead of a candle, knife, and crucible tongs), and non-executable operations (Kimi-K3 and Seed-2.1-Pro scenes do not respond to lighting, splint insertion, or beaker positioning; GPT-5.6-Sol does not support dragging beakers).

• Residual weakness. Subtle physical phenomena remain hard: condensation on a beaker wall is conveyed through text rather than a visual change.

Methodology in Plain English

Building the benchmark. The authors gathered science experiment videos from Bilibili and YouTube, filtering more than 500 candidates down to 218 by keeping only footage where the procedure, apparatus, and visible state changes could be reliably identified (excluding duplicates, videos below 720p, overly complex or occluded experiments, and videos with only narration or final results).

Annotating with evidence. For each video, an assisting model from the GLM-5.3 series drafted a structured specification working coarse to fine: a first pass over the whole video established context, phases, and candidate steps; a second pass inspected each relevant interval at denser sampling to fix event boundaries. Human reviewers then checked every draft against the footage, taking about 13 minutes per video on average. Each element was tagged with one of three evidence labels: video_visible (seen directly), apparatus_inferred (read from apparatus context), or domain (textbook knowledge not shown). Every claim had to survive all three categories or be discarded.

Turning specifications into tests. Fixed templates convert each gold field into a rubric item — apparatus entities become identity (AF) items, steps become execution (PE) items, preconditions become constraint (PCC) items, and state transitions and initial conditions become state (STF) items. Additional "violation probes" deliberately break the procedure (skipping a step, swapping adjacent steps, multiplying a quantity by five, applying a step to the wrong container) to test whether the page blocks invalid actions.

The framework. EduVideo2Code runs in three stages. Hierarchical procedure extraction (HPE) analyzes the video coarse-to-fine so that brief state transitions are not missed and unsupported steps are not invented; every extracted step must cite supporting frames. State-machine-driven generation (SMG) compiles that procedure into an explicit state machine — states assign values to object attributes such as flask.solution.color, operations implement steps, and preconditions gate them — then translates the machine into a page where each operation becomes a gesture (drag, pour, light) and each state becomes a visual appearance. Execution-based verification (EBV) uses a separate verifier session to run the page in a browser, compare apparatus and spatial relations with the video, attempt invalid operations, and replay the procedure, comparing screenshots and state summaries against video evidence. If the score falls below θ = 0.8, targeted repair is triggered (new asset search for apparatus errors, procedure re-inspection or state-machine revision for operation/state errors), up to T_max = 10 rounds; pages average 5.3 repair rounds across both backbones.

Evaluation setup. Six baselines (Seed-2.1-Pro, GLM-5.3-Flash, GPT-5.6-Sol, GPT-6-Astra, DeepSeek-V4.1-Flash, Kimi-K3) generate pages directly from 32 uniformly sampled frames. Generated pages run in a sandboxed Chromium browser at 2560×1440. Evaluation is split to decouple operation from judgment: Qwen-3.8-Max locates controls and executes steps via clicking, hovering, and scrolling, recording at 30 FPS; Gemini-3.8-Flash then scores each rubric item against the attached video evidence. On 50 sampled cases the judge agrees with blind human judgments at a Cohen's κ of 0.79.

Why This Matters

Research impact. The paper separates two things the field often conflates: a page that runs and a page that faithfully reproduces an experiment. By showing that >93% runnable pages can coexist with apparatus fidelity as low as 20.62, it argues that front-end code generation benchmarks built on interface transitions do not transfer to domains with physical state and procedural constraints. The evidence-labeling scheme and the four-dimension rubric offer a reusable evaluation design for any task where a generated artifact must match observed physical behavior.

Real-world applications.

• Science education at scale: converting existing instructional video libraries into practice environments where students perform steps rather than watch them.
• Laboratory pre-training: letting learners rehearse procedures and see precondition violations (e.g., pouring from an empty container) before touching real equipment.
• Accessible and remote learning: browser-based experiments for schools without lab equipment or for remote students.
• Content reuse for educators and publishers: turning demonstration footage into interactive courseware.

Industry relevance. The work sits at the intersection of multimodal video understanding, code generation, and educational technology. The finding that verification-and-repair rounds measurably improve fidelity — at the cost of more model calls — is directly relevant to anyone deploying agentic generation pipelines where output correctness must be checked against a reference rather than assumed.

Future Directions

• Evaluating behavior the video does not show. Lab2Sim scores pages only against what the source footage demonstrates; scientifically correct behavior that the video omits is not assessed, leaving open how to credit justified inference.

• Stronger precondition enforcement. PCC was the least consistent gain (22.00 points with GLM-5.3-Flash versus 1.19 with GPT-5.6-Sol), and the ablation suggests enforcement depends on preconditions recovered by HPE and checked by EBV working together rather than on the state machine alone.

• Rendering subtle physical phenomena. Condensation on a beaker wall was conveyed as text rather than a visual change, pointing to a gap in representing gradual or fine-grained physical effects.

• Human and cost validation. The authors note that scoring partly relies on a model judge validated on 50 sampled cases, that EduVideo2Code requires more model calls than direct generation, and that they have not studied effects on learners — all natural targets for follow-up work, alongside broadening beyond 218 videos in three disciplines from two public platforms.

Target Audience

Researchers and graduate students working on multimodal video understanding, code generation from visual references, and agentic generate–verify pipelines; builders of educational technology and virtual lab platforms; and benchmark designers interested in evaluating fidelity and operability separately. Educators and curriculum developers evaluating whether automatically generated virtual experiments are trustworthy enough for classroom use will also find the failure-mode analysis and stated limitations directly relevant.

Authors’ abstract

Science experiment videos demonstrate how experiments are performed, but offer no direct interaction for learners. Turning them into virtual experiments requires not only reproducing the observed apparatus and state changes, but also enforcing the preconditions governing each operation. We introduce Lab2Sim, a benchmark of 218 science experiment videos with evidence-grounded specifications and 11,881 rubric items. To address this task, we propose EduVideo2Code, which extracts experimental procedures hierarchically, encodes them as explicit state machines to guide code generation, and refines the resulting pages through execution-based verification. Experiments show that existing multimodal models readily generate runnable pages but struggle to faithfully reconstruct experimental apparatus. EduVideo2Code consistently improves all four evaluation dimensions across two backbone models, with apparatus fidelity gains of 24.15 points.