LabInstruct: Benchmarking Situated Instructional Video Generation for Lab Procedures

AI summary

Overview

Research area: Robotics / video generation evaluation — specifically, AI-generated instructional video for laboratory procedures (self-driving laboratories, human-in-the-loop automation).

Technical level: Intermediate. Readers should be comfortable with image-to-video (I2V) generation models and standard benchmark evaluation design; the scoring math is explained but not deeply technical.

Scope (1 sentence): The paper introduces LabInstruct, a 204-task benchmark of real laboratory procedures for testing whether image-to-video models can generate procedurally correct instructional videos, not merely visually plausible ones.

What This Paper Is About

Self-driving laboratories aim to automate experiments end to end, but reliable robotic execution in real labs is still hard, so humans remain the practical interface between an AI-generated plan and the physical experiment. Because lab work is visual, spatial and dynamic, video is a natural way to show someone how to carry out a procedure. The paper asks whether current video generation models, given a single starting image of the workspace plus a natural-language instruction, can generate a video that actually demonstrates the procedure correctly — and shows that visual realism frequently hides procedural errors.

Key Contributions

1. A new task formulation. The authors frame laboratory video generation as human-facing procedural communication: the generated video must use the right apparatus, act on the right objects, preserve causal order, produce observable state transitions, and visibly reach the intended outcome, while preserving the identity and spatial geometry of the workspace.

2. The LabInstruct benchmark. 204 tasks across 5 scientific disciplines and 2 complexity levels, each with a real reference execution from the same workspace plus structured annotations of objects, roles, spatial relations, ordered actions, contact types, and observable pre-/post-action states. Source videos come from Bilibili (61 videos, 163 clips), FineBio (15 videos, 23 clips), ExpVid (3 videos, 14 clips) and YouTube (2 videos, 4 clips) — 81 source videos yielding 204 clips.

3. A procedure-aware evaluation protocol. A three-layer scoring scheme built on 3,229 expert-verified checklist items (1,312 for Level 1, 1,917 for Level 2; averaging 13.0 and 18.6 items per task), combining four core visual-quality dimensions, a critical-completion factor, and scene-consistency and safety validity gates.

4. A systematic study of 8 frontier image-to-video models (Wan 2.2, LTX 2.3, Cosmos3 Nano, Cosmos3 Super, LingBot Video, MiniMax H3, plus commercial Seedance 2.0 and Wan 3.0), quantifying the gap between visual plausibility and end-to-end procedural correctness under both automated (GPT-5.6 Sol) and human evaluation.

Main Findings

• Procedural execution is the bottleneck, not appearance. Across models, object consistency and physical plausibility stay relatively strong while action fidelity, state correctness and especially completion collapse on Level 2. For example, under GPT-5.6 Sol, Wan 2.2 scores 93.7 on object consistency and 90.1 on physical plausibility at Level 1, but only 59.4 on action fidelity.

• High validity gates do not imply high performance. Scene consistency and safety gates remain near-ceiling (for instance, LingBot Video reaches 100.0 scene consistency and 92.0 safety at Level 1 under the VLM evaluator) while pooled Overall scores stay in the low tens, showing that well-formed but procedurally empty videos pass the gates.

• Completion drives the drop from Level 1 to Level 2. MiniMax H3 under GPT-5.6 Sol goes from S_core 82.6 / completion 58.4 / Overall 53.6 at Level 1 to S_core 77.0 / completion 45.6 / Overall 39.2 at Level 2. Weak models are far worse: LTX 2.3 drops from Overall 22.1 at Level 1 to 22.1 pooled overall but only 0.70 Level 2 overall under human evaluation.

• Three models form a stable top tier. MiniMax H3, Seedance 2.0 and Wan 3.0 stay in the top three across all six evaluation views (Level 1, Level 2, Overall × human and VLM), with mean ranks of 1.33, 2.00 and 2.67 respectively (T0); Wan 2.2 and Cosmos3 Super form T1 (4.67, 5.17) and the remaining three form T2.

• The best model differs by level and metric. MiniMax H3 leads on Level 1 (81.6 core, 53.6 overall under GPT-5.6 Sol), while Wan 3.0 has the highest Level 2 core score (80.3) and Seedance 2.0 the highest Level 2 Overall (79.6) under the VLM evaluator; under human evaluation MiniMax H3 leads overall (44.3 pooled), ahead of Seedance 2.0 (43.0) and Wan 3.0 (37.3).

• VLM evaluation is more permissive than humans. Human pooled Overall scores are much lower for weaker models — LTX 2.3 falls from 22.1 to 3.00 and Cosmos3 Nano from 19.4 to 5.80 — while trends and top-tier membership are preserved.

• Human–VLM agreement is strong after pooling. Pearson r = 0.903 (Level 1), 0.942 (Level 2), 0.980 (pooled Overall); Spearman ρ = 0.904, 0.523 and 0.928; MAE 9.54, 10.77 and 9.35; maximum rank shift 2, 4 and 2; top-3 overlap 3/3 in all settings.

• Difficulty is spread across the benchmark, not one category. Level 2 scores decline across multiple operation domains and disciplines. MiniMax H3 is strong in instrument operations, solid handling, agronomy and chemistry; Wan 3.0 leads in liquid handling, measurement and physics; Seedance 2.0 excels in container operations, heating/cooling and materials science; Wan 2.2 is strongest in filtration/separation.

• Three recurring failure modes. (1) Procedural omission and misordering — required operations skipped, reordered, or applied to the wrong object; (2) incomplete state realization — actions performed without the required observable consequence (liquid-level change, instrument reading, final configuration); (3) object drift — objects deforming, disappearing, merging, or becoming untrackable, plus occasional scene discontinuities.

Methodology in Plain English

The team collected real laboratory procedure videos from two video platforms (Bilibili, YouTube) and two existing datasets (FineBio, ExpVid). Each candidate video was manually screened for four things: an operator-centered view that keeps the manipulation region visible, genuine laboratory relevance with visually verifiable steps, clear continuous footage with stable lighting, and safety compliance.

Each retained video was then annotated at two temporal granularities. Level 1 clips capture a single experimental step of 1–3 atomic actions with a nominal 4-second generation target; Level 2 clips capture short procedures of 4–10 atomic actions with at least one inter-step dependency and a nominal 12-second target. Reference clips average 4.5 s (median 4.3 s) for Level 1 and 12.7 s (median 12.0 s) for Level 2. The benchmark splits into 101 Level 1 and 103 Level 2 tasks; by discipline there are 73 physics, 51 chemistry, 44 biology, 23 agronomy and 13 materials-science tasks, and by operation domain 60 instrument operations, 59 liquid handling, 42 solid handling, 42 container operations, 33 measurement, 8 filtration/separation and 7 heating/cooling (multi-label, so counts do not sum to 204).

For each clip, Qwen3.6-27B drafted a structured task specification from a template, which human annotators then corrected: objects and roles, spatial relations, ordered actions with step index, acting hand, verb, target, contact type, and visible pre- and post-action states. After verification the specification was frozen and Qwen3.6-27B converted it into the natural-language instruction given to the video generator, deliberately abstracting away reference-specific details like hand choice and exact motion trajectories.

Evaluation uses per-task checklists generated by DeepSeek-v4-flash (which sees only the instruction, task ID and difficulty level — no videos) and then verified by domain experts, yielding 3,229 items each labeled Critical, Standard or Supplementary. Each item gets a verdict of Yes, No or Unjudgeable, where No and Unjudgeable both count against the model. Violations are weighted (1, 0.5, 0.25), summed per dimension, and converted to a dimension score by inverse decay, S_d = 1/(1+V_d). The four core dimensions are combined by geometric mean into S_core. Completion C(n_c) is 1, 0.5 or 0 for 0, 1 or ≥2 non-Yes Critical items, and scene consistency and visual safety act as binary gates, giving S_overall = S_core · C(n_c) · G_scene · G_safety.

All 8 models received the same initial frame and verified instruction, with one sample per model–task pair. Automated evaluation used GPT-5.6 Sol on video sampled at 4 FPS; human evaluation applied the same frozen checklists.

Why This Matters

Impact on research. The paper separates "looks real" from "teaches correctly," which most video-generation benchmarks do not. Table 1 shows that prior scientific video benchmarks (VideoPhy-2, VideoScience-Bench, Physics-IQ, PhyGround, Morpheus, Sci-VBench) test physical or scientific unfolding but not human-executed procedures in a real lab with reference executions; LabInstruct is presented as the only one checked in all three columns (real lab, reference executions, human procedures). This gives the field a measurement target for procedural fidelity.

Real-world applications:

• Human-in-the-loop lab automation: generating visual guidance that a researcher can follow when a robotic system cannot execute a plan.
• Protocol training and onboarding: turning a written protocol plus a workspace photo into a step-by-step visual demonstration for new lab members.
• Safety and compliance checking: using the visual-safety gate to flag generated instructions that depict unsafe handling or omitted protective measures.
• Model development feedback: failure attribution by omission/misordering, incomplete state realization, or object drift points to specific capabilities (temporal ordering, contact modeling, long-horizon state tracking) that need improvement.

Industry relevance. For self-driving-lab and lab-automation vendors, the results are a caution: near-perfect scene and safety gate scores coexisted with pooled Overall scores as low as 19.4 under GPT-5.6 Sol and 3.00 under human evaluation. Deploying generated video as standalone instruction is not supported by these results; a human-verified loop is.

Future Directions

• Closing the completion gap on long procedures. Level 2 (4–10 atomic actions, 12 s target) is where completion collapses; whether longer context, explicit step conditioning, or intermediate state supervision is needed remains open.

• Reducing the human–VLM gap. Human judges were substantially stricter on weak models (LTX 2.3: 22.1 → 3.00 pooled Overall). Better automated judges, or calibration of the completion factor, would make large-scale evaluation match expert judgement.

• Extending beyond short, observable procedures. The authors note the filtering deliberately focuses on short, observable, verifiable procedures rather than all laboratory conditions, equipment configurations and procedure types — broader coverage is left open.

• Understanding generation randomness. The re-sampling analysis (Appendix C.4.4) shows exact ordering among top models varies with generation randomness, so stability of tier assignments versus intra-tier ordering needs further study.

Target Audience

Researchers in video generation and embodied AI who need evaluation beyond visual fidelity; self-driving-laboratory and lab-automation engineers considering generated video as a human interface; benchmark designers interested in procedure-aware, checklist-based scoring with validity gates; and human–AI evaluation researchers studying agreement between VLM judges and expert humans.

Authors’ abstract

Self-driving laboratories (SDLs) aim to automate the full experimental loop, from scientific decision-making to physical execution. Ideally, AI-generated plans could be carried out directly by robotic systems, but reliable automation remains difficult in complex, open-world laboratory environments, where experiments often involve fine-grained manipulation, long-horizon procedures, and substantial variation across tasks and setups. Humans therefore remain an important execution interface between AI-generated plans and physical experiments, creating a need for clear and effective human-facing experimental guidance. Because laboratory procedures are inherently visual, spatial, and dynamic, video is particularly well suited to communicating apparatus configurations, manipulation actions, temporal dependencies, and state changes. Recent advances in video generation now make it possible to synthesize experimental demonstrations directly from an initial workspace image and a natural-language instruction. However, whether such models can reliably communicate real laboratory procedures has not been systematically studied. We introduce LabInstruct, a benchmark for situated instructional video generation in real laboratories. LabInstruct contains 204 tasks across 5 scientific disciplines, with real reference executions and structured annotations of objects, actions, contacts, and state transitions. Evaluating 8 frontier image-to-video models, we find that visually plausible generations frequently remain procedurally incorrect, revealing a substantial gap between visual realism and the reliability required for experimental instruction.