BEAKER: An Expert-Curated Benchmark for Embodied Brains in Self-Driving Chemical Laboratories

AI summary

Overview

Research area: Multimodal AI evaluation — specifically benchmarks for Multimodal Large Language Models (MLLMs) acting as "embodied brains" in Self-Driving Chemical Laboratories (SDCLs).

Technical level: Intermediate. Readers should be comfortable with vision-language models, question-answering benchmarks, and basic notions of affordance and trajectory prediction; the chemistry content is explained at a graduate-adjacent level but the benchmark results are readable without it.

Scope: The paper introduces BEAKER, a human-expert-curated benchmark of 1,000 question-answer samples across 11 image and video task types, and reports an evaluation of 28 MLLMs on execution-oriented chemical laboratory capabilities.

What This Paper Is About

Self-driving chemical laboratories need models that can look at a lab scene and decide what is physically operable, what state the experiment is in, and what step comes next — not just describe what they see. Existing laboratory benchmarks mostly test safety, anomaly detection, or general scientific knowledge, so it is unclear whether current MLLMs can support actual experimental execution. BEAKER is built to measure that gap, organizing evaluation into four capabilities (Actionability, Knowledge, Experimentation, Reasoning) and testing models offline in a single-turn setting.

Key Contributions

1. A four-dimension evaluation framework for SDCL embodied brains. BEAKER decomposes laboratory execution into Actionability (operable regions, feasible interactions, motion paths), Knowledge (apparatus, reagent states, procedural norms), Experimentation (operations, apparatus connections, step progression), and Reasoning (temporal change, causal and outcome inference), drawing on the sense-plan-act paradigm.

2. A fully human-authored benchmark of 1,000 QA samples across 11 task types. Seven image-based task types (AFF, TRAJ, ERI, ESS, SR, EAS, EOP) and four video-based task types (SVOR, MVTS, TSR, LVPR) were written and reviewed by chemistry and embodied-AI experts with no AI-assisted sample selection, question generation, or annotation, addressing self-bias concerns in LLM-generated benchmarks.

3. A documented expert curation and cross-group blind-review pipeline. Ten chemistry experts and twelve embodied experts screened 11,128 candidate images down to 883, curated 734 video clips, wrote 1,617 expert captions, and produced 1,500 candidate questions of which 500 (33.3%) were removed, leaving 1,000. The full pipeline spans about 1,200 expert-hours, and inter-group annotation consistency was measured (86.9% mean box IoU on AFF, 100.0% bidirectional point-in-region agreement, 85.6/100 mean trajectory agreement on TRAJ).

4. An evaluation of 28 MLLMs plus a human baseline. The study covers general-purpose, science-oriented, chemistry-specialized, and embodied models, scored with expert-defined weights (Actionability 0.25, Knowledge 0.15, Experimentation 0.35, Reasoning 0.25) and supported by aggregation-robustness, confidence-interval, and ablation analyses.

Main Findings

• A large human-model gap. Three chemistry experts uninvolved in benchmark construction average 93.3 Overall. The best MLLM, Claude Opus 4.7, scores 59.2 — 34.1 points lower. The gap widens to 48.0 points for the best embodied brain model, RoboBrain2.5-8B (45.3).

• Actionability is the dominant bottleneck. Averaged over the 28 models, Affordance Grounding (AFF) reaches only 9.7 and Trajectory Planning (TRAJ) 34.7, while the human baseline scores 88.0 and 91.0 respectively. Averaged over the 13 representative models in the capability profile, Actionability is lowest at 21.9, versus Experimentation at 61.2 and Reasoning at 56.7.

• Fine-grained laboratory perception is weak, but coarse procedure following is easier. ERI, ESS, and SR average only 40.0–43.9 against a 30.4% random-choice baseline for image MCQs, while human experts score at least 93.3 on each. In contrast, video tasks do better: SVOR averages 65.7 and MVTS 56.5.

• Mid-length video step ordering is the hardest temporal task. Within TSR, M-TSR averages 37.1, compared with 68.8 for S-TSR and 57.0 for L-TSR, because step boundaries in mid-length clips are less visually salient.

• Specialization does not reliably help. Science-oriented and chemistry-specialized models do not consistently outperform general-purpose MLLMs. ChemVLM-8B scores 28.4 and ChemVLM-26B 30.1 Overall, Intern-S1-Pro 44.0, Intern-S1-Mini 35.2, and Intern-S1 40.5 — all below the top general models. Relative to Qwen3-VL-8B (47.1), the best embodied comparator RoboBrain2.5-8B reaches 45.3, gaining on TRAJ (+4.08) and TSR (+1.11) but losing elsewhere.

• Embodied-design variants trade off rather than improve. Against RynnBrain-8B (39.2), RynnBrain-CoP-8B raises AFF sharply (28.6 vs 3.2) but scores 42.9 Overall, and RynnBrain-Plan-8B gains on EOP (60.0 vs 43.3) yet drops on TRAJ and MVTS, ending at 37.4.

• Coordinate output is far harder than multiple choice. Across five representative models, the Coord./MCQ ratio is at most 0.7. Qwen3.5-397B-A17B scores 62.2 on MCQ but 20.6 on coordinate QA; RoboBrain2.5-8B scores 52.2 and 16.6. Even the strongest coordinate performers, Claude Opus 4.7 (41.7) and GPT-5.5 (40.0), reach only a 0.7 ratio, while Fill-in-the-Blank ratios approach 1.0.

• Removing images consistently hurts, unevenly. On 300 image MCQs, text-only input lowers accuracy for all eight tested models; gains from images range from +2.33 points (ChemVLM-8B) and +5.67 (Intern-S1) to +23.00 (Qwen3-VL-8B) and +20.67 for both GPT-5.4 and Qwen3.5-397B-A17B.

• Expert captions substantially boost video performance. Adding expert-written captions to video inputs improves M-TSR, L-TSR, and LVPR, with the largest effect on M-TSR, where captions roughly double the scores of all three tested models — implying the difficulty lies in extracting step boundaries from frames rather than ordering described steps.

• Results are stable under reaggregation. Four alternative aggregation schemes give Spearman ρ ≥ 0.980 and Kendall τ ≥ 0.889 with identical top-5 and top-10 sets; task-stratified bootstrap 95% confidence intervals of Overall scores span at most 5.5 points, with 95% rank ranges of at most seven positions.

Methodology in Plain English

The authors first gathered raw visual material. For images, they surveyed public chemical-laboratory datasets and identified 12 representative sources; chemistry experts screened 11,128 candidate images and kept 883, split into 275 "simple" images (one apparatus or reagent on a clean background) and 608 "complex" ones (cluttered scenes, multi-step setups, active experimenters). For video, they took 334 four-second egocentric clips from the IBM Lab Actions dataset, 200 long JoVE Chemistry recordings averaging about 9 minutes with expert transcripts, and sliced the long recordings into 1,753 candidate mid-length segments using FFmpeg, Whisper timestamps, and step-boundary detection — of which experts kept 200 averaging about 50 seconds. Whisper was used only to propose segment boundaries, not to select samples or author questions.

Ten chemistry experts then wrote free-form captions for every retained item, describing apparatus, reagent states, operations, and safety details, producing 1,617 captions. Annotation was organized into ten interdisciplinary groups, each pairing one chemistry expert with one embodied-AI expert, and each assigned a quota of 150 questions per task-type matrix. Each group's questions were then blind-reviewed by two other groups on scientific correctness, embodied relevance, difficulty calibration, and answer unambiguity, with duplicate and flawed items sent to consensus adjudication. Questions answerable from surface visual cues, trivially binary, beyond graduate-level chemistry, or lacking embodied grounding were discarded.

The final 1,000 samples come from 410 images (152 simple, 258 complex) and 195 short, 125 mid-length, and 80 long videos. Two spatial tasks use coordinate annotations made in CVAT following RoboBrain 2.0's point-and-box protocol (150 each), five image tasks use 60 multiple-choice questions each, and video tasks use 150 TSR, 125 SVOR, 75 MVTS, and 50 fill-in-the-blank LVPR items.

Evaluation is offline and single-turn with standardized prompts. Models ran through remote APIs or locally on 16 NVIDIA A100 GPUs, with temperature set to 0.7 where configurable and images at original resolution; short and mid-length videos were represented by 8 and 32 sampled frames, and long videos by 128 frames arranged as 32 four-frame grids. The human baseline averages three chemistry experts who had no role in constructing BEAKER and answered all 1,000 questions without ground truth.

Why This Matters

Impact on research. BEAKER separates "understanding a procedure" from "knowing what to physically do next," and shows the second is far weaker in current models. That distinction gives model developers a diagnostic target rather than a single leaderboard number, and the fully human-authored construction gives the community a reference point free of the self-bias that arises when LLMs generate and judge their own benchmark items.

Real-world applications:

• Robotic lab platforms that must decide which flask neck, stopcock, or pipette nozzle is safe to grasp or turn.
• Automated experiment monitoring that distinguishes normal state change (precipitate formation, color shift) from a failure that requires intervention.
• Safety and standardization checking of apparatus setups and connections before a protocol step is executed.
• Procedure-tracking assistants that verify an operator or robot is at the correct step in a multi-stage protocol.

Industry relevance. Pharmaceutical, materials, and fine-chemical organizations investing in closed-loop autonomous experimentation need to know whether a vision-language model can be trusted at the actuation layer, not just the documentation layer. The finding that chemistry-specialized models do not beat general models suggests that domain fine-tuning alone is not the missing ingredient — spatial grounding and laboratory-specific perception are.

Future Directions

• Improving action grounding. Coordinate-based performance is far below multiple-choice performance even for the strongest models, so methods that produce precise operable regions and feasible trajectory waypoints in cluttered lab scenes are the clearest open problem.

• Better mid-length video reasoning. M-TSR is the weakest temporal task, and captions roughly double scores, which points to step-boundary detection from raw frames as a specific capability to develop rather than a general video-understanding deficit.

• Closing the loop beyond offline single-turn evaluation. The paper evaluates models offline; extending the benchmark to interactive, multi-turn, or closed-loop execution settings where a model receives feedback from its own actions is a natural next step.

• Understanding specialization failures. The paper notes that differences in backbones and training recipes prevent attributing specialized-model gaps to specialization alone, leaving a controlled study of what domain and embodiment training actually transfer to laboratory execution as an open question.

• Interactive human-in-the-loop and safety-critical deployment. The human-model gap of 34.1 (best MLLM) to 48.0 (best embodied model) points suggests near-term systems need expert oversight; how to design that oversight for SDCL workflows is not addressed.

Target Audience

Researchers building or evaluating multimodal embodied agents, especially those working on scientific automation and laboratory robotics; benchmark designers interested in expert-curated, non-LLM-generated annotation pipelines; MLLM developers who need a diagnostic beyond general-purpose vision benchmarks; and automation engineers in chemistry-facing industry who want a realistic picture of what current models can and cannot do at the execution layer.

Authors’ abstract

Self-Driving Chemical Laboratories (SDCLs) are moving chemical experimentation toward embodied automation, where machines must perceive laboratory scenes, reason about experimental states, and act under physical, procedural, and safety constraints. Although Multimodal Large Language Models (MLLMs) are increasingly considered as embodied brains for such systems, existing laboratory benchmarks mainly focus on safety, anomaly detection, or general scientific understanding, leaving their execution-oriented embodied capabilities underexplored. To fill this gap, we introduce BEAKER, a Benchmark for evaluating Embodied Actionability, Knowledge, Experimentation, and Reasoning in SDCLs. BEAKER contains 1,000 expert-curated question-answer samples across 11 image and video task types, covering key judgments required for chemical laboratory execution, including affordance grounding, trajectory planning, equipment, state, and safety recognition, apparatus setup, operation and step reasoning, and temporal procedure reasoning. All questions and annotations are manually created and reviewed by chemistry and embodied Artificial Intelligence (AI) experts, without AI-assisted generation or annotation. We evaluate 28 general-purpose, science-oriented, chemistry-specialized, and embodied MLLMs in an offline, single-turn setting. Results show that current models can follow coarse experimental procedures in videos, but still struggle to ground actions into operable regions and feasible trajectories, to recognize fine-grained laboratory apparatus, states, and safety norms, and to order steps in mid-length procedures. BEAKER thus reveals a clear gap between coarse procedural understanding and the precise, action-grounded perception required for laboratory execution, providing a diagnostic benchmark for future MLLMs in SDCLs.