The Pulse
BEAKER Preprint Finds Multimodal Models Struggle to Ground Actions in Chemical Labs
A preprint introduces BEAKER, a benchmark for testing multimodal models in self-driving chemical laboratories. Its evaluation finds that models can follow coarse experimental procedures in videos but struggle with action grounding, laborato
AI.info Team · Written with AI · fact-checked ·
Current models can follow coarse experimental procedures in videos, but they still struggle to ground actions into operable regions and feasible trajectories. Those are findings from BEAKER, a preprint introducing a benchmark for multimodal large language models considered as embodied brains for self-driving chemical laboratories. The preprint was submitted to arXiv on 4 November 2026, and the arXiv record says it was accepted by NeurIPS 2026 in the Track on Evaluations and Datasets as a Spotlight.
Self-driving laboratories and embodied automation
Self-Driving Chemical Laboratories, abbreviated SDCLs, are moving chemical experimentation toward embodied automation. Machines in these systems must perceive laboratory scenes, reason about experimental states and act within physical, procedural and safety constraints. The preprint describes multimodal large language models as increasingly considered for the role of embodied brains in such systems.
Existing laboratory benchmarks mainly address safety, anomaly detection or general scientific understanding, the authors write. They say execution-oriented embodied capabilities have received less attention. In a chemical experiment, recognizing a flask, liquid level or connected tube is not enough by itself: the system also needs to judge whether apparatus is ready to operate, whether a connection supports the next step, and whether a change indicates normal progress or a possible failure.
A benchmark of expert-curated tasks
BEAKER contains 1,000 question-answer samples across 11 image and video task types: seven based on images and four on video. The benchmark is designed to test judgments needed for laboratory execution, including affordance grounding and trajectory planning; recognition of equipment, experimental states and safety; apparatus setup; operations and step reasoning; and temporal procedure reasoning.
The questions draw on more than 10,000 publicly available real-world images and videos related to chemical laboratories. Most samples come from first-person or experiment-execution-related viewpoints and cover a range of laboratory scenes and workbenches. Chemistry experts and embodied artificial intelligence experts jointly selected, reviewed and annotated the samples. The questions were written by human experts for chemical-laboratory execution contexts, without AI tools for sample selection, question generation or annotation.
Models tested in an offline setting
The authors evaluated 28 multimodal models, spanning general-purpose, science-oriented, chemistry-specialized and embodied systems. The evaluation used an offline, single-turn setting. The preprint says models can follow coarse experimental progress in videos, but have difficulty making more precise judgments needed for execution.
Reported weaknesses include grounding actions in operable regions and feasible trajectories, recognizing fine-grained laboratory apparatus and states, and identifying safety norms. The models also struggled to order steps in mid-length procedures. The results distinguish following a broad procedure from interpreting the visual and temporal details needed to carry out laboratory work safely and correctly.
A diagnostic for future work
The authors write: “BEAKER thus reveals a clear gap between coarse procedural understanding and the precise, action-grounded perception required for laboratory execution, providing a diagnostic benchmark for future MLLMs in SDCLs.” The benchmark is intended to assess those execution-oriented capabilities rather than only general scientific understanding or object recognition. Its task categories cover perception of possible actions, knowledge of equipment and states, understanding of experimental operations, and reasoning about temporal changes and procedure outcomes.