RxnMiner: A Multimodal Agent for Structured Reaction Data Extraction from Chemistry Figures

AI summary

Overview

Research area: Artificial intelligence for chemistry — multimodal agent systems for optical chemical structure recognition and automated reaction data extraction from scientific literature.

Technical level: Advanced. The paper assumes familiarity with multimodal LLM agents, tool calling, SMILES/E-SMILES molecular representations, and reaction informatics.

Scope: The paper presents RxnMiner, an agent that converts chemistry figures into complete, machine-readable reaction records, together with ChemRxn-Bench, a 603-figure benchmark for evaluating this task.

What This Paper Is About

Chemical reaction data train models for reaction prediction, condition recommendation, and retrosynthesis, but much of it is locked inside figures in journal articles, where a single scheme, table, and footnote may jointly encode dozens of experiments. Reading those figures correctly requires not just recognizing molecules, but working out which evidence belongs to which experiment and rebuilding molecules that are implied by a shared template rather than drawn. The paper builds an agent that performs this extraction end to end and a benchmark that scores systems on the exact molecular identity and reactant–product pairing of each recovered record.

Key Contributions

1. RxnMiner, a multimodal extraction agent. It combines expert-informed extraction instructions with a chemistry toolbox to produce source-linked reaction records. Each record carries explicit provenance marking whether a structure was read from the figure, derived from a template, or written by the model. With a locally deployed Qwen3.8-Flash-Next backbone it reaches 71.0 stereochemistry-aware reaction macro-F1, versus 21.0 for the same model under direct prediction.

2. ChemRxn-Bench. A benchmark of 603 reaction figures containing 8,547 manually annotated and expert-reviewed concrete reaction records. It defines one figure-level extraction task and ranks systems by macro-averaged reaction F1, requiring exact agreement on reactant and product molecular identities, including stereochemistry, within each matched record. Conditions and outcomes are scored separately by a multimodal judge.

3. A silver reaction corpus. Applying the agent without human intervention to 10,107 figures yields 88,202 source-linked reaction records — approximately 3.6 times the 24,716 reaction annotations associated with the full source collection.

4. Open resources. Code (GitHub, project website), the benchmark on Hugging Face with access granted on request, and the silver corpus openly on Hugging Face.

Main Findings

• Existing extraction systems perform poorly on exhaustive recovery. The strongest, ChemEAGLE, reaches 25.0 reaction macro-F1 (precision 25.1, recall 49.7). OpenChemIE and MERMaid both reach 6.0; RxnScribe and MinerU.Chem 2.0 each; RDE2 1.0; RxnCaption and RxnID 0.0.

• Vanilla models leave large headroom. The primary locally deployed backbone, Qwen3.8-Flash-Next, reaches 21.0 as a vanilla LLM. Kimi K3 reaches 40.0 as a vanilla LLM, outperforming all evaluated existing extraction systems.

• RxnMiner roughly triples the primary backbone's score. With Qwen3.8-Flash-Next it reaches 71.0 reaction macro-F1 (precision 69.9, recall 89.2), the highest among evaluated configurations, gaining on both precision and recall simultaneously.

• Gains hold across all five backbones. Over Bare Pi, gains range from 9.4 to 25.5 percentage points: Qwen3.8-Flash-Next 59.6 to 71.0 (+11.4), DeepSeek V4.1 Flash 54.6 to 64.0 (+9.4), GLM 5.3 Flash 15.5 to 41.0 (+25.5), Kimi K3 57.7 to 69.0 (+11.3), Doubao Seed 2.1 Pro 26.5 to 47.0 (+20.5).

• The base agent alone is not enough. Bare Pi underperforms vanilla LLM prediction on GLM (15.5 vs 31.0) and Doubao (26.5 vs 34.0), whereas RxnMiner improves over both alternatives.

• Stereochemistry remains the hard part. With the Qwen backbone, RxnMiner scores 78.0 without stereochemistry versus 71.0 with it — a seven-percentage-point gap that identifies stereochemical agreement as an unresolved challenge despite the verification procedures.

• The benchmark skews hard. Difficulty tiers defined by the baseline's stereo-aware exact precision place 435 of 603 figures (72.1%) in the very-hard tier (A < 0.2), and 78 (12.9%) in very easy (A ≥ 0.8). Median reference records per figure is 10, maximum 82.

• Corpus scale is real but unvalidated. RDKit parsed 25,223 of 25,225 molecular strings (99.99%) in a random sample of 1,000 figures, indicating high string parseability — but the paper states this does not establish agreement with source figures or correct reactant–product pairing, and the corpus has not undergone manual validation.

Methodology in Plain English

The system is built on Pi, an open-source agent framework that supplies the agent loop, model interface, and tool calling. On top of it, the authors add two things: written instructions encoding how an expert chemist reads a reaction figure, and a set of chemical tools.

The workflow follows three stages. Recognition comes first: a molecular recognition tool (MolParser) reads structures and templates from the image into Extended SMILES, along with bounding boxes and confidence scores, and these readings are cached and reused. If a whole-figure pass misses a structure, the model picks a crop and asks for another recognition call. The model also reads compound identifiers, substituent assignments, conditions, and footnotes.

Association is where the language model's reasoning does the work: for each experimental entry it builds a plan naming which molecular readings play reactant and product roles, which shared templates apply, and how substituent labels bind to template positions. Conditions and outcomes are attached to the same entry using identifiers, table rows, and footnotes.

Reconstruction turns partial specifications into complete molecules. Drawn participants keep their tool readings; participants specified indirectly are instantiated by a deterministic R-group engine using explicit bindings, product-template matching, or both. When reactants are undrawn, the engine matches the recognized product to its product template, recovers the substituent bindings, and applies them to the reactant templates. Printed assignments are also used, because a leaving group replaced during the reaction cannot be recovered from the product alone. Stereochemistry-bearing structures are checked and rendered with RDKit for visual comparison, and a fallback written by the model is explicitly flagged. Assembly then produces the final reaction records with conditions, outcomes, and provenance.

For evaluation, predicted and reference structures get the same normalization, and records are matched one-to-one within a figure, requiring exact agreement on both reactant and product sets including stereochemistry. Failed runs are retained and receive zero credit rather than being dropped.

Why This Matters

Impact on research. Reaction prediction, condition recommendation, and retrosynthetic planning all depend on curated reaction records, and the paper points to AbSynth — which relied on undergraduate students over several years to assemble more than 3,000 total syntheses spanning close to 60,000 steps — as evidence of how expensive manual transcription is. A common evaluation protocol also matters because prior systems were evaluated on heterogeneous datasets, schemas, and matching rules, making reliability comparisons impossible.

Real-world applications:

• Expanding training data for reaction models by converting published figures into machine-readable records at a scale manual curation cannot match.
• Reaction condition recommendation, since records retain extracted conditions, reagents, catalysts, time, and outcomes.
• Retrosynthetic planning and synthesis strategy analysis, where multi-step schemes are decomposed into individual steps with conditions attached to each.
• Prioritizing manual curation effort, using extracted records as candidates for future expert-verified benchmark annotation.

Industry relevance. Pharmaceutical and fine-chemical organizations publish and consume vast amounts of figure-encoded chemistry, including optimization tables and substrate scopes that map directly onto process development work. An agent that produces source-linked, provenance-tagged records gives organizations a way to mine their own figure archives while keeping each record traceable back to its source region.

Future Directions

• Closing the stereochemistry gap. The seven-point difference between stereochemistry-aware (71.0) and stereochemistry-blind (78.0) scoring persists even with RDKit-based verification, and the paper treats stereochemical agreement as a remaining challenge.
• Validating the silver corpus. The 88,202 records have not been manually verified; the reported 99.99% parseability figure checks only string validity, not fidelity to the source figure or correct reactant–product pairing, so source-level auditing is the obvious next step.
• Diagnosing errors by capability. The paper explicitly notes that its taxonomy does not assign a unique cause to a system's errors, and that the seven extraction operations are not scored independently — a per-operation breakdown would show where the workflow breaks down.
• Broadening backbone and judge coverage. Results come from five backbones, and the secondary assessment of conditions and outcomes relies on a single multimodal judge (Claude Opus 5.5), whose sampled assessments were judged correct more than 90% of the time in an expert audit, with validation limitations acknowledged in the appendix.

Target Audience

Researchers and engineers in AI for science and cheminformatics building extraction pipelines or reaction datasets; chemists and data curators who need to convert literature figures into structured reaction records; benchmark designers interested in exact-match, reaction-level evaluation protocols; and teams in pharmaceutical or chemical software organizations evaluating whether agentic systems can replace manual figure transcription. Readers without background in molecular representations or multimodal agents will find the paper demanding.

Authors’ abstract

Reaction data underpin machine learning for reaction prediction, condition recommendation, and retrosynthetic planning, yet extracting these data from literature figures still requires substantial manual effort. Many figures encode multiple experiments through shared templates, substituent tables, and product grids. Recovering concrete reaction records therefore requires molecular recognition, association of evidence with individual experimental entries, and reconstruction of indirectly specified participants. We present RxnMiner, a multimodal agent system that combines an expert-informed extraction procedure with a chemistry toolbox to coordinate these operations. The agent reads molecular structures, resolves shared and entry-specific evidence, and instantiates participants from templates to assemble source-linked reaction records. We also introduce ChemRxn-Bench, a benchmark of 603 reaction figures with 8,547 manually annotated and reviewed reaction records, evaluated under a common protocol for exact molecular identity and reactant-product pairing. On this benchmark, with Qwen3.8-Flash-Next as the backbone, RxnMiner raises stereochemistry-aware reaction macro-F1 from 21.0% under direct prediction to 71.0%, compared with 25.0% for the strongest existing extraction system. Across five backbones, it also outperforms the same base agent without the extraction procedure and chemistry toolbox by 9.4 to 25.5 points. Applied without human intervention to 10,107 literature figures, RxnMiner produces a silver corpus of 88,202 source-linked reaction records.