SynRXN: An Open Benchmark and Curated Dataset for Computational Reaction Modeling

AI summary

Overview

Research area: Computational chemistry and machine learning for computer-aided synthesis planning (CASP), specifically benchmark and dataset construction for reaction informatics.

Technical level: Intermediate.

Scope: SynRXN is a unified, open benchmarking framework and curated, provenance-tracked dataset collection that decomposes end-to-end synthesis planning into five task families and supplies leakage-aware splits, standardized metrics, and reproducible build recipes for each.

What This Paper Is About

Machine learning for chemistry has advanced quickly for molecules, partly because resources such as MoleculeNet and the Therapeutics Data Commons provide shared data and evaluation protocols. Reaction informatics, by contrast, has remained fragmented: studies rely on bespoke subsets, inconsistent preprocessing, and opaque train/test splits, which makes cross-paper comparisons nearly impossible. The authors introduce SynRXN, a unified benchmarking suite that curates reaction corpora from heterogeneous public sources into a harmonized representation, packages them as versioned datasets with machine-readable manifests and checksums, and provides deterministic splitting functions and standardized evaluation workflows for five task families spanning the full CASP pipeline.

Key Contributions

1. A five-task decomposition of CASP for isolated benchmarking. SynRXN splits end-to-end synthesis planning into reaction rebalancing, atom-to-atom mapping, reaction classification, reaction property prediction, and synthesis route design, each released as its own benchmark module with curated data, splitting functions, and tailored metrics.

2. Curated, provenance-tracked, license-tagged corpora with machine-readable manifests. Data are assembled primarily from USPTO-derived sources into a harmonized representation. Every release ships a top-level manifest.json enumerating file paths, cryptographic checksums, row and column counts, column names, descriptions, and file-level license tags, so provenance and filtering outcomes are reproducible.

3. Leakage-aware, deterministic splitting plus held-out gold standards. Transparent splitting functions paired with versioned manifests and RNG seeds (seed = 42) recreate identical train/validation/test partitions without shipping massive index files. Sensitive tasks — reaction rebalancing and atom-to-atom mapping — are distributed only as evaluation sets and are explicitly not intended for model training.

4. Scripted build recipes, reference baselines, and CI-enabled regression checks. Bitwise-reproducible regeneration of all corpora is supported across machines and over time, and reference baselines using DRFP and RXNFP embeddings with off-the-shelf RandomForest models give stable comparison points. The resource is released under permissive open licences (the release is on PyPI and archived on Zenodo, DOI 10.5281/zenodo.17297258; the repository is mirrored at https://github.com/TieuLongPhan/SynRXN).

Main Findings

• Reaction rebalancing test set reflects curation, not automatic accuracy. The test split was manually inspected and only verified corrections retained; consequently SynRBL attains 100% success rate and 100% exact match accuracy on the released test set, which the authors explicitly attribute to manual selection of reliable labels rather than unconstrained automatic performance. Two metrics are reported: success rate (fraction of inputs whose rebalanced output is element-wise stoichiometrically balanced) and exact match (fraction whose canonical reaction SMILES equals the curated ground truth).

• Atom-mapping accuracy varies strongly by mapper and by domain. Across five strictly held-out datasets, accuracies were: RXNMapper 0.4.1 — EColi 72.53, Recon3D 48.69, USPTO_3K 93.53, Golden 87.43, NatComm 87.58; Graphormer (built with Cython 1.7.8) — 42.12, 34.82, 95.10, 89.59, 92.87; LocalMapper 0.1.5 — 69.96, 50.79, 97.77, 89.08, 92.67; RDTool 2.4.1 — 78.02, 54.97, 90.87, 82.54, 84.11. Biochemical datasets (EColi, Recon3D) are markedly harder than the chemistry sets for all tools.

• Classification baselines are strong but not uniform across label granularity. Using DRFP and RXNFP embeddings with a scikit-learn RandomForest under 5×5 stratified cross-validation (seed = 42), weighted F1 and multiclass MCC exceed 0.9 on most datasets. Examples: Schneider_U with DRFP reaches F1 0.968 ± 0.002 and MCC 0.968 ± 0.002; ECREACT 1st level with DRFP reaches F1 0.977 ± 0.001; on the finest SynTemp level (R2) DRFP reaches F1 0.913 ± 0.003 versus RXNFP 0.737 ± 0.004.

• DRFP generally outperforms RXNFP on classification, with exceptions. RXNFP has the better mean on USPTO_50k_U (F1 0.958 ± 0.002 versus DRFP 0.953 ± 0.002), while DRFP leads on the finer-grained and biochemical label sets. Most differences are reported as statistically significant at p < 0.0001.

• Property prediction is learnable, with task-dependent difficulty. DRFP and RXNFP with a RandomForestRegressor under repeated cross-validation (5 repeats of 5 folds, 25 runs) or, for RGD1, five seeds on a single split, yield single-digit to low-tens MAE. Examples: E2 activation energy MAE 3.247 ± 0.206 (DRFP) versus 7.377 ± 0.354 (RXNFP); E2SN2 4.150 ± 0.126 versus 7.116 ± 0.133; SNAr 1.402 ± 0.158 versus 1.447 ± 0.139 (not significant); RDB7 30.136 ± 0.210 versus 18.812 ± 0.240; RGD1 16.704 ± 0.074 versus 15.953 ± 0.032 (not significant); LogRate 1.054 ± 0.068 versus 1.077 ± 0.059 (not significant). Relative rankings therefore depend on the endpoint.

• Synthesis planning is scaffolded, not reimplemented. The authors supply datasets, lightweight data loaders, and split/evaluation scripts, but do not reimplement full-scale planners because of computational and dependency constraints. Instead they formally define coverage, recognition rate, F_beta, top-K accuracy, max-fragment overlap, and round-trip success, and refer to published results in Han et al. (USPTO_50k, USPTO_MIT) and Lu & Zhang (USPTO_500).

• Dataset inventory is large and heterogeneous. Rebalancing: MNC 33,147, MOS 12,781, MBS 491, Complex 1,748 (a Complex set of 1,892 examples was curated from manually validated Golden and Jaworski collections, with 1,748 in the final compendium). Atom mapping: Golden 1,785, NatComm 491, USPTO_3K 3,000, totaling 5,276 small-molecule reactions, plus biochemical Recon3D 382 and EColi 273, totaling 655. Classification includes Schneider_U/Schneider_B (50,000 each, 50 classes), USPTO_TPL_U/USPTO_TPL_B (445,115 each, 1,000 classes), USPTO_50k_U/USPTO_50k_B (50,016 each, 10 classes), SynTemp R0/R1/R2 (43,441 each; 143, 356, and 680 classes), and ECREACT_1st/_2nd/_3rd (185,734 each; 7, 63, 175 classes). Property prediction spans B97XD3 (16,365), SNAr (503), E2SN2 (3,625), Rad6Re (31,923), LogRate (778), Phosphatase (33,354), E2 (1,264), SN2 (2,361), RDB7 (23,852), CycloAdd (5,269), and RGD1 (353,984). Synthesis planning uses USPTO_50k (50,016), USPTO_MIT (479,035), and USPTO_500 (143,535).

• A standardized data format is enforced across tasks. Records live under a top-level Data/ directory partitioned into rbl, aam, class, prop, and synthesis subdirectories; datasets are gzip-compressed UTF-8 CSVs with one reaction per line and empty fields for missing values. Core columns are r_id (stable record identifier), rxn (canonical unmapped reaction SMILES), aam (atom-mapped reaction SMILES when available), plus label (integer class codes) or property (floating-point values such as activation energy ea, enthalpy dh, or logarithmic rate lograte).

Methodology in Plain English

The authors assembled reaction data from public repositories and community benchmarks — primarily the patent-extracted USPTO collection, supplemented by USPTO_50K and larger USPTO compilations — while retaining full provenance and licensing metadata in a task-specific manifest. All records pass through an automated, deterministic canonicalization and chemical-sanity pipeline implemented in SynKit (the Standardize and CanonRSMI modules), which normalizes charges and valence states, standardizes aromaticity, and enforces consistent SMILES canonicalization. Duplicate detection is automatic, using both exact SMILES matches and structure-level isomorphism checks; failing records (invalid SMILES, unparsable fields, impossible atom counts, inconsistent valence) are excluded without manual correction.

For rebalancing, targeted perturbation sets were derived from USPTO_50K representing three modes of stoichiometric violation — MNC (Missing Non-Carbon), MOS (Missing One Side), and MBS (Missing Both Sides) — plus a Complex set for skeletal rearrangements. Corrections were generated with SynRBL, a hybrid rule- and graph-based completion algorithm, and only corrections resolved with confidence of at least 90% were retained as ground truth.

For atom mapping, the corpus was stratified into small-molecule and biochemical domains, standardized, and evaluated by two criteria: graph isomorphism of Imaginary Transition State (ITS) graphs, and exact-match accuracy of canonicalized reaction SMILES verified via SynKit. Four mappers were run on five held-out datasets.

For classification and property prediction, the approach is deliberately simple and reproducible: reactions are embedded with DRFP or RXNFP, and a RandomForest classifier or RandomForestRegressor from scikit-learn is trained on those fixed features. Evaluation uses repeated stratified cross-validation (5 repeats of 5 folds), preserving class proportions across folds, with weighted F1 and multiclass MCC for classification and MAE/MSE for regression; the large RGD1 dataset uses a single split repeated with five random seeds. Statistical comparisons follow the procedure of Ash et al.

Why This Matters

Impact on research. The paper argues that reaction informatics lacks the standardized ecosystem that molecular machine learning gained from MoleculeNet and the Therapeutics Data Commons. Broad aggregation efforts such as the Open Reaction Database solve the data access problem but do not provide the unified benchmarking layer needed to evaluate the full CASP pipeline. By fixing preprocessing, splitting, and metrics, SynRXN aims to make head-to-head comparisons of methods reproducible across papers and over time, and to enable rigorous ablations and stress tests along the pipeline.

Real-world applications.

• Pharmaceutical and fine-chemical route design, where retrosynthesis modules must generalize beyond the curated corpora they were trained on.
• Reaction database curation in industry, where automated rebalancing and atom mapping are needed to repair patent- and ELN-derived records before they can train downstream models.
• Reaction yield, rate, and barrier prediction for experimental prioritization, using the property-prediction benchmarks covering quantum-chemistry and high-throughput experimental targets.
• Enzyme and biocatalysis annotation, supported by the EColi, Recon3D, and ECREACT datasets with hierarchical EC number labels.

Industry relevance. Deterministic splits, checksummed manifests, and CI-enabled regression checks lower the barrier for practitioners who need robust and comparable performance estimates for real-world synthesis planning workloads, and reduce the hidden cost of re-curating and re-splitting data for each internal project.

Future Directions

• Full-scale synthesis planners. The authors explicitly did not reimplement planners due to computational and dependency constraints, leaving the standardized coverage, recognition rate, F_beta, top-K, max-fragment overlap, and round-trip success metrics to be applied by others; broader harmonization across single- and multi-step settings remains necessary.

• Improving atom mapping on biological data. Biochemical datasets (EColi 78.02% at best, Recon3D 54.97% at best across the four tools) remain far below chemistry-set accuracy, indicating a domain where current mappers fall short.

• Reducing annotation dependence in non-trainable tasks. Rebalancing and atom-to-atom mapping are released only as evaluation sets because they are contamination-prone; a direction of interest is whether methods can be trained and evaluated without leaking the held-out gold standards.

• Handling out-of-distribution transfer. The paper notes limited out-of-distribution transferability for transition-state properties, motivating QM-augmented descriptors and transition-state-based architectures, which the property benchmarks could be used to test systematically.

Target Audience

Machine learning researchers working on molecular and reaction models who need reproducible train/test protocols; cheminformatics and computational chemistry practitioners building CASP pipelines; benchmark and dataset maintainers interested in provenance, licensing, manifests, and deterministic splitting; and industrial cheminformatics teams that need comparable performance estimates for curation, mapping, property prediction, and route design tasks. Readers who want a working resource rather than a new model are the primary audience.

Authors’ abstract

We present SynRXN, a unified benchmarking framework and open-data resource for computer-aided synthesis planning (CASP). SynRXN decomposes end-to-end synthesis planning into five task families, covering reaction rebalancing, atom-to-atom mapping, reaction classification, reaction property prediction, and synthesis route design. Curated, provenance-tracked reaction corpora are assembled from heterogeneous public sources into a harmonized representation and packaged as versioned datasets for each task family, with explicit source metadata, licence tags, and machine-readable manifests that record checksums, and row counts. For every task, SynRXN provides transparent splitting functions that generate leakage-aware train, validation, and test partitions, together with standardized evaluation workflows and metric suites tailored to classification, regression, and structured prediction settings. For sensitive benchmarking, we combine public training and validation data with held-out gold-standard test sets, and contamination-prone tasks such as reaction rebalancing and atom-to-atom mapping are distributed only as evaluation sets and are explicitly not intended for model training. Scripted build recipes enable bitwise-reproducible regeneration of all corpora across machines and over time, and the entire resource is released under permissive open licences to support reuse and extension. By removing dataset heterogeneity and packaging transparent, reusable evaluation scaffolding, SynRXN enables fair longitudinal comparison of CASP methods, supports rigorous ablations and stress tests along the full reaction-informatics pipeline, and lowers the barrier for practitioners who seek robust and comparable performance estimates for real-world synthesis planning workloads.