Procrustean Bed for AI-Driven Retrosynthesis: A Unified Framework for Reproducible Evaluation

AI summary

Overview

• Research area: Computer-aided synthesis planning (CASP) and machine learning for chemistry, specifically the evaluation methodology for AI-driven retrosynthesis.
• Technical level: Advanced. The paper assumes familiarity with retrosynthetic search, single-step models, Top-K accuracy, and statistical bootstrapping.
• Scope: A one-sentence summary — the authors build a standardized, open-source evaluation infrastructure (RetroCast plus the SynthArena web platform), use it to compare leading search-based and sequence-based retrosynthesis models on new curated benchmarks, and demonstrate that the field's dominant metric, Stock-Termination Rate (STR), is a misleading proxy for chemical validity.

What This Paper Is About

Retrosynthesis models propose sequences of reactions that convert commercially available starting materials into a target molecule, but the field has no shared evaluation infrastructure and relies on a metric ("solvability," renamed here to Stock-Termination Rate) that only checks whether a route's terminal nodes are purchasable, not whether the chemistry along the way makes sense. The authors argue that this measurement gap, combined with incompatible output formats across tools and high computational costs, makes apples-to-apples comparison of models impractical and can reward chemically impossible routes. Their goal is to replace ad-hoc evaluation with a unified, reproducible framework and to expose the divergence between topological "solving" and genuine chemical plausibility.

Key Contributions

1. RetroCast, an open-source evaluation suite with an adapter-based universal translation layer that parses heterogeneous model outputs into a single standardized schema. Ready-to-use adapters are provided for search-based planners (AiZynthFinder, Retro*, ASKCOS, Syntheseus, RetroChimera, DreamRetroEr, MultiStepTTL, SynPlanner), sequence-based models (DirectMultiStep, SynLLaMa), and standard dataset formats such as PaRoutes.
2. A reproducible benchmarking pipeline with stratified sampling by route length and topology, bootstrapped 95% confidence intervals (10,000 resamples), paired bootstrap difference tests, and cryptographic SHA256 manifests at every processing stage to create a verifiable chain of data provenance.
3. Two evaluation tracks with curated benchmarks derived from PaRoutes n5 (and n1 for one benchmark): the Chemist-Aligned (mkt-) series evaluated against the ASKCOS Buyables stock, and the Developer-Aligned (ref-) series evaluated against original PaRoutes patent-derived stocks under a common-training-data restriction. A Multi-Ground-Truth (MGT) protocol expands the reference set to include valid commercially terminated sub-routes.
4. SynthArena (syntharena.ischemist.com), an interactive open-source platform for side-by-side route inspection, difference overlays, and annotation of commercial availability, plus the release of a standardized database of model predictions as a community asset.

Main Findings

• STR is a misleading signal of chemical validity: On the public USPTO-190 benchmark evaluated with ASKCOS Buyables, the original Retro* model achieves a dominant 73.2% STR [66.8, 79.5], but qualitative inspection of the underlying routes reveals chemically implausible intermediate steps that STR cannot detect.
• Rewarding pattern-matching of corrupted data: For target USPTO-082, the top model's "solved" route depends on a chemically implausible seven-reactant step that is an exact reproduction of a flawed transformation in the benchmark's own reference route; newer models that avoid the nonsensical step are penalized.
• A catalog of chemical hallucinations: Five other "solved" routes generated by the same model show apparent violations of fundamental principles, including mass balance errors (un-sourced chloro-phenyl group; un-sourced pyridylmethylamine), implausible transformations (tartaric acid as a propargyl source; amino acids to tryptophan core), and an unspecified reagent (epoxidation with carbonic acid). These cases are indexed by targets USPTO-114, USPTO-169, USPTO-93, USPTO-16, and USPTO-181.
• Stock sets are wildly inconsistent across tools: The authors' survey found that stock sets used for CASP evaluation vary by over three orders of magnitude, from physically in-stock compounds to massive "make-on-demand" virtual libraries (Table S1, values in supplementary material).
• PaRoutes routes are not always commercially grounded: Of the 13,306 unique leaf molecules in the n5 evaluation stock, only 7,513 (56%) appear in the ASKCOS Buyables set; consequently, only 4,279 of the 10,000 reference routes in n5 have all leaves within this realistic commercial stock.
• Divergent architectural signatures under MGT evaluation: On mkt-cnv-160 (convergent routes), search-based models achieve near-perfect STR (Retro* (High) at 100.0%, AiZynF MCTS (High) at 99.4%) but the sequence-based DMS Explorer XL shows substantially higher Top-10 accuracy (57.5% [50.0, 65.0]) versus 33.8% for Retro* (High) and 38.8% for AiZynF MCTS (High). The same inverse profile appears on the linear mkt-lin-500 benchmark, where DMS Explorer XL reaches 55.4% Top-10 accuracy while search-based models range from 16.2% (ASKCOS) to 35.8% (AiZynF MCTS).
• A "complexity cliff" on long routes: Stratifying mkt-cnv-160 by reference route length shows search-based models excel on short routes but decay sharply (AiZynF MCTS Top-10 accuracy falls from 75.0% at length 2 to 15.0% at length 5), while DMS Explorer XL stays more robust (75.0% at length 2, 57.5% at length 3, 45.0% at length 4, 52.5% at length 5). On the ref-lng-84 stress test of exclusively long routes (lengths 8–10), the route-matching accuracy of all evaluated search-based models collapses to near-zero (Table S7); exact values are in the supplementary material.
• A fundamental measurement gap: Reference-based metrics cannot distinguish a failed search from a creative success, since a valid novel route is scored as a failure. The authors frame this as the central challenge for next-generation retrosynthesis evaluation.
• Cost as a quantifiable axis: The Pareto plot of Top-10 route-matching accuracy versus computational cost in USD on mkt-cnv-160 shows order-of-magnitude cost differences between existing models; the evaluated per-target times range from 0.5–1.2 seconds (Retro*) to 133.0 seconds (AiZynF Retro* (High)) on USPTO-190. The cost-performance landscape is reported as robust across route topologies (Fig. S1).

Methodology in Plain English

The authors first built a translation layer: because every retrosynthesis tool writes its answers in a different format (nested JSON, string recipes, node-and-edge schemas), RetroCast uses adapters to convert any output into one shared schema, like a universal power adapter for model predictions. Then they defined two benchmark tracks. The "practical" track (mkt-) lets models use unrestricted training data and judges them against the ASKCOS Buyables stock of 313,458 commercially available compounds. The "fair comparison" track (ref-) requires models to train on a common open-source dataset and is evaluated with PaRoutes' own patent-derived stocks, isolating algorithmic differences from stock availability. All benchmarks were built by stratified sampling of routes by length and topology—for example, mkt-lin-500 drew 100 routes from each length category (2, 3, 4, 5, and 6) from the commercially relevant linear subset—which gives high diagnostic signal at a fraction of the cost of evaluating everything.

To avoid cherry-picked results, the authors generated 15 candidate subsets with 15 distinct random seeds, ran a reference model (DMS Explorer XL) on each, computed a deviation score as the sum of squared Z-scores across three metrics (solvability, Top-1, Top-10 accuracy), and kept the seed with the lowest deviation for the final benchmark. For statistics, they used non-parametric bootstrapping with 10,000 resamples for 95% confidence intervals, plus paired bootstrap difference tests to check whether two models genuinely differ, flagging unstable estimates when the sample size was below 30 or when fewer than 5 positive or negative outcomes occurred.

They also changed how Top-K accuracy is computed. Raw outputs are first filtered for structural integrity, then filtered against all explicit task constraints such as stock termination, and only then ranked—so a model is judged on producing and correctly ranking valid solutions. Finally, they introduced Multi-Ground-Truth evaluation: for each reference route, any intermediate that is itself commercially available becomes a pruning point, and valid sub-routes are generated from combinations of these points that form an antichain in the route's directed acyclic graph, so a model is credited for finding a shorter, valid commercial route rather than penalized against a single patent-derived answer.

Why This Matters

The paper reframes what counts as progress in a field where reported success rates are routinely above 90% but practical utility is unclear. Rather than proposing another model, it proposes the measuring stick—and shows that the existing one can be gamed by topological pathfinding that produces chemically nonsensical steps. The argument that stock termination should be demoted from a primary objective to a necessary-but-insufficient post-hoc filter is a direct challenge to how CASP papers report results.

Real-world applications:

• Pharmaceutical route design: Chemists choosing between AI-proposed syntheses need confidence that intermediate steps are chemically valid, not merely that the leaves are purchasable.
• Synthesis-aware virtual screening: Selecting candidate molecules by synthetic feasibility is cited as a significant bottleneck; reliable cost and plausibility metrics change which compounds are worth pursuing.
• Cross-tool procurement and benchmarking: Standardized adapters and a shared prediction database let labs compare tools without writing bespoke parsers or rerunning costly searches.
• Expert-curated chemical bug hunting: SynthArena can be extended into a distributed annotation system where chemists flag invalid reaction steps, turning a static data release into a living dataset of "chemical bugs."

Industry relevance centers on economics and reproducibility. The authors note that order-of-magnitude differences in compute cost separate a feasible screening campaign from an impossible one, and that without standardized stocks and metrics, published leaderboard positions are not directly comparable across vendors, academic groups, or cloud deployments.

Future Directions

• Develop plausibility metrics that can reward novelty: The authors release their standardized prediction database specifically to decouple metric development from the computational cost of running planners, letting researchers prototype new scoring functions on existing state-of-the-art predictions.
• Turn SynthArena into a community annotation platform: Extending it so chemists can flag and categorize invalid reaction steps would move evaluation from passive metrics to an active, adversarial process of chemical bug hunting.
• Establish formal separation of research tracks: The paper advocates recognizing distinct tracks for methods and for evaluation, discouraging the practice of proposing new metrics alongside the methods they are meant to evaluate.
• Make secondary metrics meaningful: Route diversity, average length, and elegance become high-signal diagnostics only after a baseline of chemical validity is established, since these measures are currently computed over pools that include invalid pathways. The disambiguation of a failed prediction from a creative one is named as the central open challenge.

Target Audience

This paper is most valuable to CASP and ML-for-chemistry researchers who report or compare retrosynthesis performance, benchmark maintainers and dataset curators, and reviewers or program committees evaluating retrosynthesis claims. It also serves industry computational chemists at pharmaceutical and synthesis-planning companies who need to judge whether a tool's reported success rate translates into usable routes, and method developers who want a ready-made, auditable evaluation harness rather than building one from scratch.

Authors’ abstract

Progress in computer-aided synthesis planning (CASP) is obscured by the lack of standardized evaluation infrastructure and the reliance on metrics that prioritize topological completion over chemical validity. We introduce RetroCast, a unified evaluation suite that standardizes heterogeneous model outputs into a common schema to enable statistically rigorous, apples-to-apples comparison. The framework includes a reproducible benchmarking pipeline with stratified sampling and bootstrapped confidence intervals, accompanied by SynthArena, an interactive platform for qualitative route inspection. We utilize this infrastructure to evaluate leading search-based and sequence-based algorithms on a new suite of standardized benchmarks. Our analysis reveals a divergence between "solvability" (stock-termination rate) and route quality; high solvability scores often mask chemical invalidity or fail to correlate with the reproduction of experimental ground truths. Furthermore, we identify a "complexity cliff" in which search-based methods, despite high solvability rates, exhibit a sharp performance decay in reconstructing long-range synthetic plans compared to sequence-based approaches. We release the full framework, benchmark definitions, and a standardized database of model predictions to support transparent and reproducible development in the field.