Beyond Benchmark Accuracy: A Diagnostic Test for LLM Chemical Prediction

AI summary

Overview

Research area: Evaluation methodology for large language models applied to chemistry — odor prediction, molecular property prediction, retrosynthesis, and reaction yield prediction.

Technical level: Intermediate. The chemistry (SMILES, stereochemistry, functional group edits) and the evaluation design (control conditions, collapse ratio) require some domain familiarity, but the central argument is accessible without deep ML background.

Scope in one sentence: The paper introduces and applies an "Identity-Blinded Reasoning Diagnostic" that hides the target molecule's identity and defines it only through a source molecule plus a described structural edit, in order to test whether LLM chemical predictions come from genuine structure–property reasoning or from contextual cues such as source labels.

What This Paper Is About

Large language models score well on chemical prediction benchmarks, but those benchmarks usually hand the model the target's own name or structure string. The paper argues this makes it impossible to tell whether the model is reasoning about structure–property relationships or simply recognizing the target and recalling associated information. To separate the two, the authors define each prediction target indirectly — by a known source molecule plus a natural-language description of the structural edit that produces the target — and then measure how much the model's answer is swayed by purely contextual information about the source molecule.

Key Contributions

1. The Identity-Blinded Reasoning Diagnostic framework, which withholds the target molecule's name and SMILES while preserving the chemical information needed to determine the target, so that prediction targets and ground-truth answers remain identical to the original tasks.

2. A controlled four-condition design (C0–C3) that separates target-structure access, source-structure access, true source labels, and false source labels, combined with a label-consistent versus label-inconsistent group analysis and a paired C2/C3 source-label swap.

3. A unified diagnostic across six tasks in four domains — odor prediction, molecular property prediction, retrosynthesis, and reaction yield prediction — showing that the tested models' predictions depend systematically on non-target cues such as source molecule structure and labels.

4. A contextual evidence augmentation experiment across eight odor-related tasks, testing whether task-relevant tool evidence improves prediction and whether the gains track how well the evidence covers the target property.

Main Findings

• Benchmark accuracy varies widely and does not clear simple baselines. Under the conventional C0 condition (target structure given), BACE accuracy was 57.40% for Claude-Opus-4.8 and 51.50% for the other three models; HIV accuracy was 57.90% (Claude), 31.40% (DeepSeek), 34.30% (GPT), and 61.20% (Gemini). Retrosynthesis exact match ranged from 34.9% to 48.8%. Yield MAE ranged from 32.34 to 56.46 percentage points, above the mean baseline of 21.98 and the median baseline of 21.65.

• The identity-blinded setting preserves enough information to identify the target. Structure reconstruction accuracy ranged from 86.05% to 100% across the five molecular tasks, and all four models correctly recovered every chiral target molecule.

• Source-structure dependence appears even without source labels. Under C1, all four models scored higher F1 in the label-consistent group on both odor tasks, and three models showed the same trend on BACE. Overall, 10 of 24 model–task combinations showed a significant label-consistent advantage, but nearly half of the tasks showed no evident dependence.

• True source labels amplify the effect sharply. Under C2, the label-consistent group held an advantage in 22 of 24 model–task combinations. Claude-Opus-4.8 reached 100% accuracy in the label-consistent group on BACE and HIV while dropping to 0% in the label-inconsistent group. On the chirality change odor task, GPT-5.5 scored F1 0.994 versus 0.061 across the two groups, and Gemini-3.1-Pro 0.952 versus 0.074. On many tasks the label-inconsistent metric fell to 0, producing an infinite collapse ratio.

• Swapping only the source label shifts predictions. In the binary tasks, 91.2% of Claude-Opus-4.8 predictions moved toward the false source label, versus 58.8% (GPT-5.5), 43.4% (Gemini-3.1-Pro), and 36.0% (DeepSeek-V4-Pro-Preview) — with the target molecule, edit, and ground truth held fixed.

• Set-valued and numerical outputs shift too, but unevenly. Normalized directional response ranged from 0.307 to 0.869 on the chirality change odor task but only 0.059 to 0.135 on the structural change odor task. In yield prediction, 61.2% to 98.0% of predicted values moved in the same direction as the replacement source yields.

• External evidence helps only when it covers the target. On the structural change task, DeepSeek-V4-Pro-Preview improved from F1 0.160 to 0.312 and Qwen3.5-9B from 0.035 to 0.248, with consistent gains also in odor retrieval, main odor selection, and receptor matching. Where tool coverage is weak, gains were limited or reversed: Gemini-3.1-Pro's mixture-task F1 fell from 0.190 to 0.149.

Methodology in Plain English

The core move is to stop naming the target. Instead of giving the model the target molecule's name or SMILES, the researchers give it a source molecule's structure and a plain-language description of the edit that turns the source into the target (a functional group change, a fragment added or deleted, a chirality inversion, or — for reaction yield — a single component swapped in a reaction recipe). The model must predict the target's property or outcome from that.

Four conditions are compared. C0 is the ordinary benchmark setting with the target structure supplied. C1 supplies only source structure and edit description. C2 adds the true source label. C3 keeps everything from C2 but replaces the source label with a deliberately false one — inverted for binary labels, a non-overlapping alternative set for set-valued labels, and a value at least 25 percentage points away for numerical labels — without telling the model the label is false.

Samples are split into label-consistent and label-inconsistent groups based on whether true source and target labels agree (binary: identical; set-valued: label overlap; numerical: property difference). The ratio of group scores, called the collapse ratio (CR), is defined so that CR > 1 always means the label-consistent group did better. The C2/C3 comparison is then done on matched target samples, measuring how often predictions move toward the substituted source label.

The main diagnostic ran six tasks with independent per-sample calls to Claude-Opus-4.8, DeepSeek-V4-Pro-Preview, GPT-5.5, and Gemini-3.1-Pro. The evidence augmentation experiment covered eight odor-related tasks with Gemini-3.1-Pro, DeepSeek-V4-Pro-Preview, Gemma-4-31B, Qwen3.5-27B, and Qwen3.5-9B, using the NOSE tool for odor and receptor matching evidence and RDKit for structural and chirality edits.

Data sources: Pyrfume olfactory data curated through the NOSE procedure (38,111 structure–label records over 7,757 distinct structures; 6,985 single-component structures retained), MoleculeNet for BACE and HIV (1,513 and 38,034 single-component structures after cleaning), the USPTO-50K test set curated by RetroXpert (5,007 reaction records), and Buchwald–Hartwig yield data (3,600 reaction records covering 15 substrates, 4 ligands, 3 bases, and 20 additives). Diagnostic sample counts were 179 structural odor (122 consistent / 57 inconsistent), 56 chiral odor (24/32), 136 BACE (68/68), 242 HIV (121/121), 42 retrosynthesis (28/14), and 100 yield (50/50). The C3 yield condition retained 98 samples after filtering.

Why This Matters

Impact on research. The paper argues that high benchmark scores are insufficient evidence of reliable structure–property reasoning, and offers a reusable control: hide target identity while keeping the information needed to derive it. It situates itself against prior diagnostics, noting that hiding property names or substituting SMILES characters (Busch et al.) also removes chemical information, making declines hard to attribute, and that contextual-label studies (Zhao et al.) still reveal the target's identity. The framework is designed to be portable to other AI for Science tasks where a target can be defined as a transformation of a known entity.

Real-world applications:

• Fragrance, flavor, and consumer-product formulation, where odor descriptors are the prediction target and Pyrfume-derived data such as IFRA, GoodScents, and Flavornet supply labels.
• Drug discovery screening, where BACE and HIV activity predictions decide which compounds advance.
• Synthetic route planning, where retrosynthesis models propose reactant sets for a target molecule.
• Process and catalysis chemistry, where yield prediction over ligand, base, and additive combinations guides reaction optimization, as in the Buchwald–Hartwig dataset used here.

Industry relevance. The authors are affiliated with DP Technology, and the diagnostic speaks directly to groups deploying LLM assistants on chemical data: if a model's output moves when only a source reference label changes, benchmark dashboards overstate capability, and tool-backed evidence pipelines need to be evaluated for coverage of the specific property being predicted — the paper shows unmatched evidence can make predictions worse. The AI use statement also discloses practical pipeline details (Gemini-3.1-Pro generating structural edit descriptions and assisting concentration-label curation, with deterministic verification of the underlying edits).

Future Directions

• Extending the framework to other AI for Science domains where targets can be defined by source entities and edit operations, beyond the four domains tested here.
• Building better tool coverage. The mixed results in the evidence augmentation experiment, including the mixture-task decline from F1 0.190 to 0.149, point to a need for tools that model enantiomer differences, component interactions, and concentration effects, which fall outside the current tools' dedicated scope.
• Investigating why sensitivity varies so much across tasks — the normalized directional response spanned 0.307 to 0.869 on chirality change odor but only 0.059 to 0.135 on structural change odor, and 10 of 24 combinations showed dependence under C1 versus 22 of 24 under C2.
• Characterizing which models are most susceptible. Claude-Opus-4.8 switched to false source labels in 91.2% of binary predictions versus 36.0% for DeepSeek-V4-Pro-Preview, raising the question of what training or prompting choices drive this difference. The paper does not report an explanation.

Target Audience

Chemists and cheminformatics researchers who use LLMs for property, odor, retrosynthesis, or yield prediction; machine learning evaluation researchers interested in contamination, shortcut learning, and contextual-cue sensitivity; benchmark designers for AI for Science; and industry practitioners at pharma, fragrance, and catalysis organizations who rely on benchmark numbers when selecting or deploying chemical prediction models. The paper is most useful to readers comfortable interpreting F1, accuracy, exact match, and MAE, but the diagnostic logic is explained in enough plain language to be followed by anyone who understands what a chemical structure edit is.

Authors’ abstract

LLMs have achieved strong performance on chemical prediction benchmarks. However, existing evaluation methods typically provide target entity identity information directly, making it difficult to determine whether model predictions arise from transferable structure-property reasoning capabilities or rely on target-related contextual information or memory retrieval. We propose the Identity-Blinded Reasoning Diagnostic framework to evaluate chemical prediction models when target entity identity information is hidden. In this framework, the target is no longer directly defined by its entity identity, but instead by a known source entity and an explicit structural edit operation. This setting preserves the information required to determine the target while reducing the opportunity for models to access target identity cues. We evaluate this framework across six tasks spanning odor prediction, molecular property prediction, retrosynthesis, and reaction yield prediction. The results show that models generally achieve better prediction performance when source and target entity properties are consistent. Providing explicit source entity labels further strengthens this dependence. Moreover, while keeping the target entity, structural change, and ground-truth label unchanged, replacing only the source entity label causes some model predictions to systematically shift toward the substituted source labels. These results indicate that high benchmark accuracy in chemical prediction tasks does not necessarily imply that LLMs possess reliable structure-property reasoning capabilities. Our diagnostic framework provides a controlled evaluation approach for testing whether models can genuinely update predictions according to entity changes. The framework can be extended to other AI for Science tasks, particularly those where targets can be defined through transformations of known entities.