How Well Do LLMs Understand Drug Mechanisms? A Knowledge + Reasoning Evaluation Dataset

AI summary

Overview

Research area: Natural language processing / LLM evaluation, applied to biomedical and pharmaceutical knowledge (drug mechanisms of action, drug repurposing, personalized medicine).

Technical level: Intermediate. The paper is readable for anyone familiar with LLM benchmarking, but it assumes comfort with biomedical entity graphs and accuracy-style metrics.

Scope: The paper introduces the Drug Mechanisms Counterfactuals Dataset — a benchmark built from DrugMechDB that tests whether large language models both recall known drug mechanisms of action and reason correctly about counterfactual alterations to those mechanisms — and uses it to evaluate four OpenAI models plus one Qwen model.

What This Paper Is About

Drug mechanisms of action (MoAs) are naturally expressed as directed chains of interactions between biomedical entities, running from a drug to a disease. If an LLM is going to be useful for drug development, repurposing, or personalized medicine, it must not only recall these chains but also reason about what happens when a link in a chain changes in a novel way. The authors build a dataset of such novel situations — counterfactuals derived by adding, deleting, or inverting interactions in real MoA graphs — and measure how well current LLMs handle them, in both a "closed world" setting (the needed facts are supplied in the prompt) and an "open world" setting (the model must recall the facts itself).

Key Contributions

1. A new benchmark, the Drug Mechanisms Counterfactuals Dataset, containing both positive and negative counterfactuals across three counterfactual types (Add-Link, Delete-Link, Invert-Link), each split into "surface" counterfactuals (the altered link touches the drug named in the prompt) and "deep" counterfactuals (the altered link is an interior link in the chain). The authors describe it as the first large dataset of this kind for evaluating LLM understanding of drug and disease mechanisms.

2. A dataset and protocol for factual MoA knowledge, using 1,000 Drug-Disease MoAs randomly drawn from DrugMechDB as positives, and 1,000 negatives drawn from PrimeKG (500 randomly drawn Drug-Disease pairs with a contra-indication relationship and 500 pairs with no direct Drug-Disease relationship).

3. A graph-consistency evaluation method that does not require exact match to a reference MoA. Entity names are matched by approximate text match, and a returned graph is judged "directionally consistent" with the reference rather than identical, reporting interior-node match and reduced-edge match proportions.

4. An empirical comparison of five LLMs — GPT-4o, o3, o3-mini, o4-mini, and Qwen3-4B (Thinking) — on these tasks, showing that reasoning models beat the general-purpose GPT-4o, that open-world reasoning is harder than closed-world reasoning, and that deep counterfactuals are much harder than surface ones.

Main Findings

• Model ranking: Overall performance in decreasing order is Qwen3-4B and o4-mini in near ties, followed by o3, o3-mini, and GPT-4o. Grouped open-world accuracy (mean ± 90% c.i.) is 0.60 for 4o, 0.71 for o3, 0.63 for o3-mini, 0.76 for o4-mini, and 0.77 for Qwen3-4B. Closed-world grouped accuracy is 0.86, 0.95, 0.92, 0.97, and 0.94 respectively.

• Smaller models can outperform larger ones: The authors report it as surprising that the smaller o4-mini outperforms o3, and that the small Qwen3-4B performs so well — beating o4-mini in some cases and losing to it in others. There are also cases where o3-mini outperforms the larger o3 (for example, closed-world negatives and surface counterfactuals).

• Open world is harder than closed world: All models do better when the relevant MoA knowledge is included in the prompt. For example, deep counterfactual accuracy in the closed world is 0.75 (4o), 0.93 (o3), 0.86 (o3-mini), 0.95 (o4-mini), 0.89 (Qwen3-4B), versus 0.45, 0.59, 0.51, 0.64, and 0.66 in the open world.

• Negatives are generally easier than positives in the open world: Open-world grouped accuracies on negatives (0.77, 0.74, 0.61, 0.84, 0.81) exceed those on positives (0.44, 0.69, 0.65, 0.69, 0.73) for every model except o3-mini, whose performance is very similar on both. In the closed world, positive and negative accuracies are much closer.

• Add-Link is the exception: For open-world Add-Link queries, accuracies on positives (0.65, 0.99, 0.94, 0.97, 0.91) are actually higher than on the corresponding negatives (0.68, 0.66, 0.72, 0.77, 0.77), which the authors call unexpected since both require the same information.

• Deep counterfactuals are much harder than surface ones: This holds for all models in both settings. The worst-performing cases across all models are open-world positive samples where an interior link is deleted or inverted ("D.D.I. pos, o/w": 0.16 for 4o, 0.25 for o3, 0.22 for o3-mini, 0.26 for o4-mini, 0.41 for Qwen3-4B).

• "Partially Blocked" is a commonly chosen answer: For positive Delete-Link and Invert-Link deep counterfactuals, tabulating LLM responses showed "Partially Blocked" was the most commonly returned option, at 55% of responses. Adding it as an accepted response produces a large jump in accuracy (relaxed open-world numbers include 0.849 for 4o on Delete-Link deep positives and 0.853 for o3 on Invert-Link deep positives).

• Reasoning-chain length: A combined set of 2,000 open-world positive deep-counterfactual instances for Delete and Invert-Link across the four OpenAI models contained 444 correct (22.2%) and 1,556 incorrect (77.8%) responses. Charts suggest accuracy declines with increasing MoA length and possibly with distance from the disease node, but a logistic regression using the three distances as variables did not yield significant coefficients other than for the intercept.

• Factual knowledge results (GPT-4o only): Average accuracy was 0.93 in the Anonymized Disease mode, 0.92 in Named Disease, and 0.94 in Named Disease with Associations. Positives scored 0.89 ± 0.01, 0.85 ± 0.00, and 0.90 ± 0.00; negatives scored 0.97 ± 0.00, 0.98 ± 0.00, and 0.97 ± 0.00. Interior-node match was 0.50 ± 0.00, 0.37 ± 0.00, and 0.51 ± 0.00; reduced-edge match was 0.98 ± 0.00, 0.90 ± 0.00, and 0.98 ± 0.00.

• MoA returned (Add-Link positives, open world): Surface values were 0.812 (4o), 0.987 (o3), 0.962 (o3-mini), 0.973 (o4-mini), 0.955 (Qwen3-4B); deep values were 0.478, 0.984, 0.911, 0.961, and 0.860. In the closed world these rise to 0.995–1.000 for surface and 0.822–1.000 for deep, with 0% "Potentially very different MoAs" for every model.

Methodology in Plain English

The authors start from DrugMechDB, a manually curated database of graphical mechanisms of action for 4,664 Drug-Disease pairs, in which each interaction is directed from a source entity to a target entity with a label describing the relationship or direction of effect. Curators encoded 14 entity types and 66 interaction relation labels, and followed the convention that each MoA is a directed acyclic graph (DAG) with the drug as the single root and the disease as the single sink.

From this they build counterfactuals by changing one relationship at a time:

• Add-Link: introduce a new interaction between two entities taken from two MoAs for different drug-disease pairs, so that a new MoA is now enabled for drug-1 treating disease-2. In negative samples, the direction of effect of that new interaction is inverted, so the chain connects drug to disease but does not compose into a therapeutic action.
• Delete-Link: deactivate a cut-edge interaction in an existing MoA, disconnecting drug from disease. In negative samples, the deleted interaction goes from an entity in the MoA to a random protein not participating in that MoA.
• Invert-Link: invert the direction of effect of a cut-edge interaction, rendering the MoA therapeutically invalid. In negative samples, the observed interaction goes from the source entity to a randomly chosen protein not in the MoA.

Each counterfactual is framed to the model as an observation in patients with an unusual symptom combination. Queries are asked either in a closed world (relevant MoAs inserted into the prompt) or an open world (model must recall everything). For Delete and Invert-Link, the model chooses among four labeled options: No Effect, Partially Blocked, Fully Blocked, or Contra-indicated. Correct answers are Fully Blocked for positive Delete-Link, No Effect for negative Delete-Link, Contra-indicated or Fully Blocked for positive Invert-Link, and No Effect for negative Invert-Link.

Because no model is guaranteed to use the same entity names, the authors do not require exact graph matches. They match entity names by approximate text match and test for directional consistency: for every simple directed path between mapped nodes in the reference graph, there must be a directed path between the corresponding nodes in the candidate graph. Edge labels are ignored in this test. They also built a mechanism-of-action simulator to validate the interaction labels on the Add and Invert-Link counterfactuals.

For prompting, they used in-context learning: 4 positive examples and one negative for the factual knowledge test, and 5 positive and 3 negative examples for Add-Link. Delete and Invert-Link queries were zero-shot, asking the model to explain its reasoning and then select an answer option.

Sample sizes: Add-Link has 1,000 samples for each of the two sub-types, for both positive and negative; Delete and Invert counterfactuals have 250 samples for each of the eight cases. All test samples were issued as separate queries in separate sessions through the OpenAI API with seed = 42, temperature = 1.0, presence_penalty = 0, max_completion_tokens = None, and reasoning_effort = "medium" for the reasoning models. Counterfactual metrics are averaged across five runs. The Qwen model was deployed locally using a single vllm-based server with a 600-second timeout and, due to time constraints, only a single run was performed.

Why This Matters

The paper shows that current LLMs can reason about drug mechanisms when the relevant facts are handed to them, but that recalling those facts and reasoning about them simultaneously is substantially harder. It also shows that performance drops sharply when the altered interaction is buried inside the mechanism rather than attached to the drug named in the prompt — exactly the kind of indirect, multi-step inference that real scientific and clinical use would require.

Real-world applications:

• Drug repurposing: Asking whether a drug might work for a disease it was not designed for, based on a chain of molecular interactions.
• Personalized medicine: Reasoning about whether a drug's mechanism is still intact in a patient with an unusual combination of conditions.
• Clinical decision support: Judging whether an observed molecular change should make a drug contraindicated or fully blocked for a given patient.
• Drug safety / contraindication checks: Recognizing when an interaction change means a drug could worsen rather than treat a disease.

Industry relevance: Because the best performers include a small open model (Qwen3-4B-Thinking) that nearly matches or beats larger proprietary ones, the benchmark is relevant to teams deciding how much model size and cost they actually need for biomedical reasoning tasks. The authors also stress that because biology and medicine constantly evolve, LLM answers in these critical fields should always be validated by an expert, and that deployment should pair pre-trained LLMs with an actively updated knowledge base.

Future Directions

• Investigate why deep counterfactuals fail: The authors call it worth investigating further how deep counterfactuals affect LLM reasoning, noting that their logistic regression on chain depths was inconclusive and that response-validation noise may be a confounding factor.
• Disentangle recall from reasoning: The gap between open-world and closed-world performance suggests a need for methods that separate a model's ability to retrieve relevant mechanisms from its ability to reason over them.
• Resolve the "Partially Blocked" ambiguity: The authors argue true metrics lie somewhere between their strict and relaxed numbers, and that DrugMechDB is not a complete representation of all Drug-Disease mechanisms — so future work may need more complete mechanism data and better scoring of partial mechanisms.
• Keep factual knowledge current: The authors suggest combining pre-trained LLMs with actively updated knowledge bases so that factual knowledge does not go stale as disease and drug understanding evolves.

Target Audience

This paper is most useful for researchers and engineers working on LLM evaluation, biomedical NLP, and AI for drug discovery or clinical decision support. It also suits benchmark designers who want a template for building counterfactual evaluation sets over structured scientific knowledge, and machine learning practitioners choosing among reasoning models for biomedical tasks. Clinicians and biologists may find the framing accessible, but the paper assumes familiarity with LLM benchmarking terminology and graph-based evaluation metrics.

Authors’ abstract

Two scientific fields showing increasing interest in pre-trained large language models (LLMs) are drug development / repurposing, and personalized medicine. For both, LLMs have to demonstrate factual knowledge as well as a deep understanding of drug mechanisms, so they can recall and reason about relevant knowledge in novel situations. Drug mechanisms of action are described as a series of interactions between biomedical entities, which interlink into one or more chains directed from the drug to the targeted disease. Composing the effects of the interactions in a candidate chain leads to an inference about whether the drug might be useful or not for that disease. We introduce a dataset that evaluates LLMs on both factual knowledge of known mechanisms, and their ability to reason about them under novel situations, presented as counterfactuals that the models are unlikely to have seen during training. Using this dataset, we show that o4-mini outperforms the 4o, o3, and o3-mini models from OpenAI, and the recent small Qwen3-4B-thinking model closely matches o4-mini's performance, even outperforming it in some cases. We demonstrate that the open world setting for reasoning tasks, which requires the model to recall relevant knowledge, is more challenging than the closed world setting where the needed factual knowledge is provided. We also show that counterfactuals affecting internal links in the reasoning chain present a much harder task than those affecting a link from the drug mentioned in the prompt.