Do Perturbation Predictors Learn Gene Interactions? Diagnosing Extrapolation in Agentic Screening

AI summary

Overview

Research area: Computational functional genomics and machine learning reliability—specifically, ML models that predict cellular responses to combinatorial CRISPR perturbations, and how such predictors would be used inside agentic experimental-design workflows.

Technical level: Advanced. The paper assumes familiarity with Perturb-seq data, out-of-distribution (OOD) detection, and perturbation-response models such as GEARS, CPA, and scGPT.

Scope: A diagnostic study (single author, NeurIPS 2026 Workshop on Agentic Systems for Molecular Sciences) that decomposes combinatorial perturbation predictions into an observed additive part and a model-extrapolated interaction part, and evaluates that decomposition as a reliability signal across eight prediction methods and three Perturb-seq datasets.

What This Paper Is About

Machine-learning models are increasingly asked to predict the effect of combining two gene perturbations that were never measured together, even when each gene was measured alone. Because both genes look familiar to the model, standard novelty and uncertainty detectors have little to latch onto. The paper asks whether a simple quantity—how far a model's prediction strays from the naive sum of the two measured single-gene effects, called additive disagreement (AD)—can flag predictions that rely on unsupported guessing about gene interactions, and whether that signal is useful in an agentic screening loop.

Key Contributions

1. A formal decomposition of combinatorial predictions. Every prediction for an unseen pair (A, B) is written as f_θ(A, B) = (δ_A + δ_B) + Î_θ(A, B), where δ_A and δ_B are measured single-gene effects and Î_θ is the model-implied interaction correction. AD is defined as the mean absolute value of that correction across the G evaluated genes, giving a biologically interpretable measure of the model's departure from additivity.

2. A benchmark of eight prediction methods on three Perturb-seq datasets. The methods span additive prior, Ridge, MLP, Transformer, CPA, scGPT, GEARS, and GEARS+scGPT, evaluated on Norman (compositional), Joung–Zhang (compositional), and Adamson (single-gene OOD) under a shared protocol with three seeds (42, 43, 44) on Norman and leave-one-out cross-validation over 44 pairs on Joung–Zhang.

3. A head-to-head comparison of AD against eight alternative reliability signals—prediction magnitude, gene-wise variance, ensemble variance, kNN distance, Mahalanobis distance, nearest-neighbour error proxy, cosine to nearest neighbour, and a random baseline—using failure-detection AUROC, error rank correlation, and MAE improvement under selective abstention.

4. Ablations and decision-level analysis. GEARS graph ablations isolate the co-expression graph as a major error source, and a cross-fitted "conservative fallback" analysis tests whether high-risk predictions can be replaced by the additive reference inside a screening workflow.

Main Findings

• Specialised neural predictors do not consistently beat the additive baseline. On Norman (75 evaluations), Ridge achieves the lowest full-transcriptome MAE (0.0118) versus 0.0142 for Additive, and wins 60 of 75 evaluations. On Joung–Zhang (44 pairs), Ridge again has the lowest MAE (0.0187) versus 0.0190 for Additive, but CPA wins the most pairs (29/44) and Transformer wins 20/44. Additive achieves the best Top20 MAE on Norman (0.2121), while Ridge achieves the best Top20 MAE on Joung–Zhang (0.2496).

• GEARS never outperforms the additive baseline. GEARS records 0/75 wins on Norman and 0/44 on Joung–Zhang, with MAE 0.0336 and 0.0512; GEARS+scGPT also records 0/75 and 0/44, with MAE 0.0338 and 0.0500. Initialising with scGPT embeddings provides little improvement.

• Larger departures from additivity track larger errors. AD shows the highest failure-detection AUROC at all three thresholds for GEARS on Norman (0.994 at Top50%, 0.989 at Top25%, 0.996 at Top10%) and a Spearman error correlation of 0.982. Abstaining on the highest-risk 10%, 20%, and 50% of predictions reduces MAE by 12%, 19%, and 38% respectively.

• Generic OOD and uncertainty signals are weaker and inconsistent. Mahalanobis distance performs near random (AUROC 0.491/0.463/0.418, Spearman −0.081) and can worsen MAE under abstention (−2% at 20%, −7% at 50%). Cosine to nearest neighbour is strongly anti-correlated with error (Spearman −0.752). Ensemble variance reaches only 0.670/0.708/0.782 AUROC and +1%/+4%/+17% improvement. kNN distance reaches 0.623/0.638/0.599 and +4%/+6%/+16%.

• The co-expression graph drives GEARS's extrapolation failures. Co-expression degree correlates with both predicted interaction magnitude (ρ = 0.633) and prediction error (ρ = 0.626). Removing the co-expression graph reduces MAE from 0.0336 to 0.0228; removing both graphs yields 0.0259; removing only the GO graph slightly worsens MAE to 0.0342. Graph-ablated variants still underperform the additive baseline, and training on single-gene perturbations alone gives 0.0291, worse than additive.

• The failures are not caused by weak or trivial interactions. The interaction component accounts for 68% of the total effect on Norman and 104% on average on Joung–Zhang, with no pair below 10%. Interactions are structured: on Norman the first principal component explains 31.3% of variance, four explain 50%, and 21 explain 90%; on Joung–Zhang the values are 21.3%, four, and 15 components.

• Representative failure cases show corrections adding noise. On CEBPE+CEBPA, GEARS reaches MAE 0.096 versus 0.036 for additive (AD 0.087), nearly tripling the error. On SET+CEBPE, additive already achieves 0.025 while GEARS introduces a smaller correction (AD 0.038). GEARS assigns 9.7× larger corrections to top-20 differentially expressed genes than to the rest of the transcriptome (mean |Î_g| = 0.324 vs 0.033), and improves over additive on only 25.3% of pairs under Top20 MAE.

• AD's advantage depends on the evaluation target. On Norman, AD's failure-detection AUROC falls from 0.990 under full-transcriptome MAE to 0.619 under Top20 MAE, and from 0.961 under full-transcriptome MSE to 0.597 under Top20 MSE. The authors attribute this to AD aggregating deviations across all genes while Top20 metrics score only the most differentially expressed genes.

• AD is the most consistent ranking signal across models and datasets. Across all evaluated models, AD–error Spearman correlations range from 0.528 to 0.982 on Norman and from 0.767 to 0.939 on Joung–Zhang, whereas prediction magnitude ranges from strong (0.812 for Ridge on Joung–Zhang) to near zero (0.041 for Transformer, 0.021 for scGPT). AD is undefined on Adamson because the held-out gene itself is unobserved.

• Fallback routing helps but does not close the gap. In the cross-fitted conservative fallback analysis, most Norman models routed by AD land at 0.0144 MAE, slightly worse than always using additive (0.0142) and above the oracle (0.0135–0.0142). On Joung–Zhang a few routed models slightly beat additive: kNN routing for Ridge (0.0187 to 0.0185), prediction magnitude for CPA (0.0192 to 0.0186), and gene-wise variance for Transformer (0.0201 to 0.0187) against additive at 0.0190.

Methodology in Plain English

The researchers restricted evaluation to gene pairs where both genes had been perturbed individually during training but the pair itself was held out. This makes the problem genuinely compositional: nothing in the input looks new, only the combination does.

For each such pair they built an "additive reference" by simply adding the two measured single-gene expression changes together. They then computed AD—the average size of the gap between a model's prediction and that reference—and asked whether large gaps coincide with large errors.

They trained eight predictors (a no-training additive baseline, Ridge regression, an MLP, a Transformer, the CPA compositional VAE, a scGPT-embedding-based predictor, GEARS, and GEARS with scGPT-initialised embeddings) on condition-level mean effect vectors, except GEARS, which used its original single-cell pipeline. Norman used three random splits (42, 43, 44) of 25 test pairs each, giving 75 test evaluations; Joung–Zhang used leave-one-out over 44 eligible pairs with models retrained per fold; Adamson contributed 48 single-gene test evaluations across three predefined splits. Experiments ran on an internal cluster of 10 NVIDIA A100 GPUs.

Reliability was scored three ways: AUROC for detecting the highest-error predictions at 50%, 25%, and 10% thresholds; Spearman correlation between risk score and error; and MAE on the retained set after removing the highest-risk 10–50% of predictions. Finally, they simulated an agentic decision rule: if a signal flags a prediction as high-risk, swap in the additive reference instead, with thresholds tuned on calibration groups and tested on disjoint held-out groups.

Why This Matters

The paper reframes a practical failure mode: models can appear confident on gene combinations they have never seen because both components are familiar, so input-space novelty checks give false reassurance. AD provides a cheap, interpretable check that any system querying a predictor can compute using only single-gene estimates—no retraining, no ensembles, no embedding-space bookkeeping.

Real-world applications:

• Experiment triage in functional genomics: prioritising which of many candidate gene pairs to actually test, and flagging predictions that should be discounted before committing wet-lab resources.
• Agentic screening pipelines: giving an autonomous agent a routing rule—trust the model, or fall back to the additive reference—when it queries a perturbation predictor during experimental design.
• Model selection and QA for virtual-cell models: using the AD–error relationship as a diagnostic when deciding whether a new architecture genuinely learns interactions or just adds structured noise.
• Architecture debugging: the co-expression graph ablation shows how this kind of analysis can localise where an error source lives inside a model, which generalises to other graph-based biological models.

Industry relevance: pharmaceutical and biotech groups using perturbation screens for target discovery and combination-therapy design face exactly the cost problem described—follow-up experiments are expensive, and confidently wrong predictions distort candidate rankings. A reliability signal that requires only already-collected single-gene data is inexpensive to deploy, and the finding that a linear Ridge baseline is competitive with specialised neural models has direct consequences for build-versus-buy decisions on virtual-cell infrastructure.

Future Directions

1. Gene- and pathway-level disagreement metrics. The paper shows AD transfers poorly from full-transcriptome MAE (AUROC 0.990) to Top20 MAE (AUROC 0.619) on Norman, and explicitly proposes extending the framework to gene-level and pathway-level disagreement for applications focused on biomarkers or selected gene sets.

2. Extending beyond pairwise additivity. The current reference assumes simple two-gene additivity. The author notes that higher-order combinations will require richer references—hierarchical, pathway-aware, or mechanism-informed priors.

3. Closing the gap to oracle routing. Routed performance remains above the oracle in most cases on both datasets, indicating current signals do not fully capture when fallback is beneficial; better routing policies and better-calibrated thresholds are open problems.

4. Explaining why interaction corrections fail to transfer. The co-expression ablation identifies one mechanism in GEARS, but the broader question of why the evaluated architectures do not generalise structured interaction effects to unseen combinations—given that those interactions are substantial and low-dimensional—remains unanswered.

Target Audience

Machine-learning researchers working on perturbation prediction and virtual-cell models; computational biologists designing or interpreting Perturb-seq screens; and engineers building agentic or automated pipelines for experimental design who need to know when to trust a model's prediction. It is also useful for benchmark designers, since the Top20-versus-transcriptome discrepancy shows how conclusions about reliability can flip with the evaluation metric. Readers without a background in single-cell genomics or OOD detection will find the framing accessible but the details demanding.

Authors’ abstract

High-throughput perturbation screens increasingly use machine-learning models to predict the effects of gene combinations that were never measured. A difficult case arises when both genes in a test pair were seen individually during training, but their combination was not. In this setting, standard OOD methods often fail because neither gene appears unfamiliar on its own. We therefore compare each model prediction with a simple additive prediction obtained by summing the two observed single-gene effects. The difference between them, which we call additive disagreement (AD), measures how much extra interaction effect the model introduces for an unseen combination. Across eight prediction methods and three Perturb-seq datasets, specialised neural predictors do not consistently outperform this simple additive baseline. More importantly, predictions with larger deviations from the additive baseline tend to have larger errors. Conventional output-based and embedding-based confidence signals are inconsistent across datasets and models, whereas AD remains informative across all evaluated compositional settings. We further study how this signal can guide decisions in an agentic perturbation-screening workflow. The code is available at https://github.com/zygao930/OOD.