SpatialBench: Can Agents Analyze Real-World Spatial Biology Data?

AI summary

Overview

Research area: AI agent evaluation applied to computational biology, specifically spatial transcriptomics data analysis.

Technical level: Intermediate. The paper is readable without deep expertise in spatial biology, but it assumes familiarity with concepts such as principal component analysis, clustering, quality-control thresholds, and agent harnesses.

Scope: SpatialBench is a 146-problem benchmark built from real spatial transcriptomics workflows across five platforms and seven task categories, used to measure how frontier AI agents recover verifiable biological results from messy experimental data.

What This Paper Is About

Spatial transcriptomics assays are growing in scale and complexity faster than classically trained biologists can analyze them, creating a bottleneck in biological discovery. AI agents have improved at software engineering and general data analysis, but it is unclear whether they can extract real biological insight from raw spatial datasets. The authors build SpatialBench, a benchmark of 146 verifiable problems drawn from practical spatial analysis workflows, to measure and diagnose agent capability on this capability that sits between "knowing biology" and "writing code."

Key Contributions

1. A verifiable benchmark from real workflows. 146 problems spanning five spatial transcriptomics technologies (AtlasXomics, MERFISH, Seeker, Visium, Xenium) and seven task categories (quality control, normalization, dimensionality reduction, clustering, cell typing, differential expression, spatial analysis). Each problem snapshots analysis state immediately before a target step and pairs it with a deterministic grader.

2. Quantification of frontier model performance on spatial data. Base model accuracy remains low, at 20–38% across model families, with strong model–task and model–platform interactions that aggregate scores hide.

3. Demonstration that harness design is a first-class variable. The same base model (Opus-4.5) moves from 38.4% under a base configuration to 48.1% with Claude Code and 61.7% with the Latch agent, an absolute uplift of 23.3 percentage points from base to Latch.

4. Trajectory-level diagnosis of failure modes. Manual inspection of session logs (reasoning traces, tool invocations, stdout/stderr) reveals distinct behavioral regimes across model families, including instruction-following failures, domain miscalibration, and differences in productive exploration.

Main Findings

• Aggregate accuracy is low and separated by model. Opus-4.5 attains the highest mean accuracy (38.4%), followed by GPT-5.2 (34.0%), Sonnet-4.5 (28.3%), while Gemini-2.5-Pro (20.1%) and Grok variants (22–25%) consistently underperform.

• Efficiency separates model families more sharply than accuracy. GPT-5.1 and GPT-5.2 operate at approximately $0.02–$0.04 per evaluation and 56–89 s latency, while Anthropic, Gemini, and Grok models exceed $0.08 in cost and typically exceed 115 s in latency. This yields a trade-off between peak accuracy (Opus-4.5) and cost-effective performance (GPT-5.2).

• Task category changes the ranking. Accuracy ranges from approximately 10–22% on QC tasks to roughly 40–53% on dimensionality reduction and spatial analysis for the best models. Opus-4.5 performs best on spatial analysis (52.9%) and ties GPT-5.2 on clustering (33.3%); Sonnet-4.5 achieves the highest accuracy on dimensionality reduction (53.3%). All models perform poorly on QC and cell typing. Gemini-2.5-Pro underperforms across all categories, most notably on spatial analysis (9.8%).

• Platform matters. Individual models differ by 15–20 percentage points across datasets. Opus-4.5 leads on four of five platforms, while GPT-5.2 leads on Seeker. Seeker exhibits uniformly low accuracy across models (approximately 19–31%) despite having the largest number of evaluations.

• Harness choice materially changes outcomes. Opus-4.5 reaches 61.7% with the Latch harness versus 48.1% with Claude Code and 38.4% in the base configuration. The Claude Code to Latch uplift of 13.6 points exceeds the Opus–Sonnet gap under Claude Code (48.1% vs. 45.1%).

• Harness gains concentrate on multi-step programming tasks. Latch versus Opus-4.5 base: clustering (65.9% vs. 33.3%), differential expression (64.1% vs. 37.2%), and dimensionality reduction (75.6% vs. 51.1%). Gains are also robust across spatial platforms.

• Step counts differ dramatically by family. Claude and GPT models complete tasks in approximately 2–3 steps on average (mean 2.1–2.8), Grok variants require nearly four times as many (mean 9.8–9.9), and Gemini is intermediate (3.6). All 119 instances of 100-step limit exhaustion occurred exclusively in Grok runs, and all resulted in failure.

• More steps help only some models. For Opus-4.5, pass rate rises monotonically from 26.0% for one-step runs to 50.0% for runs with six or more steps. Grok-4 shows nearly flat pass rates across step buckets (approximately 27–31%), indicating retry-driven behavior. Gemini exhibits bimodal behavior, with 50.6% of runs completing in a single step and 21.5% requiring six or more.

• Instruction following differentiates families. Grok variants average more than seven format errors per evaluation, accumulating over 6,000 total errors across both variants. GPT-5.2 produces zero format errors, and Claude and Gemini models exhibit at most 0.06 errors per evaluation.

• Domain calibration diverges on quality control. In assays where per-cell gene counts are intrinsically low, reasonable min_genes thresholds lie between 5 and 20. Opus-4.5 applies spatially calibrated cutoffs (median 10), whereas other models default to scRNA-seq-like thresholds (median 100–200). This aligns with Opus-4.5's higher QC pass rate (approximately 25%) relative to GPT and Grok models (approximately 5%).

• Inspection is not the same as utilization. Opus-4.5 checks adata.uns most frequently (34.0% of evaluations vs. 17–18% for GPT models). When Opus inspects adata.uns, its pass rate increases by 26 percentage points (56.6% vs. 30.7%), compared to just 4–6 percentage points for Grok despite similar inspection rates (approximately 26%).

Methodology in Plain English

The authors partnered with scientists and spatial technology manufacturers to break real end-to-end analysis workflows into individual gradeable steps, such as quality control, normalization, and cell typing. For each step they froze the data state just before that step, wrote a natural-language prompt describing the goal at a scientific level, specified an output format, and wrote a deterministic grader that checks whether the agent recovered the key biological result.

Three design criteria guided problem selection: verifiability (each task has an automatically checkable success condition), scientific durability (the intended conclusion holds across reasonable algorithms and hyperparameters, so the benchmark tests interpretation rather than implementation detail), and anti-shortcut structure (tasks require interacting with the data, and items solvable from prior knowledge alone were removed). All problems underwent manual quality control, including inspection of agent trajectories across multiple runs and adversarial testing against agents instructed to exploit shortcuts.

Graders fall into five families: MultipleChoice (45 evaluations), MarkerGenePrecisionRecall (45), NumericTolerance (36), LabelSetJaccard (18), and DistributionComparison (3). Graders inspect only the final structured answer, not intermediate reasoning, and include conservative tolerances for scientifically equivalent answers.

Models were run under multiple harnesses — the system prompt, tools, control flow, answer-schema enforcement, and runtime environment wrapping a base model. Each evaluation was run three times (K = 3), with the evaluation item as the statistical unit. Per-eval mean pass rates were computed first, then aggregated into accuracy with Student-t based 95% confidence intervals. Efficiency metrics (steps, latency, cost) were summarized the same way.

Why This Matters

SpatialBench addresses a gap between benchmarks of general biological knowledge and the actual work of manipulating messy data objects. It shows that agent performance on real spatial analysis is far from solved and that improvements will not come from base models alone.

Real-world applications:

• Assisting biologists without programming skills. The benchmark targets workflows where a scientist would otherwise need procedural coding to answer a question from raw data.
• Quality-control calibration. The min_genes findings identify a concrete failure mode where models apply scRNA-seq conventions to assays with intrinsically low per-cell gene counts.
• Platform-aware tooling. The 15–20 point swings across assays argue for assay-specific tools and self-calibration rather than one-size-fits-all pipelines.
• Test-driven agent development. The benchmark is designed to support iterative diagnosis of agents that interact with spatial data faithfully, transparently, and reproducibly.

Industry relevance: The authors are affiliated with LatchBio in San Francisco, and the harness comparison speaks directly to practical agent product design. The result that harness design can change outcomes as much as base model choice implies that tools, prompts, control flow, and execution infrastructure should be evaluated and reported as rigorously as model versions. Cost and latency differences across families also bear directly on deployment economics.

Future Directions

• Joint optimization of model and harness. Since harness choice moves accuracy by 23.3 points for the same base model, the paper argues that the full agent stack should be the unit of study and that benchmarks should report harness details as rigorously as model versions.

• Platform-aware context and tooling. Reliable agents will likely require assay-specific tools, deliberate prompt engineering, and self-calibration heuristics rather than a general workflow.

• Training exposure to representative spatial workflows. The uniformly poor performance on quality control and cell typing suggests models need exposure to spatial workflows across tissues, diseases, and assay types, either in training data or via harness-level scaffolding such as retrieval and calibration routines.

• Longer-horizon, end-to-end evaluation. The current benchmark snapshots individual steps. Longer analyses involve compounding errors and iterative revision, such as revisiting QC thresholds after poor clustering, which are not yet captured. The authors frame SpatialBench as a first focused contribution toward a broader benchmark family spanning major biological data classes.

Target Audience

Researchers and engineers building or evaluating AI agents for scientific data analysis; computational biologists and bioinformaticians working with spatial transcriptomics; benchmark designers interested in verifiable, workflow-derived evaluation; and product teams developing agent harnesses for life-science applications. The paper is also relevant to model developers seeking to understand where frontier systems fail on domain-specific reasoning rather than general coding.

Authors’ abstract

Spatial transcriptomics assays are rapidly increasing in scale and complexity, making computational analysis a major bottleneck in biological discovery. Although frontier AI agents have improved dramatically at software engineering and general data analysis, it remains unclear whether they can extract biological insight from messy, real-world spatial datasets. We introduce SpatialBench, a benchmark of 146 verifiable problems derived from practical spatial analysis workflows spanning five spatial technologies and seven task categories. Each problem provides a snapshot of experimental data immediately prior to an analysis step and a deterministic grader that evaluates recovery of a key biological result. Benchmark data on frontier models shows that base model accuracy remains low (20-38% across model families), with strong model-task and model-platform interactions. Harness design has a large empirical effect on performance, indicating that tools, prompts, control flow, and execution environment should be evaluated and improved as first-class objects. SpatialBench serves both as a measurement tool and a diagnostic lens for developing agents that can interact with real spatial datasets faithfully, transparently, and reproducibly.