CryoAtlas: A Large Curated Dataset and Unified Benchmark for Cryo-EM Atomic Model Building

AI summary

Overview

Research area: Computational structural biology and machine learning — specifically, automated atomic model building from cryo-electron microscopy (cryo-EM) density maps.

Technical level: Intermediate. Readers should know roughly what a cryo-EM density map, a protein atomic model, and a benchmark evaluation are, but the paper is written around a dataset-and-benchmark framing rather than deep architectural novelty, so it is approachable for graduate students and practitioners entering the field.

Scope: The paper introduces CryoAtlas, a curated 17,809-entry cryo-EM map–model–sequence resource with a 16,657-entry training split and a 618-target unified benchmark that jointly scores structural accuracy, model-to-map fit, and stereochemical validity across six modeling methods.

What This Paper Is About

Automated model building from cryo-EM density maps has advanced quickly, but each research group evaluates its method on its own small, differently filtered test set with its own metric implementations, so reported gains cannot be cleanly compared. The authors address this by building a large curated dataset from EMDB and PDB entries (up to March 4, 2026) and defining a single evaluation protocol that measures three distinct aspects of model quality instead of one global similarity number. The goal is to make it possible to ask, and answer reproducibly, which method is most reliable for a given map.

Key Contributions

1. A large curated multimodal dataset. From 24,052 starting EMDB entries, the authors apply cross-modal alignment, Q-score-based quality control, and removal of maps with large unmodeled regions, arriving at 17,809 map–model–sequence entries with coordinate files, FASTA sequences, and density maps — including a 16,657-entry training split.

2. A unified three-dimensional evaluation protocol. Rather than a single score, the benchmark defines composite scores for stereochemistry (Geom Score), structural accuracy against the deposited reference (GT Score), and model-to-map fit (Map Score), plus the raw metrics behind each, so strengths and failure modes remain visible.

3. A 618-target benchmark with stratified analysis. All targets are ones on which five learning-based methods succeeded, drawn from a 1,152-map held-out pool that is far larger than prior benchmarks (ModelAngelo's 177 targets and Cryo2StructData's 390 targets, both restricted to pre-March-2023 structures). Performance is broken down by map resolution, sequence length, protein category, and test–train sequence identity.

4. A training-utility case study. Retraining ModelAngelo's CNN component on the CryoAtlas split, under a shared recipe, improves downstream benchmark accuracy relative to training on Cryo2StructData and slightly exceeds the released ModelAngelo checkpoint.

Main Findings

• CryoAtom2 leads among density-guided methods. It achieves the highest GT Score (0.787) and Map Score (0.568) of the density-guided methods, with sequence recall 0.899 and TM-score 0.763. ModelAngelo (GT 0.723, Map 0.520) and EMProt (GT 0.702, Map 0.517) are comparable to each other, while E3-CryoFold is substantially lower (GT 0.539, Map 0.305).

• AlphaFold 3 has the best geometry but the worst density agreement. AF3 records the highest Geom Score (0.598) and the highest TM-score (0.813), yet its Q-score is 0.202 and its Map Score 0.220 — the lowest of the learning-based methods — because sequence-only prediction does not commit to the experimental density or its coordinate frame.

• Map resolution is the dominant performance factor. GT Score declines with decreasing resolution for all models, while Geom Score stays largely stable, indicating that resolution mainly harms structural accuracy and density fitting rather than stereochemical plausibility. CryoAtom2 is the most robust to resolution degradation: comparable to EMProt and ModelAngelo above 3 Å, with a widening advantage at lower resolution.

• Sequence length matters only mildly for density-guided methods, but strongly for AF3. Map Score stays stable across length bins for density-guided methods and GT Score declines only mildly; AF3 shows a stronger length-dependent decline in both GT Score and Map Score, consistent with errors accumulating in longer chains without density constraints.

• Sequence identity to training data does not produce a clean trend. Across the four density-guided methods, score distributions overlap across the ≥70%, 30–70%, and <30% identity bins with no consistent monotonic ordering.

• Completion rates separate robustness from accuracy. CryoAtom2 completes 719/721 (0.9972) and AF3 935/947 (0.9873); EMProt 701/721 (0.9723), ModelAngelo 699/721 (0.9695), E3-CryoFold 677/721 (0.9390). Phenix completes only 366/618 (0.5922) within a 24-hour budget.

• Phenix builds few residues with plausible local geometry. Its Geom Score (0.462) matches ModelAngelo's, but sequence recall (0.063), TM-score (0.075), GT Score (0.194), and Map Score (0.193) are low because it reconstructs only limited portions of the reference structure.

• CryoAtlas retraining improves downstream accuracy. Against matched Cryo2StructData training, CryoAtlas reduces Cα RMSD from 1.108 Å to 0.769 Å and raises GT Score from 0.697 to 0.755; it also slightly exceeds the official checkpoint (GT 0.723, RMSD 1.014 Å). At 5,000 steps, CryoAtlas reaches an EMA loss of 0.0898 ± 0.0009 over three seeds, versus 0.1127 for raw Cryo2StructData; restoring discarded entries raises the loss to 0.1140, while applying CryoAtlas filters to raw Cryo2StructData lowers it to 0.0865.

Methodology in Plain English

Building the dataset. The authors start from every EMDB entry up to March 4, 2026 and keep only maps at 6 Å resolution or better. They drop entries lacking deposited structures and chains made entirely of unknown residues, leaving 21,229 triplets. Three problems remain: structures misaligned with their maps, sequences mismatched to the chains in the structure, and maps containing large regions nobody modeled. They fix alignment with ChimeraX's fitmap (improving average Q-score by 0.103 across 989 realigned entries), align full-length FASTA sequences to structure chains with Clustal Omega while marking unmodeled residues as gaps, and then discard 994 entries whose aligned Q-score falls below 0.3 plus severely incomplete models identified by an unmodeled-density ratio.

Splitting without leakage. Chains are clustered with MMseqs2 at 40% sequence identity, and maps sharing identical chain-cluster signatures are grouped into 6,062 non-redundant complex-level clusters. Sampling one representative from 10% of large clusters and all maps from 10% of small clusters yields a 1,152-map held-out pool and 16,657 training maps.

Defining the benchmark. Entries that overlap with the training sets of the density-guided baselines are excluded, leaving 721 eligible targets; AF3 is restricted to targets within its 5,000-residue server limit; the final 618 targets are those all five learning-based methods could model. Phenix is run on the same 618 as a conventional reference.

Scoring. Three composites are computed from standard tools. Geom Score aggregates seven phenix.molprobity indicators into four sub-groups (steric, backbone, sidechain, bonded), averaged within groups by geometric mean and across groups arithmetically. GT Score is a fixed weighted mix: 0.15 · TM-score + 0.45 · exp(−Cα RMSD / 3) + 0.40 · sequence recall, balancing topology, positional accuracy, and residue recovery. Map Score averages the global term (mean of four real-space correlation variants) and a coverage-weighted local Q-score term, so a method cannot win by building only the easy, well-resolved regions. Geom Score agrees with MolProbity at Spearman |ρ| > 0.87 across methods.

Running the comparison. All methods run with default parameters and no post-refinement. AF3 is run via its official server using the highest-confidence of five predictions, and both AF3 and E3-CryoFold outputs are aligned to the maps because their coordinate frames differ from ground truth. Density-guided structure-prediction methods (e.g., CryoBoltz, EmbedOpt, CryoACE) were excluded because they were not robust enough on complex targets in preliminary tests.

Retraining experiment. The authors reimplement ModelAngelo's CNN training (keeping the official GNN checkpoint and the downstream refinement stage fixed), train for 50,000 steps at effective batch size 96 on 8 × NVIDIA H20 GPUs (96 GB each) in about 39 hours, and compare against the same recipe run on Cryo2StructData maps, with additional controls for normalization, filtering, and restoring discarded entries.

Why This Matters

Impact on research. Fragmented evaluation has made it hard to tell whether a reported improvement comes from a better model or from an easier test set. CryoAtlas fixes the corpus, the split, the metric implementations, and the reporting axes, so comparisons become attributable. The finding that no single axis characterizes a method — AF3 wins on geometry while losing badly on map fit — argues directly against single-score leaderboards for this task.

Real-world applications:

• Structure-based drug discovery, where an atomic model of a target–ligand-adjacent pocket must actually explain the experimental density, not merely be physically plausible.
• Disease-mechanism studies of large complexes and conformational states that X-ray crystallography cannot easily access.
• Cryo-EM facility and annotation workflows, where completion rate over diverse real-world inputs determines whether automation is usable at all — Phenix's 0.5922 completion within 24 hours versus CryoAtom2's 0.9972 is a practical deployment signal.
• Model validation and triage, giving structural biologists a way to judge when an automatically built model needs manual correction in tools such as Coot.

Industry relevance. Pharmaceutical and structural-biology groups running high-throughput cryo-EM pipelines need to know which automated builder to trust at a given resolution and chain length, and need a reproducible yardstick for vendor or in-house tools. The released training split is also a direct asset for companies training their own density-guided models, and the paper is explicit that data curation quality — not just data scale — drives downstream accuracy, as E3-CryoFold's results show.

Future Directions

• Extending curation to nucleic acids. CryoAtlas currently keeps only protein chains, which precludes fully all-atom modeling; preserving nucleic acids is described as the natural next step.
• More robust evaluation algorithms. The authors state that composite-score design reduces but does not eliminate single-metric biases, and that better evaluation methods remain an open problem.
• Region-aware scoring. Current whole-structure scores miss class-specific functional regions such as enzyme active sites, transmembrane interfaces, antibody CDRs, and nucleic-acid contact surfaces.
• Multi-state and ensemble-aware modeling. Each density is currently treated as a single static target, leaving heterogeneous and dynamic assemblies unaddressed.

Target Audience

Method developers working on density-guided or density-conditioned structure prediction will get the most from this paper, since it supplies both a training corpus and a fixed evaluation harness. Structural biologists and cryo-EM facility staff evaluating which automated builder to adopt will find the completion rates and stratified accuracy breakdowns directly actionable. Benchmark and dataset researchers will find the curation pipeline and the three-composite scoring design worth studying as a template for other multimodal scientific benchmarks. Readers wanting novel network architectures will not find them here; the contribution is infrastructure, protocol, and measurement.

Authors’ abstract

Deep learning has accelerated automated atomic model building from cryo-EM maps, yet progress remains limited by the lack of large, high-quality datasets and standardized evaluation benchmarks. We address this gap by introducing CryoAtlas, a rigorously curated dataset and unified benchmark for density-guided atomic model building of protein chains. By integrating and filtering EMDB and PDB entries (up to March 2026), CryoAtlas provides 17,809 map-model-sequence entries, including a curated 16,657-entry training split. From this resource, we construct a 618-target benchmark with a shared evaluation protocol that jointly assesses structural accuracy, model-to-map fit, and stereochemical validity. Using this framework, we benchmark state-of-the-art learning-based methods against Phenix and AlphaFold3, comprehensively analyzing the impact of map resolution, sequence length, protein category and sequence identity on modeling accuracy. As a case study, retraining the ModelAngelo CNN component on CryoAtlas improves performance on the benchmark. CryoAtlas provides an open, reproducible foundation for future cryo-EM atomic model building research. Project details and resource release updates are available at https://oulling.github.io/cryoatlas/.