ProtGlycanDock: Towards Accurate Protein-Glycan Docking with Tailored Dataset, Benchmark and Model

AI summary

Overview

• Research area: Computational structural biology and machine learning — specifically 3D structure prediction of protein–glycan (protein–carbohydrate) complexes.
• Technical level: Advanced. The paper assumes familiarity with molecular docking, diffusion/structure-prediction architectures, and glycan stereochemistry (glycosidic dihedral angles, anomeric linkages).
• Scope (one sentence): The paper builds a new trainable protein–glycan docking benchmark (ProtGlycanDock), benchmarks ten existing methods on it, and proposes two finetuning techniques that push an open-source structure-prediction model past AlphaFold 3.

What This Paper Is About

Predicting how a sugar molecule (glycan) physically binds to a protein is hard to do experimentally because glycans are flexible, branched and weakly binding. Computational docking should help, but existing docking tools and structure-prediction models were designed for protein–small-molecule or protein–protein problems, and existing protein–glycan benchmarks contain fewer than 100 samples and cannot be used for training. The paper's goal is to fill that gap with a trainable, generalization-focused benchmark plus targeted finetuning methods that make general-purpose structure models better at glycan docking.

Key Contributions

1. A trainable protein–glycan docking benchmark (ProtGlycanDock). Starting from 13,716 PDB structures containing at least one glycan, the authors keep 2,150 noncovalent protein–glycan complexes, remove structures with more than 4 protein chains, and end with 1,781 repaired structures split into 1,374 training, 125 validation, 134 glycan-holdout test and 148 protein-holdout test samples.
2. Two held-out generalization test sets. A glycan-holdout set built from the 45 most recently deposited of 332 distinct glycans (134 samples), and a protein-holdout set of 31 under-represented MMseqs2 clusters (148 samples), 20 of which have fewer than 5 PDB samples and the largest only 8.
3. A systematic benchmark of ten models in three categories — physics-based docking (AutoDock Vina, Vina-Carb), machine-learning docking (DiffDock, SurfDock), and molecular structure prediction (AlphaFold 3, RoseTTAFold All-Atom, Chai-1, Boltz-2, Protenix, Protenix-constraint) — all under the blind-docking setting with no ground-truth receptor structure or binding-site information.
4. Two complementary finetuning techniques for injecting glycan knowledge into a general-purpose structure model: glycan topology embedding at the input level (37 monosaccharide types, 11 glycosidic bond types) and a Carbohydrate Intrinsic (CHI) energy loss on glycosidic dihedral angles, combined as L = L_pred + γ·L_CHI with γ = 0.01.

Main Findings

• AlphaFold 3 is the strongest off-the-shelf model. Without finetuning it leads on RMSD metrics, contact recovery and the count of geometrically valid and physically plausible poses. On the protein-holdout test it reaches 34.6 samples with Ligand RMSD < 2 Å (24.8 of which also pass all PoseBusters tests), Ligand RMSD 14.27 Å, Ring sRMSD 0.73 Å, Complex RMSD 7.98 Å and 34.7% contact recovery. Its 0.73 Å Ring sRMSD on that split is the lowest among all models, including finetuned ones.
• Learned docking beats physics-based docking. DiffDock (12.8 samples under 2 Å on protein-holdout) and SurfDock (15.2) clearly outperform AutoDock Vina (1.2) and Vina-Carb (2.0) on valid and plausible poses and on contact recovery (SurfDock 26.3% vs. AutoDock Vina 17.2%).
• Structure-prediction models dominate blind docking. All evaluated structure-prediction models except RoseTTAFold All-Atom outperform all four docking methods on both test sets; RoseTTAFold All-Atom itself scores only 1.8 samples under 2 Å on the protein-holdout split.
• Finetuning alone already helps. Finetuned Protenix-constraint without extra techniques improves from 30.0 to 35.6 samples under 2 Å (protein-holdout) and from 28.4 to 32.4 (glycan-holdout).
• The two techniques are complementary. Each helps individually, and combined they give the best model: finetuned Protenix-constraint with topology embedding and CHI loss reaches 42.0 samples under 2 Å (31.2 PoseBusters-valid), Ligand RMSD 11.93 Å, Ring sRMSD 0.84 Å, Complex RMSD 6.79 Å and 44.7% contact recovery on the protein-holdout set, and 42.2 / 31.8 / 11.03 Å / 1.02 Å / 2.89 Å / 45.2% on the glycan-holdout set — best on 11 of 12 evaluation metrics.
• Performance degrades with glycan size, but finetuning narrows the gap. On the glycan-holdout set the best finetuned model goes from 10.00 Å (2–4 monosaccharides) to 19.02 Å (>7 monosaccharides), a 9.02 Å gap, versus 19.88 Å for non-finetuned Protenix-constraint (12.18 Å to 32.06 Å).
• The same holds for protein size. On the protein-holdout set the best finetuned model spans 8.19 Å to 13.18 Å (gap 4.99 Å), compared with 10.07 Å to 18.59 Å (gap 8.52 Å) for the non-finetuned counterpart.
• Temporal generalization is stable. Across deposition periods from 1990–2010 to 2021–2026, Ligand RMSD rises only by 1.38 to 1.80 Å for all six analyzed models (staying below 2 Å), and the model ranking is unchanged.
• Single-case evidence for the CHI loss. On PDB 2JEN, AlphaFold 3 obtains a Ligand RMSD of 4.348 Å while the finetuned Protenix-constraint obtains 0.699 Å, with the difference concentrated in one glycosidic bond; the finetuned model recovers a hydrogen bond reported by Gloster et al. that AlphaFold 3 misses.
• Two failure modes remain. For 3BC9 the model places the glycan at the wrong binding site; for 7DKU (Cellobiose, a glycosyl donor) it predicts a deeply bound conformation where the ground truth is more shallowly bound, leaving space for an acceptor.

Methodology in Plain English

The authors first assembled data. They pulled every PDB entry containing at least one glycan (13,716 structures), discarded covalently attached glycosylation to keep only noncovalent binding events (2,150), and dropped overly complex assemblies with more than 4 protein chains (1,781 left). They repaired the structures with PDBFixer for non-standard residues plus Amber energy minimization, and MODELLER loop modeling with DOPE scoring for missing residues — noting that all residues requiring fixing were more than 6 Å from the glycan, so binding pockets were untouched.

Splitting was designed to test generalization rather than memorization. The glycan-holdout set takes the 45 most recently deposited glycans. Because the newest complexes were already used there, the protein-holdout set instead uses MMseqs2 clustering to select 31 small, under-represented protein clusters that are hard for baselines to have memorized from PDB-scale pretraining (baseline training cutoff: 2023-06-01). The validation set deliberately mixes both kinds of novelty: the 25 clusters richest in glycans unique to that cluster.

For the modeling contribution, the authors take Protenix and Protenix-constraint — chosen because their training pipeline is open-source — and finetune them. The first technique fixes a blind spot: existing models treat each monosaccharide as a separate small molecule with a reference conformer, which captures local atoms but not the glycan's global branching. The authors therefore add a 37-dimensional one-hot embedding of monosaccharide type into the single representation of all atoms in that sugar, and an 11-dimensional one-hot embedding of glycosidic bond type into the pair representation of the linked atom pairs. The second technique adds a physics-informed penalty: the CHI energy function, a set of Gaussian expansions fitted to quantum-mechanical rotational energies for φ and ψ dihedral angles in α- and β-linkages, is averaged over all K glycosidic bonds and added to the prediction loss with weight γ = 0.01, chosen because of the scale of the CHI term relative to the original loss. The idea is that a glycan in a real complex should sit at a low-energy conformation.

Training used Protenix's Finetuning Stage 2 recipe (crop size 640, training the PAE head, distogram head and diffusion module), EMA decay 0.9999, learning rate 1×10⁻⁴, Adam optimizer, 15,000 steps, with validation every 100 steps and checkpoint selection by best average ranking score. All models were run blind with seeds 42–46 to produce error bars, on 100 CPU cores and 8 NVIDIA HGX H20 GPUs (96 GB). Evaluation covers geometry (Ligand RMSD, Ring sRMSD, Complex RMSD, contact recovery) and physical plausibility (PoseBusters tests for stereochemistry, bond lengths and angles, planarity, steric clashes and conformational energy).

Why This Matters

Protein–glycan interactions underpin cell metabolism, immune response, pathogen recognition and cellular adhesion, but experimentally solving these complexes is difficult because glycans are flexible, branched and weakly binding. A reliable computational route to protein–glycan structures directly supports biology and drug discovery.

• Antibody and therapeutic engineering: Many biologics carry glycans that affect stability and immune recognition; accurate docked poses help interpret and design glycosylation.
• Immunology and vaccine design: Glycan recognition drives immune signaling and pathogen detection, so pose-level models support epitope and lectin-binding analysis.
• Enzyme and biocatalysis work: The 7DKU failure case shows why donor/acceptor geometry matters — relevant to glycosidase and glycosyltransferase engineering.
• Structure-determination pipelines: A benchmark with protein-holdout and glycan-holdout splits gives model developers a realistic testbed before deployment on newly discovered systems.

Industry relevance: The work is directly usable by structure-based drug design and biotherapeutics groups, and by developers of structure-prediction platforms, since it shows that finetuning an open-source model with a small, domain-specific dataset and a cheap energy term can surpass a closed frontier system on this task.

Future Directions

• Scaling the data. The authors state the benchmark is not large enough and plan to synthesize large-scale protein–glycan complexes via cross-docking.
• Relaxing the blind setting. The current benchmark is blind docking only; the authors intend to extend it to cases with partially known apo receptor structures and binding sites.
• Better binding-site and functional-context modeling. The 3BC9 and 7DKU failures suggest gains require improved protein–glycan recognition modeling and awareness of the glycan's biological role (e.g., donor versus acceptor).
• Combining physical priors with learned representations more broadly. Whether the topology-embedding and CHI-loss recipe transfers to other models with open training pipelines, and how sensitive it is to the loss weight γ (analyzed only in the appendix), remain open questions.

Target Audience

Machine-learning researchers working on molecular structure prediction and generative docking; computational chemists and structural biologists interested in carbohydrate–protein recognition; and practitioners who need practical guidance on which docking or structure-prediction tool to trust for glycans, and on how much finetuning buys. Readers without background in glycan stereochemistry or protein structure models will find the experimental sections accessible but the methodology sections demanding.

Authors’ abstract

Performing accurate protein-glycan docking is critical to understanding protein-glycan interactions which govern many biological processes. However, this important task is hampered by the lack of tailored datasets, benchmarks and models for protein-glycan docking. To fill this blank, we first curate a standard benchmark ProtGlycanDock for training and evaluating protein-glycan docking models under both protein and glycan generalization settings. On this benchmark, we evaluate ten models in three categories, physics-based docking methods, machine-learning-based docking models and molecular structure prediction models. To obtain more precise binding poses, we further propose two finetuning techniques to inject the knowledge of glycan conformations into a general-purpose molecular structure prediction model. These two techniques respectively augment model inputs with glycan topological features and refine glycan binding pose with a glycan energy loss. Benchmark results show the superiority of AlphaFold 3 among existing methods, and the two proposed finetuning techniques can be well combined to achieve a state-of-the-art model for protein-glycan docking. Also, we perform stratified benchmark analysis and case studies to better understand the strengths and weaknesses of current models, providing insights for future model improvements. We maintain datasets at https://huggingface.co/datasets/ProtGlycanDock/ProtGlycanDock and source codes at https://github.com/kasawa1234/ProtGlycanDock