Protein Autoregressive Modeling via Multiscale Structure Generation

AI summary

Overview

Research area: Deep generative modeling of proteins, specifically autoregressive (AR) modeling of three-dimensional protein backbone structure.

Technical level: Advanced. The paper combines autoregressive sequence modeling, multi-scale representation learning, and flow matching on continuous 3D coordinates.

One-sentence scope: The paper introduces PAR, a multi-scale autoregressive framework that generates protein backbones coarse-to-fine by next-scale prediction, with a flow-based decoder producing Cα coordinates directly in continuous space.

What This Paper Is About

Nearly all current protein backbone generators are diffusion or flow-matching models that operate at a single scale, while autoregressive (AR) modeling — the paradigm behind large language models — has been largely absent from this domain. The paper argues this is because AR approaches for continuous 3D data typically require discretizing structures into tokens (losing fidelity), and because standard AR models assume unidirectional ordering, which conflicts with the strong bidirectional dependencies between protein residues. PAR's goal is to bring AR modeling to protein backbone design by predicting structure across multiple scales of granularity instead of along a single sequence order.

Key Contributions

1. A first-of-its-kind framework. PAR is presented as the first multi-scale autoregressive model for protein backbone generation, addressing the two limitations (discretization loss and unidirectional ordering) that have kept AR modeling out of this area.

2. Three coupled components. The framework consists of multi-scale protein downsampling (which produces coarse-to-fine training context and targets), a non-equivariant autoregressive transformer that produces scale-wise conditional embeddings, and a flow-based backbone decoder that models Cα atoms directly in continuous space.

3. Exposure-bias mitigation. The authors adapt noisy context learning (NCL) and scheduled sampling (SS) for multi-scale protein AR modeling, letting the model learn from corrupted and self-generated context rather than only ground truth.

4. Interpretable, flexible, and efficient generation. The model forms a global topology at coarse scales and refines detail at finer scales, supports zero-shot prompt-based generation and motif scaffolding without fine-tuning, and yields a reported FPSD of 161.0 against the PDB after fine-tuning.

Main Findings

• Unconditional generation quality. After a two-stage training procedure (200K steps on the AFDB representative dataset, then 5K steps fine-tuning on a PDB subset of 21K designable samples), PAR reached 96.6% designability with an FPSD of 161.0 against PDB and 228.4 against AFDB, and an sc-RMSD of 1.04. Before PDB fine-tuning, PAR at 400M parameters reported 96.0% designability and an sc-RMSD of 1.01.

• Noise scaling matters. Reducing the SDE noise scaling parameter γ from 0.45 to 0.3 reduced sampling stochasticity and improved designability from 88.0% to 96.00%.

• Favorable scaling behavior. Training 3-scale PAR at 60, 200, and 400 million parameters over 200, 400, and 600K steps improved distributional metrics, reaching FPSD scores of 187 against PDB and 170 against AFDB. Extending training duration alone gave negligible gains, while increasing model size substantially enhanced designability and lowered sc-RMSD. A moderately sized autoregressive transformer (60M) was found sufficient.

• Exposure bias is a real problem, and the proposed fixes work. With teacher forcing only, the 60M PAR trained for 100K steps reported an sc-RMSD of 2.20 and FPSD of 99.66 (PDB) / 37.64 (AFDB). Adding noisy context learning improved sc-RMSD to 1.58 and FPSD vs. AFDB to 23.69. Adding scheduled sampling on top reached an sc-RMSD of 1.48.

• Zero-shot prompt-based generation. Using a 5-scale PAR with scales {16, 32, 64, 128, 256}, a 16-point prompt (obtained by downsampling a real structure or specified manually) initializes the first scale, and the model autoregressively upsamples to a full backbone without fine-tuning. The paper reports this preserves coarse layout in some cases and explores entirely novel structures in others, and notes longer prompts such as 32 points could give finer control.

• Zero-shot motif scaffolding. Also with a 5-scale model, the model teacher-forces ground-truth motif coordinates at each scale before propagating to the next, producing plausible scaffolds that preserve motifs with high fidelity and introduce new secondary structure elements — without extra conditioning or fine-tuning. The paper states this is benchmarked in Table 11 of the appendix.

• Sampling efficiency through scale orchestration. Because coarser scales have shorter sequences and the first scale has a fixed size of 64, PAR can mix sampling methods across scales. Using SDE only at the first scale and ODE for the remaining scales reduced sampling from 400 steps to 2 steps at the last two scales without harming designability (97%), which the paper reports as a 4.7x inference speedup. Compared with the Proteina 400-step baseline, PAR achieved 1.96x and 2.5x speedup at lengths 150 and 200 respectively. Naively cutting SDE steps to 50 dropped designability to 22%, and Proteina's own designability fell to 89% at 200 SDE steps.

• Scale configuration ablation. Among configurations tried with 60M PAR, {64, 128, 256} gave 85.0% designability with FPSD 279.63 (PDB) and 267.35 (AFDB); {64, 256} gave 83.0% with 282.85/274.32; {64, 128, 192, 256} gave 77.8% with 296.70/282.69; {64, 96, 128, 192, 256} gave 81.0% with 276.00/263.58; and ratio-based {L/4, L/2, L} gave 86.4% with 310.64/298.30. Defining scales by length was reported as slightly better for modeling data distributions and adopted as the default.

• Attention behaves hierarchically. Averaged attention scores over 50 test samples with lengths in (128, 256] across five scales showed that most scales barely attend to the first scale (its input is a bos token), each scale primarily attends to the previous scale, and the model retains non-negligible attention to earlier scales.

• Comparison with baselines. In the paper's Table 1, PAR variants are compared against FrameDiff (17M), RFDiffusion (60M), ESM3 (1.4B), Genie2 (16M), and Proteina (200M and 400M), evaluated with designability, sc-RMSD, FPSD against PDB and AFDB, fS metrics, TM-score-based diversity and novelty, and secondary structure content. Notably, ESM3 (1.4B) reported 22.0% designability and FPSD of 933.9 (PDB) / 855.4 (AFDB), and the authors state PAR's fine-tuned result of 161.0 FPSD against PDB highlights superior distributional fidelity relative to pure diffusion baselines. Results for non-Proteina baselines were taken directly from Geffner et al.

Methodology in Plain English

The model treats a protein backbone as a stack of representations at progressively finer resolution, rather than a linear string of residues. Starting from a full Cα backbone of length L, deterministic downsampling interpolates it into a small number of 3D centroids at the coarsest scale, then progressively finer scales, ending at the original full-resolution structure. The model then learns, scale by scale, to predict the next finer scale given all coarser ones — the same coarse-to-fine idea used in multi-scale image generation, and the authors describe it as analogous to sculpting a statue.

Two networks do the work. An autoregressive transformer reads the tokens of all preceding scales (each previous scale is interpolated up to the current length, with a learnable beginning-of-sequence token prepended) and outputs a conditioning embedding for the current scale. A separate flow-based decoder then takes that embedding and generates the Cα coordinates for that scale, trained with a flow-matching objective against interpolated noise. Because the decoder works on continuous coordinates, no tokenization or discretization of protein structure is needed. Positional indices for each scale are drawn with linspace from 1 to L, so coarse scales have widely spaced indices (encouraging global layout) and fine scales have dense ones (encouraging local detail). Self-conditioning and a learnable scale embedding are added as extra conditions, and KV caching is used during inference.

To combat exposure bias — the mismatch between training on ground-truth context and inference on the model's own predictions — the authors add two techniques. Noisy context learning interpolates random noise into the ground-truth prior-scale inputs during training (context only, not targets), forcing the model to work with imperfect context. Scheduled sampling runs the forward process across scales during training and, with probability 0.5, replaces ground-truth context at later scales with the decoder's own prediction. Training also used a two-stage recipe, first on AFDB representative data and then fine-tuning on a designable PDB subset, and both PAR and Prote

Authors’ abstract

We present protein autoregressive modeling (PAR), the first multi-scale autoregressive framework for protein backbone generation via coarse-to-fine next-scale prediction. Using the hierarchical nature of proteins, PAR generates structures that mimic sculpting a statue, forming a coarse topology and refining structural details over scales. To achieve this, PAR consists of three key components: (i) multi-scale downsampling operations that represent protein structures across multiple scales during training; (ii) an autoregressive transformer that encodes multi-scale information and produces conditional embeddings to guide structure generation; (iii) a flow-based backbone decoder that generates backbone atoms conditioned on these embeddings. Moreover, autoregressive models suffer from exposure bias, caused by the training and the generation procedure mismatch, and substantially degrades structure generation quality. We effectively alleviate this issue by adopting noisy context learning and scheduled sampling, enabling robust backbone generation. Notably, PAR exhibits strong zero-shot generalization, supporting flexible human-prompted conditional generation and motif scaffolding without requiring fine-tuning. On the unconditional generation benchmark, PAR effectively learns protein distributions and produces backbones of high design quality, and exhibits favorable scaling behavior. Together, these properties establish PAR as a promising framework for protein structure generation.