Auto-Research Agent on MSA Organization to Improve Protein-Protein Complex Structure Prediction

AI summary

Overview

Research area: Computational structural biology and AI agents — specifically, automating the construction of multiple sequence alignments (MSAs) for protein–protein complex structure prediction with AlphaFold 3.

Technical level: Advanced. The paper assumes familiarity with MSAs, co-evolutionary signals, DockQ, and AlphaFold-style confidence metrics (pLDDT, ipTM, pTM, PAE).

Scope: An LLM-driven "auto-research" agent that, for each target complex, iteratively redesigns which homolog sequences go into the MSA and how they are paired across chains, using only a frozen AlphaFold 3's label-free confidence feedback, evaluated on 1,015 protein–protein interfaces across three post-training-cutoff benchmarks.

What This Paper Is About

Structure predictors like AlphaFold 3 depend heavily on the MSA fed into them, but existing pipelines apply one uniform recipe to every target — typically maximizing MSA depth and pairing sequences by species. The authors show that the value of depth and pairing is strongly target-dependent, so no single recipe is best across interfaces, and even whether cross-chain pairing helps AF3 is contested. Their goal is an autonomous agent that treats a frozen AF3 as a black-box instrument and, for each new target, hypothesizes, tests, and revises a target-specific MSA organization using only predictor confidence signals — never ground-truth structures — while often using far shallower MSAs than the default.

Key Contributions

1. An autonomous auto-research agent for MSA organization. The framework keeps the frozen structure predictor in the loop: the agent proposes MSAs, reads prediction-derived feedback, writes and runs its own analysis programs, and refines sequence selection and pairing across rounds, with a persistent memory of round summaries as the only cross-round information.
2. State-of-the-art results across three benchmarks. On 1,015 protein–protein interfaces from three benchmarks released after AF3's training cutoff, the agent improves overall interface DockQ over AF3's default pipeline and over fixed MSA recipes (mMSA, pMSA, uMSA) on every benchmark, while using on average 60% fewer MSA sequences than the default.
3. Demonstration that MSA construction quality, not depth, drives accuracy. The agent's nominated MSAs average only 40% of AF3-Default's depth and half retain fewer than 20% of its rows, yet outperform both the deep default and shallower fixed recipes — what matters is which rows are kept, not how many.
4. Target-specific biological reasoning made explicit and reproducible. The agent converts biological diagnoses (paralog mixing, repeat-module families, missing unpaired alignments) into concrete, per-target selection and pairing procedures — including deliberate non-pairing — validated through its own ablations using confidence alone.

Main Findings

• Overall DockQ improves on all three benchmarks. Mean DockQ rises from 0.584 to 0.604 on FoldBench-v1-pp, from 0.556 to 0.583 on FoldBench-v2-pp-hard, and from 0.583 to 0.624 on HD439. The agent also achieves the highest average DockQ ranking and DockQ oracle scores across benchmarks.
• Statistically significant on most subsets. Gains in mean DockQ over AF3-Default are statistically significant in six of the eight interface-type subsets across the three benchmarks.
• Largest gains on inter-species heteromers. Mean DockQ increases from 0.465 to 0.612 on FoldBench (inter-species heteromeric, N=19) and from 0.482 to 0.562 on HD439 (N=99) — the regime where species-based pairing is unreliable.
• Gains do not come from extra sampling. All methods are scored on the same 25 structures from five fixed seeds; the agent's additional predictor runs (6.01 per target on average) are used only for search.
• Gains survive re-prediction with fresh seeds. Because nomination uses confidence on the same structures, the authors re-predict nominated MSAs with fresh seeds and find the gains persist.
• Shallower but better. AF3-Default mean MSA depth is 5,744.5 (FBv1), 10,928.3 (FBv2), and 8,940.7 (HD439); the agent's is 2,248.4, 4,097.9, and 3,417.9 respectively. mMSA is even shallower on FBv1 (904.6) and FBv2 (1,358.6) but yields lower DockQ, showing shallowness alone is not the cause.
• Sequence construction vs. pairing policy separated by a shuffle control. Randomly shuffling row order within each chain preserves sequence content but destroys the proposed pairing structure. For homomers and inter-species heteromers, shuffling does not significantly change mean DockQ, indicating gains come mainly from sequence construction. For intra-species heteromers, shuffling significantly lowers mean DockQ; on the 339 intra-species HD439 targets, preserving the pairing policy accounts for approximately half the gain over AF3-Default (0.014 out of 0.029).
• Exploration and nomination both limit the final result. On 278 interfaces from 238 matched FBv1-pp targets with predictor feedback, the exploration gain exceeds 0.05 on 34 interfaces and nomination regret exceeds 0.05 on 13. The best tested proposal averages 0.035 above AF3-Default, but nomination gives back 41.0% of that gain, leaving a final improvement of 0.020.
• Ground-truth feedback improves both stages. In an oracle setting where DockQ is visible, nomination regret is near zero and exploration gain rises from 0.035 to 0.059 (a 71.7% increase), suggesting the fidelity of predictor feedback is a key limiting factor.
• Case study 8P9C (p63/p73 tetramerization domains plus DARPin 1810 F11). In the default input, 84.5% of the 16,384 rows contained only DARPin sequences, giving mean DockQ 0.048. The agent separated p63 and p73 candidates, paired them by species, restricted DARPin sequences to a small filtered supplement, and reached mean DockQ 0.866 with only 425 rows.
• Case study 7R86 (RTN4R leucine-rich repeat domain plus BAI1 thrombospondin type 1 repeat). Species matching yielded 8,191 pairs and chain B lacked an unpaired alignment (mean DockQ 0.037). The agent kept only 78 high-identity pairs as likely orthologs, recovered 472 B-only rows from the pairing candidate pool, and reached mean DockQ 0.921.

Methodology in Plain English

The core idea is to let an LLM act as a research assistant for each protein complex, with AlphaFold 3 as the experimental instrument.

• Setup. A target complex is given to the agent as raw inputs. The agent produces an MSA proposal: which homolog sequences to include for each chain (the unpaired MSA) and which of those sequences to line up in the same rows across chains (the paired MSA). The proposal is submitted to a frozen AlphaFold 3, which returns structures and confidence signals.
• Double loop. An outer episode consists of many proposal rounds. Each round has a Construction phase, where the agent analyzes the target and earlier findings until it submits a proposal, and a Reflection phase, where it interprets the feedback and writes a structured summary. A single LLM invocation is a "turn," and a round typically spans many turns.
• Actions. In Construction the agent can run analysis, submit, or finish; in Reflection it can run analysis or submit_round_summary. analysis executes an agent-written Python program in an isolated sandbox with network access disabled (to prevent ground-truth leakage), a 120-second limit, and write access only to the agent-writable part of its workspace.
• What the agent sees. The context is fixed system prompt plus summaries of prior rounds plus the current round's action/output history. Target inputs and workspace files never enter the context automatically; the agent must read them via analysis. The workspace persists across the episode even though the LLM context resets each round.
• Feedback without labels. Feedback includes ensemble mean and maximum of ipTM, pTM, and AF3's native ranking score, plus access to per-sample coordinates, per-chain and chain-pair confidence, PAE, per-atom pLDDT, contact probabilities, disorder fractions, and clash indicators. No fixed scalar surrogate is imposed and there is no rule to maximize mean ipTM — the agent weighs interface confidence, cross-sample consistency, biological and structural plausibility, and counterevidence. DockQ is computed silently for evaluation and never shown to the agent in the main setting.
• Round 1 is a mandatory control that submits the default AF3 input, so the default is always a nominable candidate.
• Budgets. At most 60 completed generator runs per episode (including the initial default) and 600 LLM calls total, with up to two extra turns reserved for the pending round summary and finish.
• Evaluation. Predictions use AF3 v3.0.2 with the FoldBench protocol: 10 recycles and five diffusion samples for each of five fixed seeds, giving 25 structures per submission; query sequences and templates are held fixed so differences come only from the MSA. Scoring uses DockQ, decomposed into mean, ranking, and oracle variants and macro-averaged over interfaces.
• Agent model. GLM-5.3 at temperature 0.2, maximum reasoning effort, maximum response length 32,768 tokens, one independent episode per target.
• Baselines. AF3-Default (the standard pipeline's MSAs and species-based pairing), mMSA (monomeric alignments only), pMSA (plus cross-chain paired sequences), and uMSA (plus unpaired UniProt hits). On HD439, all conditions use the benchmark's released MSAs, so comparisons there are input-matched.

Why This Matters

Impact on research. The paper reframes MSA design from a fixed preprocessing recipe into a target-specific decision problem that can be optimized at test time without retraining the predictor. It gives a concrete counterexample to the aggregate claim that depth matters more than pairing: fixed heuristics behave as reported, but the agent reverses the intuition by being both shallower and better. It also introduces a careful methodology for evaluating agents whose evaluator is untrustworthy, by separating exploration gain from nomination regret.

Real-world applications:

• Antibody and antigen engineering, where the paper notes the depth-versus-pairing trend extends to antibody–antigen complexes and where species-based pairing is least reliable.
• Viral–host interface studies, an inter-species heteromer setting where the agent's largest gains appear (0.465 to 0.612 on FoldBench; 0.482 to 0.562 on HD439).
• Structure-prediction pipelines and servers, which could use shallower, agent-selected MSAs to cut compute — the agent averages 60% fewer sequences than the default — while improving interface accuracy.
• Drug discovery for complexes with sparse or misleading homology, such as designed binders with no natural orthologs (the 8P9C DARPin case) or widespread repeat modules (the 7R86 case).

Industry relevance. MSA search is a major cost in commercial structure-prediction workflows, and interface accuracy is what matters for biologics design. A training-free wrapper that improves DockQ while shrinking alignments is directly deployable on top of existing predictors, and the agentic pattern — test-time search guided by a model's own confidence signals — generalizes to other preprocessing decisions where no labels exist.

Future Directions

• Improving the fidelity of predictor feedback. The authors identify this as the key bottleneck: with ground-truth feedback, exploration gain rises from 0.035 to 0.059 (71.7%) and nomination regret nearly vanishes, so better-calibrated or richer proxy signals are the most promising lever.
• Reducing nomination regret. Under predictor feedback the agent gives back 41.0% of its best-found gain, meaning better selection among already-tested proposals is a separate, largely unsolved problem.
• Extending beyond the tested scope. Evaluation covers three protein–protein benchmarks and one predictor (AF3 v3.0.2); whether the approach transfers to other predictors, to nucleic-acid-containing complexes, or to interfaces excluded here (e.g., 8bvk, dropped because LLM API content filtering interrupted the agent) is untested.
• Reducing cost and scaling out. Each episode permits up to 60 generator runs and 600 LLM calls, and the agent uses 6.01 extra predictor runs per target on average; cheaper search and greater robustness of the harness to API-level interruptions would be needed for large-scale deployment.

Target Audience

Most valuable to researchers and engineers working on protein complex structure prediction, MSA construction and pairing methods, and biological foundation-model pipelines — particularly those deploying AlphaFold 3 or similar predictors who want accuracy gains without retraining. It is also relevant to the LLM-agent community as a case study of evaluator-guided search in a setting where the evaluator is a noisy proxy rather than the true objective, and to computational biologists interested in how co-evolutionary signal should be curated per target.

Authors’ abstract

Protein-complex structure prediction with AlphaFold 3 depends on MSA content, depth, and cross-chain pairing. Current pipelines apply uniform recipes, typically maximizing depth and pairing by species, yet even whether cross-chain pairing helps AF3 remains contested. We show that the value of both depth and pairing is strongly target-dependent, so no single recipe is optimal across interfaces. We introduce an autonomous agent that treats a frozen AF3 as a black-box experimental instrument. For each target, the agent formulates biological hypotheses, analyzes and reorganizes the MSA, and iteratively refines sequence selection and pairing using predictor label-free signals alone. On 1{,}015 protein--protein interfaces from three benchmarks released after AF3's training cutoff, the agent improves overall interface DockQ over AF3's default pipeline and fixed MSA recipes on every benchmark, while using on average 60\% fewer MSA sequences than the default. Gains are largest for inter-species heteromers, where species-based pairing is unreliable: mean DockQ increases from 0.465 to 0.612 on FoldBench and from 0.482 to 0.562 on HD439. The improvements arise from both sequence construction and pairing policy, including deliberate non-pairing. These results recast MSA design as a target-specific problem and show that an autonomous agent can adapt it at test time through biological reasoning and self-directed, confidence-guided experimentation.