BioArc: Discovering Optimal Neural Architectures for Biological Foundation Models

AI summary

Overview

Research area: Neural Architecture Search (NAS) applied to biological foundation models — specifically DNA and protein sequence modeling, with the framework described as generalizable to RNA and single-cell modalities.

Technical level: Intermediate. The paper assumes familiarity with Transformer-style architectures, self-supervised pretraining objectives (masked modeling, contrastive learning, next-token prediction), and the general concept of weight-sharing supernets, but explains each component it introduces.

Scope: The paper presents BioArc, a framework that searches a heterogeneous space of five neural block types (CNN, LSTM, Transformer, Mamba, Hyena) to discover hybrid architectures for biological sequences, and analyzes how architecture interacts with tokenization and training strategy.

What This Paper Is About

Most biological foundation models borrow architectures designed for human language (primarily Transformers), which the authors argue lack the inductive biases needed for biological data — data that simultaneously requires very long context (whole genomes) and precise local structural motifs. BioArc replaces intuition-driven architecture design with automated, data-driven discovery, searching over a heterogeneous space of architecturally distinct blocks rather than tuning parameters within a single fixed family. The goal is to find hybrid architectures that beat state-of-the-art models while being up to 25x smaller.

Key Contributions

1. A heterogeneous NAS framework. BioArc navigates a search space composed of five distinct operation primitives (CNN, LSTM, Transformer, Mamba, Hyena) combined with variable network depth and hidden dimensions, enabling open-ended composition of complementary inductive biases rather than micro-level tuning within a homogeneous space. It discovers novel hybrid architectures that achieve state-of-the-art performance on diverse biological tasks while being up to 25x smaller than existing foundation models.

2. A unified testbed for architecture, tokenization, and training. The framework disentangles the interplay between architectural topology, tokenization schemes (k-mer and Byte Pair Encoding), and pretraining objectives (Masked Modeling, Contrastive Learning, Next Token Prediction), producing actionable design guidelines.

3. Interpretability validation. Through fine-grained analysis of layer-wise activations, the authors show that discovered topologies hierarchically capture known biological regulatory mechanisms, for example the Inr+DPE synergy rule on the core promoter task.

4. An agentic architecture predictor. An agent-based framework predicts optimal architectures for unseen tasks without exhaustive search, benchmarked against predictors based on traditional neural networks and standard LLMs.

Main Findings

• BioArc beats much larger pretrained models on DNA. On the human subset of the GUE benchmark (12 datasets), the discovered architectures — reported at 3.28M, 4.89M, and 6.58M parameters depending on pretraining objective — outperform all baselines across all DNA tasks. Baselines include HyenaDNA (6.6M parameters, 3.1B data), Nucleotide Transformer NT-2500M-1000g (2500M parameters, 20.5T data), DNABERT-2 (117M parameters, 262B data), and VQDNA (103M parameters, 262B data). As reported in Table 1, the masked-modeling variant scores 84.80 on the first transcription-factor-prediction column versus 71.99 for DNABERT-2 and 72.48 for VQDNA.

• Controlled protein comparison isolates architecture from pretraining budget. BioArc 8M and a reimplemented ESM-2 8M were pretrained under identical conditions (full UniRef50, 50K steps). BioArc 8M outperformed ESM-2 8M on all six PEER tasks, including Fold (20.75 vs. 18.25). The residual gap to the released ESM-2 8M checkpoint (Fold 22.14) tracks that checkpoint's roughly 10x larger pretraining budget (full UniRef50, 500K steps, 32x A100 versus 50K steps, 4x A100).

• Top DNA architectures converge on a recurring pattern. Optimal models typically begin with a Hyena block to capture long-range dependencies, use Transformer blocks in the middle to model contextual relationships, and end with CNN blocks to extract features. Architectures are task-specific but share significant commonality within task families: the top 10% of architectures for Transcription Factor Prediction-3 and 4 share 98.0% similarity.

• BioArc works as a foundation model backbone (BioArc-F). Using only 1/20 of the model size and 1/10 of the training steps of baselines, BioArc-F outperforms human-designed architectures on downstream DNA tasks. Scaling depth to 10 layers (doubling the Hyena and Transformer layers) with a fixed hidden dimension of 1024 across all layers further improved performance. BioArc-F is also competitive with Caduceus and GENERator on the Nucleotide Transformer benchmark and the Genomic Benchmarks, achieving the best result on all histone-mark and enhancer tasks despite being substantially smaller.

• No pretraining strategy dominates, and pretraining does not guarantee gains. Training from scratch ("only-ft") achieved the highest Win Rate on the PD-tata task at 36.0%. Masked modeling generally yielded the highest performance across TFP datasets, while Next Token Prediction was superior on the CPD-notata task. Contrastive learning showed instability and high variance relative to reconstruction-based methods, suggesting it may need longer training schedules.

• Optimal tokenization is architecture-dependent. Transformer-based architectures performed best with a 6-mer tokenizer, while CNN-based models scored highest with a 1-mer tokenizer. For the pretrained BioArc-F foundation model, BPE outperformed the 1-mer tokenizer — a reversal of the pattern seen in task-specific models trained from scratch.

• The learned hierarchy mirrors biological structure. On the core promoter task, layer 0 (Hyena) establishes global genomic context, layers 1–2 (Transformer) anchor attention on the Transcription Start Site and identify its functional boundary, and layers 3–5 (CNN) transition from localized motif detection to integration of the Inr-MTE-DPE complex. Filters simultaneously activating on Inr and DPE elements indicate unsupervised rediscovery of the Inr+DPE synergy rule.

• Hybrids beat single-module baselines, and size does not predict success. Hybrid architectures consistently outperform single-module baselines, layer-wise probing shows monotonic accuracy improvements layer by layer, and there is no positive correlation between parameter count and performance across the search space — specific hybrid topologies are the decisive factor.

• Ranking reliability. Rankings from the pretrained-plus-individually-finetuned supernet correlate only weakly with train-from-scratch rankings (Spearman's ρ = 0.3222), indicating individual finetuning is essential; rankings from the pretrained supernet with individual finetuning correlate strongly with the train-from-scratch baseline (Spearman's ρ = 0.7287).

• Agent-based prediction wins. Among predictors built on traditional neural networks, standard LLMs, and autonomous agents, the agent-based system achieved the highest performance in inferring optimal designs for unseen tasks.

Methodology in Plain English

Defining the search space. A candidate architecture (a "path") is a sequence of blocks. Each path is defined by three things: its depth (number of blocks), the type of each block (drawn from CNN, LSTM, Transformer, Mamba, Hyena), and the hidden dimension at each layer. The full space is the union of all valid depth/type/dimension combinations, which yields millions of candidates.

Cutting the space down. Because distinct topological families are unevenly distributed in a heterogeneous space, naive random sampling would overfit to redundant patterns. The authors use a Representative Sampling Strategy to extract a concise subset of 360 architectures, combining distance-based filtering on log-transformed dimensions, a monotonic width constraint with the maximum dimension fixed at the last layer (which biases toward wider, parameter-heavy blocks appearing in more valid paths), and K-Means clustering on one-hot encoded architecture vectors to select cluster centroids.

Building and training a supernet. Rather than training candidates separately, all unique blocks are merged into one weight-sharing supernet, where each candidate is a path through shared weights. Training is one-shot: in each forward pass a single path is sampled uniformly, and only its shared weights are updated. This keeps the process tractable and, per the authors, acts as regularization that prevents co-adaptation between blocks. The supernet is pretrained on unlabeled biological data using one of three self-supervised objectives: Masked Modeling, Contrastive Learning, or Next Token Prediction.

Evaluating and ranking. Each sampled path is optimized independently rather than finetuning the supernet as a whole, both because downstream datasets are small and because isolating weights removes interference between paths. Paths are ranked by a unified score: performance on each task is Z-score standardized (to prevent tasks with wider numerical ranges from dominating) and sign-flipped for error-based metrics such as RMSE, then averaged across tasks. The top-ranked architecture is then randomly reinitialized, pretrained from scratch, and finetuned per downstream task to serve as a foundation model.

Data. DNA pretraining uses the full human reference genome GRCh38 with downstream evaluation on the 12-dataset human subset of GUE. Protein pretraining uses a 10% randomly sampled UniRef50 subset containing 72.1M representative sequences clustered at 50% identity, with evaluation on 6 tasks from the PEER benchmark.

Why This Matters

Impact on research. The paper argues that architecture design for biology has lacked guiding principles because, unlike human language, biological sequences are "designed by nature" and their governing rules are not fully known. By turning architecture selection into a measurable, automated search, BioArc offers a reproducible alternative to intuition-driven design and provides empirical design principles — such as the Hyena–Transformer–CNN ordering and the finding that parameter count does not predict accuracy. It also opens a methodological question the paper answers directly: whether compact models can match large ones if the architecture is right, with the controlled protein comparison supporting a "yes, within a matched budget."

Real-world applications:

• Drug development and protein binder design — the PPI Affinity task is described as directly applicable to protein binder design, where binding strength between proteins must be estimated.
• Synthetic biology and protein engineering — solubility prediction is framed as critical for functional proteins in pharmaceutical research and industry, and fold classification is described as relevant for drug design and functional analysis.
• Personalized medicine and regulatory genomics — transcription factor binding site prediction, promoter detection, and splice site prediction (relevant to understanding genetic diseases) are the DNA tasks the models are evaluated on.
• Genome-scale analysis — the models target whole-genome context and long-range regulatory interactions such as enhancer-promoter contacts.

Industry relevance. The headline result — state-of-the-art performance at up to 25x smaller size — directly addresses the cost of pretraining and deploying biological foundation models. BioArc-F reportedly matches or exceeds baselines with 1/20 the model size and 1/10 the training steps, and the paper's performance-parameter analysis explicitly argues that gains come from topology rather than raw capacity. The agentic predictor is framed as bridging offline search and real-world deployment, so new tasks can be matched to architectures without repeating an expensive search.

Future Directions

• Extending beyond DNA and protein. The paper states the framework generalizes to other modalities such as RNA and single-cell data, but the reported experiments cover only DNA (GRCh38/GUE) and protein (UniRef50/PEER).
• Improving the agentic predictor's generalization. The authors note that architecture predictors are inherently limited by the scope of their training tasks; the agent-based system performed best in their benchmark, but transferability to genuinely novel domains remains the open question they raise.
• Co-optimizing architecture and tokenization. The finding that the best tokenizer flips between task-specific models (1-mer favored by CNNs) and the pretrained foundation model (BPE favored) suggests tokenization choices should be searched jointly with architecture rather than fixed in advance.
• Reconciling pretraining strategy with architecture. Because no single pretraining objective dominated and contrastive learning was unstable within limited budgets, the paper implies further work is needed on training schedules and on when pretraining helps at all — since training from scratch achieved the highest Win Rate on one task.

Target Audience

Machine learning researchers working on neural architecture search, efficient model design, or foundation models for biological sequences; computational biologists and bioinformaticians interested in architecture-level explanations of regulatory grammar; and practitioners in drug discovery, protein engineering, or genomics who need compact, task-specialized models without large-scale pretraining budgets. Readers without a background in self-supervised learning or NAS terminology will find the framing accessible, though the technical detail (supernet weight sharing, Z-score ranking, Spearman correlation analysis) assumes intermediate familiarity.

Authors’ abstract

Foundation models have revolutionized various fields such as natural language processing (NLP) and computer vision (CV). While efforts have been made to transfer the success of the foundation models in general AI domains to biology, existing works focus on directly adopting the existing foundation model architectures from general machine learning domains without a systematic design considering the unique physicochemical and structural properties of each biological data modality. This leads to suboptimal performance, as these repurposed architectures struggle to capture the long-range dependencies, sparse information, and complex underlying ``grammars'' inherent to biological data. To address this gap, we introduce BioArc, a novel framework designed to move beyond intuition-driven architecture design towards principled, automated architecture discovery for biological foundation models. Leveraging Neural Architecture Search (NAS), BioArc systematically explores a vast architecture design space, evaluating architectures across multiple biological modalities while rigorously analyzing the interplay between architecture, tokenization, and training strategies. This large-scale analysis identifies novel, high-performance architectures, allowing us to distill a set of empirical design principles to guide future model development. Furthermore, to make the best of this set of discovered principled architectures, we propose and compare several architecture prediction methods that effectively and efficiently predict optimal architectures for new biological tasks. Overall, our work provides a foundational resource and a principled methodology to guide the creation of the next generation of task-specific and foundation models for biology.