Open-weight genome language model safeguards: Assessing robustness via adversarial fine-tuning

AI summary

Overview

• Research area: Biosecurity and AI safety for genomic language models (gLMs), specifically testing whether removing viral sequences from pretraining data — the standard "data exclusion" safeguard — holds up against post-release fine-tuning.
• Technical level: Intermediate. The paper assumes familiarity with language model fine-tuning, perplexity, and AUROC, but explains the biosecurity framing in accessible terms.
• Scope (1 sentence): The authors adversarially fine-tune the open-weight gLM Evo 2 (7B) on genomes from 110 harmful human-infecting viruses and measure how much misuse-relevant predictive capability is recovered, including on an unseen virus (SARS-CoV-2).

What This Paper Is About

Open-weight genomic language models such as Evo 2 are released publicly, and their developers try to limit misuse risk by excluding viral genomes from the training data — Evo 2's developers "excluded genomic sequences from viruses that infect eukaryotic hosts." The problem is that anyone can fine-tune a released open model, so the safeguard only works if that exclusion cannot be undone. This paper tests whether fine-tuning on publicly available pathogen sequences can restore the viral capabilities that data exclusion was meant to suppress.

Key Contributions

1. An adversarial fine-tuning test of data exclusion: The authors fine-tune Evo 2-7B on a curated set of 110 viral genomes from harmful human-infecting viruses and compare it against both the pretrained model and a control model fine-tuned on 181 bacteriophage (prokaryote-infecting virus) genomes.
2. Quantified rescue of predictive capability: They show perplexity on viral sequences drops substantially after fine-tuning (median 2.16 on training data and 3.55 on held-out data for the harmful-virus model, versus 3.84 and 3.83 for the pretrained model).
3. A generalisation result across pathogens: The harmful-virus fine-tuned model achieves an AUROC of 0.59 at identifying SARS-CoV-2 spike mutations that confer immune escape, despite never having seen any SARS-CoV-2 sequence at any stage of training or fine-tuning.
4. A proposed reframing of gLM safety work: The paper argues for gLM-specific safety frameworks and for "rule-out" evaluations designed to demonstrate that particular harms are absent, rather than red-teaming that rules harms in.

Main Findings

• Perplexity falls on in-distribution viral data: The FT-harmful model reached a median perplexity of 2.16 on the training sequences, compared with 3.84 for pretrained Evo 2 and 3.73 for the bacteriophage-fine-tuned model.
• The gain partially transfers to unseen viruses: On the 12 held-out viral genomes, medians were 3.55 (FT-harmful), 3.83 (pretrained), and 3.73 (FT-bacteriophages). The paper reports that the difference between the bacteriophage and harmful-virus fine-tuned models on test data was not statistically significant.
• Sequence length did not confound the results: In the scatterplot of perplexity against sequence length for the 110 harmful human-infecting viruses, the relationship was negligible (n=97, r=0.034).
• Baselines fail at immune escape prediction: Neither the pretrained model nor the bacteriophage fine-tuned model performed better than a random classifier, and BLOSUM-62 evolutionary conservation scores gave an AUC of 0.51.
• Fine-tuning confers a modest but real signal on an unseen virus: The FT-harmful model scored an AUC of 0.59 at predicting SARS-CoV-2 immune escape mutations, using deep mutational scanning data as the label source.
• A purpose-built tool still wins: EVEscape, the specialised deep learning predictor treated as the gold standard, achieved an AUC of 0.8, well above the fine-tuned gLM.
• Data exclusion is circumventable but raises the bar: The authors conclude that the safeguard can be partially bypassed with available pathogen data, but that the recovered capability is inferior to a narrow, task-specific tool.
• Bacteriophage fine-tuning gave essentially no viral benefit: The control model showed no meaningful improvement on the viral tasks, supporting the interpretation that the FT-harmful gains come from the viral sequences themselves.

Methodology in Plain English

The team took Evo 2-7B, an open-source genomic language model originally released with 1 billion, 7 billion, and 40 billion parameter versions, and fine-tuned the 7B version further.

They built a fine-tuning dataset by searching the literature for harmful human-infecting viruses across six groups of concern (large DNA viruses, small DNA viruses, positive-strand RNA viruses, negative-strand RNA viruses, enteric viruses, and double-stranded viruses). They then removed redundant sequences using Mash sketching with 10,000 k-mers, clustering genomes with a Mash distance below 0.01 (more than 99 percent average nucleotide identity) and keeping the longest sequence from each cluster. That produced 122 viral genomes, split 90/10 into 110 sequences for fine-tuning and 12 held out for testing. A parallel control dataset of 181 bacteriophage genomes was built the same way.

Training used the Savanna framework on 4 NVIDIA H100 GPUs with DeepSpeed ZeRO Stage 1 and BF16 mixed precision. Because of computational limits, the model's original 1M-token context window was shortened to 4,096 tokens per sample. Training ran for 150 steps with the Adam optimizer, an effective batch size of 144 sequences, gradient clipping at 0.5, weight decay of 1x10e-4, dropout of 0.05, and a cosine learning rate schedule decaying from 4x10e-6 to 8x10e-8 with 10 percent warmup.

They then ran two evaluations. First, they measured perplexity — how well the model predicts the next token in a sequence, where lower is better — across the pretrained model, the bacteriophage-fine-tuned model, and the harmful-virus-fine-tuned model, on both training and held-out sequences. Second, they tested whether the models could flag which SARS-CoV-2 spike protein mutations cause immune escape, using publicly available deep mutational scanning data as ground truth, and compared the three Evo 2 variants against BLOSUM-62 conservation scores and the specialised predictor EVEscape. Notably, no SARS-CoV-2 genome was present at any point in training or fine-tuning for any of the models. Some methodological details and all code were withheld due to information-hazard concerns, with availability by reasonable request from the authors.

Why This Matters

This work tests a widely assumed but under-verified assumption: that stripping sensitive sequences from training data makes an open-weight model safe to release. The authors show this assumption is only partially true, which shifts the burden onto evaluation and mitigation design rather than dataset curation alone. It also supplies a concrete demonstration that a model can pick up generalisable functional signals from a limited set of other viral genomes — 110 genomes — without having seen the target pathogen.

Real-world applications:

• Biosecurity policy for open-weight model releases: Informs how developers and regulators should treat data exclusion when releasing biological foundation models.
• Vaccine and antigen design: The authors note that limiting viral capabilities may also limit beneficial public health work such as rapid antigen design during pandemics and epidemics.
• Pre-deployment evaluation design: Supports building reproducible, cross-model "rule-out" evaluations and a taxonomy of misuse-enabling capabilities.
• Gene therapy and dual-use research: The paper points out that generating functional viral genomes is dual-use and can be useful for gene therapy research, complicating capability-limitation approaches.

Industry relevance: The findings are directly relevant to developers of open-weight biological models, who must decide what safeguards to attach to released weights; to biosecurity and AI governance bodies setting standards; and to the broader AI safety community, since the authors argue that mitigation techniques developed for LLMs (refusal training, constitutional classifiers) have not been systematically adapted to gLMs. The paper also flags the "daisy-chain" problem, where outputs from one tool become inputs to another in integrated systems, meaning per-model risk assessment is insufficient.

Future Directions

• Evaluate generative capabilities, not just predictive ones. The authors state that ideally an evaluation of sequence generation properties would have been conducted and that future work should address this.
• Close the compute gap. Because computational constraints forced the context window down from 1M to 4,096 tokens, the results may underestimate what a fine-tuner starting from the full-context model could achieve; the authors note many malicious actors might face similar constraints, but this remains unresolved.
• Build a gLM-specific safety toolkit. The paper calls for further work on a taxonomy of misuse-enabling capabilities and a corresponding set of implementable risk evaluation and mitigation measures, including detection, intervention, and access controls, and for exploration of mitigations suited to open-source models.
• Anticipate compounding and agentic risk. The authors highlight multi-modal models and agentic systems, plus autonomous model-development feedback loops with limited human checkpoints, as sources of unanticipated risk that current evaluations do not capture.

Target Audience

Biosecurity researchers and policy analysts; AI safety and evaluations researchers working on biological models; developers and release decision-makers at organisations shipping open-weight genomic or protein language models; computational biologists interested in what fine-tuning can recover from filtered data; and regulators or institutional reviewers designing safety frameworks for biological AI.

Authors’ abstract

Novel deep learning architectures are increasingly being applied to biological data, including genetic sequences. These models, referred to as genomic language models (gLMs), have demonstrated impressive predictive and generative capabilities, raising concerns that such models may also enable misuse, for instance via the generation of genomes for human-infecting viruses. These concerns have catalyzed calls for risk mitigation measures. The de facto mitigation of choice is filtering of pretraining data (i.e., removing viral genomic sequences from training datasets) in order to limit gLM performance on virus-related tasks. However, it is not currently known how robust this approach is for securing open-source models that can be fine-tuned using sensitive pathogen data. Here, we evaluate a state-of-the-art gLM, Evo 2, and perform fine-tuning using sequences from 110 harmful human-infecting viruses to assess the rescue of misuse-relevant predictive capabilities. The fine-tuned model exhibited reduced perplexity on unseen viral sequences relative to 1) the pretrained model and 2) a version fine-tuned on bacteriophage sequences. The model fine-tuned on human-infecting viruses also identified immune escape variants from SARS-CoV-2 (achieving an AUROC of 0.6), despite having no exposure to SARS-CoV-2 sequences during fine-tuning. This work demonstrates that data exclusion might be circumvented by fine-tuning approaches that can, to some degree, rescue misuse-relevant capabilities of gLMs. We highlight the need for safety frameworks for gLMs and outline further work needed on evaluations and mitigation measures to enable the safe deployment of gLMs.