AI Models Can Provably Hide Arbitrary Capabilities

AI summary

Overview

Research area: Machine learning security and safety evaluation, specifically cryptographic backdoors in neural network weights, connecting to capability elicitation, red-teaming, and CBRN risk assessment.

Technical level: Advanced. The paper assumes familiarity with complexity-theoretic security definitions (negligible functions, min-entropy, probabilistic polynomial-time adversaries), the random oracle model, and transformer internals such as SwiGLU blocks and the residual stream.

Scope: The paper constructs and empirically tests a backdoor that hides an entire executable neural network inside the weights of a host model under cryptographic assumptions, so that the hidden capability cannot be elicited or identified by a white-box auditor without the secret trigger.

What This Paper Is About

Capability evaluations — prompting, fine-tuning, red-teaming — are used to decide whether a model is safe to deploy, but they only work if hidden capabilities can actually be surfaced. The authors show that this assumption fails under adversarial conditions: an attacker who can modify model weights before deployment (a malicious insider or a compromised supply chain) can embed a capability encrypted under a secret trigger, so the model behaves benignly in ordinary evaluations and only executes the hidden computation when it sees the trigger. The goal is to establish a provable limit on elicitation-based capability audits, not to build a practical attack.

Key Contributions

1. A new class of attacks with cryptographic guarantees. The authors describe embedding arbitrary computations inside model weights that are provably infeasible to elicit under stated cryptographic assumptions, extending prior work from hiding fixed strings to hiding executable neural networks.
2. A PyTorch compilation framework and released model organisms. The framework embeds arbitrary algorithms as differentiable circuits in modern architectures (SwiGLU MLPs with RMSNorm), and the authors released their models to support defense research.
3. Empirical evaluation against four elicitation methods. They test supervised fine-tuning, weight noising, gradient-based trigger search (GCG), and latent adversarial training on a chemical reaction prediction task, and report resistance in each case.
4. A method for distinguishing elicitation from overelicitation. Using subset parity — a task described as provably hard for gradient-based algorithms in certain approximate-gradient settings — they build circuits that are theoretically hard to learn, so that any apparent success from training can be attributed to learning rather than surfacing.

Main Findings

• Supervised fine-tuning fails to surface the hidden capability. On the chemistry task, 500 steps of SFT leaves the encrypted model at 50.9 ± 1.0 percent accuracy, compared with 50.6 percent for a random-network control locker and 50 percent for a random baseline. The password-locked baseline reaches 92.8 ± 0.9 percent under the same procedure.
• Latent adversarial training fails as well. At eight layers, the encrypted model reaches 52.9 ± 3.9 percent, versus 53.6 percent for the random-network control and 91.7 ± 0.7 percent for the password-locked baseline.
• Gradient-based trigger search only succeeds where brute force already would. With 8-bit triggers, GCG reaches 73.5 ± 2.6 percent on the encrypted model — the same as its correct-trigger accuracy — but the authors attribute this to exhaustive enumeration of all 2^8 candidates, which matches query-matched random search. With 16-bit triggers, GCG achieves 52.7 ± 1.4 percent.
• The hidden capability still executes correctly with the trigger. The encrypted model's correct-trigger accuracy is 73.5 ± 2.6 percent, lower than the password-locked model's 92.7 ± 0.2 percent because the two constructions contain different classifiers due to size constraints. On wrong-trigger inputs, both models are at approximately chance (50.6 ± 2.1 and 50.7 ± 0.1 percent).
• Weight noising does not change the picture. Noise scales of 0.01 and 0.1 applied before fine-tuning did not materially affect the observed elicitation results.
• Subset parity separates elicitation from learning. With the correct trigger, both the compiled model and the password-locked model compute the hidden parity exactly. Under SFT without the trigger, the password-locked model reaches 100 percent accuracy while the compiled model remains at chance. Noise injection elicits the hidden parity from neither model.
• The construction works on a full-strength security setting. With a 24-round Keccak-f[1600] locker, a 128-bit trigger, and the same chemistry classifier, the model has 110 layers and 3.58 × 10^9 dense parameters. It matches the classifier on all held-out inputs given the trigger and runs a stand-in LLM circuit on all 10,000 wrong triggers tested.
• The overhead is large. The proof-of-concept has 38 layers, approximately 2.07 × 10^9 dense parameters, 1.90 × 10^6 nonzero parameters, and a maximum residual-stream width of 10,266. The hash subcircuits contribute about 1.98 × 10^9 dense parameters, roughly 95 percent of the total. The password-locked baseline has approximately 1.7 × 10^5 parameters. Each additional hidden-network weight adds roughly 3.1 × 10^6 dense host-model parameters.
• Detection of the circuit itself is not prevented. The authors state that the presence of a compiled circuit may be detectable; the guarantee covers hiding the trigger and the capability, not making the model statistically indistinguishable from an unmodified one. The guarantee also does not prevent an auditor from independently learning the same task from external data.

Methodology in Plain English

The attack works in four steps. First, the attacker trains or compiles the capability they want to hide — here, a small ternary MLP that predicts whether a set of chemical reagents produces a candidate product. Second, each weight matrix of that capability is locked inside a digital locker, a cryptographic primitive that stores a message encrypted under a key and returns the message only when given that key. Third, the locker program is compiled into the weights of an otherwise benign host model. Fourth, on every forward pass the host attempts decryption: an ordinary input produces the locker's failure symbol, but the secret trigger recovers the capability's weights, which a GLU-based execution unit then multiplies with the input activations to run the hidden computation.

The authors convert algorithms into networks through a fixed chain: a straight-line program, then a linear threshold circuit, then matrices, then gated linear units. Because line normalization is present in modern architectures, the compilation handles scale invariance by approximating ReLU units with SwiGLU via scaling up before the activation and down afterward, and by simulating step activations through two shifted ReLUs. A wider residual stream is the cost of this approach: the single-position variant needs O(n^2) residual features for an approximately width-n hidden layer, while a row-wise autoregressive variant reduces this to O(n) residual features at the cost of O(nd) scratchpad positions for a depth-d network. The authors primarily use the O(n^2) variant.

For evaluation, they build models in PyTorch 2.9 on an A100 GPU with 80 GiB of VRAM, report 95 percent confidence intervals over 3 runs, and test with SwiGLU MLPs and RMSNorm. The chemistry task encodes reagent-product pairs as 64-bit Morgan fingerprints from USPTO data with molecules represented as SELFIES strings, yielding a balanced binary classification problem on which models attain above 70 percent accuracy. Triggered inputs take the form [BOS, T, z] with an 8-bit trigger. They compare against a password-locked model trained on the same task and against a control whose locker contains a random network rather than a trained classifier.

Why This Matters

The paper reframes a core assumption of AI safety practice: if a capability cannot be elicited, that does not mean it is absent. Under adversarial conditions where weights can be modified before deployment, the authors show that elicitation-based audits face a provable ceiling. This shifts part of the problem from average-case empirical testing toward worst-case security analysis, and supports the authors' argument that safety and security require different standards of evidence. The authors do state that the results concern deliberate sabotage and that they do not claim compiled cryptographic structures arise through ordinary training.

Real-world applications and implications include:

• Frontier model deployment decisions: Developers who rely on capability evaluations to make deployment decisions need to know that a negative result may not be informative when the weight supply chain is not trusted end to end.
• CBRN risk assessment: Chemical synthesis prediction is used here as a proxy for CBRN-relevant capability, and the authors note that models predicting reaction feasibility could lower barriers to synthesizing controlled substances or precursors.
• Model supply chain security: The threat model applies wherever models are deployed in high-stakes settings without end-to-end training provenance, including military, diplomatic, and critical infrastructure contexts.
• Red-teaming and defense research: The released model organisms and compilation framework give auditors concrete artifacts for testing new elicitation and detection methods.
• Weight-level detection research: Since the presence of a compiled circuit may itself be detectable, anomaly detection on weights is a plausible complementary defense even when the trigger and payload remain hidden.

Industry relevance centres on model providers and third parties who audit model weights, plus organizations that acquire models through a supply chain they do not fully control.

Future Directions

• Integrating the full construction into a pretrained model. The authors verified that individual GLU units compute correctly when injected into Gemma 3 270M, but state that integrating the complete construction into a pretrained model remains future work.
• Reducing overhead. The proof-of-concept is described as not a turnkey attack because of substantial parameter overhead. The authors implemented two alternative constructions — per-block hashing, which reduces depth at higher parameter cost, and row-wise execution, which reduces parameter count at the cost of longer context — but scaling to practical settings remains open.
• Detecting the circuit without the trigger. Because the guarantee covers the trigger and capability rather than the anomalous presence of the circuit, developing weight-level methods that flag compiled cryptographic structures is a natural defensive direction.
• Demonstrating the augmented-host scenario. The authors describe as prospective, and explicitly not yet demonstrated, the concern that a small hidden circuit could augment an already capable host to support cyber abuse, CBRN-relevant assistance, fraud, or sabotage while remaining inactive during ordinary evaluation.

Target Audience

This paper is most useful to AI safety and security researchers working on capability elicitation, backdoor attacks and defenses, and red-teaming methodology; to interpretability researchers interested in circuits compiled directly into weights rather than learned; to cryptographers applying primitives such as digital lockers to machine learning; and to policy and governance audiences who need to understand the limits of evaluation-based safety cases for models acquired through an untrusted supply chain.

Authors’ abstract

Capability evaluations aim to surface dangerous model behaviors before deployment, but their reliability depends on the assumption that hidden capabilities can be elicited. We show this assumption does not hold under adversarial conditions by constructing backdoor attacks that embed encrypted neural networks within host model weights, decrypting and executing them only upon receiving a secret trigger. Using digital lockers, these hidden circuits remain provably hard to elicit and interpret under cryptographic assumptions, even given full access to model weights, extending prior work from hiding fixed strings to arbitrary computations. We implement our proof of concept constructions in PyTorch and evaluate elicitation through fine-tuning, weight noising, gradient-based trigger search, and latent adversarial training on a small chemical reaction prediction task - a proxy for CBRN-relevant capabilities. Our results demonstrate resistance to these methods and, together with our cryptographic guarantees, reveal a limit of elicitation-based capability audits. We release our models to support the improvement of defenses.