Toward Alignment Scaling Laws: A Framework and First Preregistered Measurements

AI summary

Overview

Research area: AI safety and scaling laws — specifically whether the effort required to keep a model safe grows faster or slower than model capability.

Technical level: Advanced. The paper combines a conceptual framework, four proved propositions about a toy model, and two preregistered empirical measurements; the accompanying narrative is written in plain language, but the propositions and exponent arithmetic require comfort with power laws and log–log reasoning.

One-sentence scope: The paper proposes measuring alignment as a family of per-risk power laws B_r(N) = a_r N^{α_r}, proves consequences of that model, and reports two preregistered measurements (a reanalysis of public adversarial-training data for Pythia classifiers and a pilot on Qwen2.5 models from 0.5B to 72B).

What This Paper Is About

Debates about whether alignment gets easier or harder as models grow usually rest on isolated findings that point in different directions, treating "alignment" as a single property. This paper argues instead that alignment should be measured separately for each risk category (jailbreaks, sycophancy, reward hacking, deception, and so on), each with its own scaling exponent describing how much alignment effort is needed to hold a fixed safety target as capability grows. The goal is to turn "in which world are we?" into a precise, pre-registrable measurement question with defined estimators, confounds and decision criteria.

Key Contributions

1. A per-risk framework (Section 3). Alignment burden B_r(N) = a_r N^{α_r} is defined for each risk r against a capability proxy N, with three operationalizations: (O1) failures found by a fixed red-teaming procedure, (O2) safety-training data or compute needed to hit a target, and (O3) capability cost, i.e. the alignment tax measured at a fixed safety level. The framework distinguishes observed, audited and latent true alignment, and defines capability headroom H = 1 − ℬ/N.

2. A toy model with four propositions (Section 4, proofs in Appendix A). A deterministic accounting model with four risk categories, simulated exactly by the paper's browser demonstrator, yields results on headroom regimes, required growth rates, estimation bias from heterogeneous exponents, and audits as upper bounds.

3. A pre-registrable measurement protocol (Section 5) with estimators, a precision analysis, confounds and decision criteria — described by the paper as designed to fill the gap that no paper directly measures B_r(N).

4. Two preregistered applications. A reanalysis of public adversarial-training data for Pythia classifiers, and a pilot on Qwen2.5 models from 0.5B to 72B parameters (one recipe at every size) replicated on Qwen3 Base from 0.6B to 14B, plus an interactive demonstrator of four browser games.

Main Findings

• Regime is set by the fastest corrected risk, not the average. Corollary 1 shows that with B(N) = Σ a_r N^{α_r} over corrected risks, if the largest such exponent α^V_max < 1 headroom tends to 1; if it equals 1 headroom tends to 1 − Σ a_r/g; if it exceeds 1 headroom is exhausted in finite time. Unseen categories cost alignment but never headroom.

• The toy's "balance" world is asymptotically a debt world. In the toy model α^V_max = α + 0.15, set by reward hacking, giving thresholds 0.85, 1.15 and 1.65 in the three worlds. Under exact doubling from the post-wave state, the balance world (base exponent α = 1) peaks at 61.1% headroom at cycle 6 and cannot pay the correction of cycle 22, whereas a single-exponent model with α = 1 would stay at 53.125%.

• Holding a headroom floor requires super-exponential growth above 1. Proposition 2 shows that once α > 1, the tight policy's growth factor increases without bound, growth is doubly exponential, and the number of cycles to reach size X is at most of order log log X; every floor-holding policy grows at least doubly exponentially. In continuous time calibrated on the demonstrator, the blow-up occurs at t* = 3.2 cycles for α = 1.5.

• Small-model fits underestimate large-scale exponents. Proposition 3 shows dα_eff/d ln N = Var_w(α) ≥ 0, so ln F is convex in ln N and α_eff rises from the smallest to the largest exponent. In the toy model, α_eff − α rises from −0.100 at N_0 to +0.30 for all failures and from −0.144 to +0.15 for visible ones; the OLS fit over the swarm phase (sizes 384 to 3184) returns α − 0.09 (0.611, 0.913 and 1.411 in the three worlds), short of the asymptotic α + 0.30 by 0.39 — 0.15 from invisibility and 0.24 from curvature.

• Audits never underestimate true alignment (without false positives). Proposition 4 shows A^obs stays at 1 by construction while A^true falls; A^aud ≥ A^true surely, with the proxy gap equal to (1−p)CU/((C+pU)(C+U)); a known p allows a consistent correction but not a bound. In the toy model, true alignment crosses 50% at N ≈ 6.5 × 10^5.

• Preregistered reanalysis (Pythia classifiers). On public adversarial-training data, the compute needed to bring attack success under 10% grows as N^{0.60} (95% interval 0.42–0.78), more slowly than pretraining compute.

• Preregistered pilot (Qwen2.5, 0.5B–72B). Exponents of −0.05 for truthfulness and 0.48 for stated dispositions, both "scaling helps" under the pilot's reduced rule over the whole window, though local slopes rise toward 1 at the larger sizes, and the truthfulness result is strongly attenuated when measured as a target relative to the starting error; both classes replicate on Qwen3 Base from 0.6B to 14B (the disposition result resting on its smallest model).

• Sycophancy undetermined; backdoor mixed. Sycophancy has an exponent of 0.89, or 0.83 with two seeds added at 72B, and is reported as undetermined. The planted backdoor is removed within 128–256 examples when its trigger is known but survives blind safety fine-tuning at four of five sizes.

• Explicit non-claims. The paper makes no claim about which regime holds for current frontier models; none of the toy model's numbers is a forecast, and two headline toy results depend on illustrative choices (reward hacking given a positive shift turns the balance world into a debt world; giving the never-detected category the largest shift makes true alignment fall in every world).

Methodology in Plain English

The authors start by rejecting the premise that alignment is one thing. For each risk category they define a "burden" — how much work a stated alignment method needs to bring a model of capability N from a stated starting point to a fixed safety target — and hypothesize that this burden follows a power law in N. Whether the exponent is below, near, or above 1 then says whether scaling helps, keeps pace, or accumulates "alignment debt."

To explore the consequences without needing frontier-scale experiments, they build a small deterministic toy model in which four risk categories generate failures as power laws, correcting a detected failure costs a fixed amount of burden, burden accumulates and eats into a capability headroom budget, and growth restores headroom. Undetected failures cost no headroom but open a gap between observed and true alignment. Every number in this model is reproducible exactly by the accompanying browser demonstrator.

They then write a preregistrable protocol with estimators, precision analysis, confounds and decision criteria, and run reduced versions of it twice: once as a reanalysis of already-public adversarial-training data for Pythia classifiers, and once as a pilot that trains one recipe at every size on Qwen2.5 models from 0.5B to 72B parameters, using low-rank fine-tuning and two-choice evaluations, covering sycophancy, truthfulness, stated dispositions and a planted backdoor, with replication on Qwen3 Base from 0.6B to 14B. Note that the provided paper content is truncated at Section 4.5, so the full protocol details of Section 5 and the full measurement details of Sections 6–7 are not available in this excerpt.

Why This Matters

For research, the paper reframes an argument often conducted through isolated findings as a measurement program with explicit exponents, estimators and preregistration, and proves that the relevant long-run regime is set by the fastest-growing corrected risk rather than by any average. It also shows that fitting a single power law to small models is systematically biased downward when burdens are mixtures of power laws, which bears directly on how scaling extrapolations are read.

Real-world applications:

• Safety evaluation design: deciding how much red-teaming and safety-training effort to allocate per risk as models grow, and treating observed benchmark scores as upper bounds that audits can only tighten.
• Release and deployment policy: the result that holding a fixed headroom floor requires super-exponential growth once the relevant exponent exceeds 1 reframes what "we will just keep scaling and fix it as we go" assumes.
• Audit and assurance practice: the framework separates what an evaluation reports, what an audit reveals, and what is actually true, which is directly relevant to third-party safety auditing.
• Model selection by risk: because exponents are properties of a (risk, method, evaluation) triple, the framework implies different risks and different alignment recipes may sit in different regimes within the same model family.

Industry relevance: the alignment tax is already a practical concern in post-training pipelines, and this work suggests measuring it at a fixed safety level and reporting a per-risk exponent rather than a single number. The paper explicitly states that neither of its two applications identifies which regime frontier assistants are in.

Future Directions

• Extend the protocol to more risk categories, more alignment methods, and more than the two to five sizes or narrow size ranges that the paper notes dominate existing comparisons.
• Test whether the downward bias from heterogeneous exponents matters in practice, for instance by fitting one risk category in isolation versus the full set, and by checking the "invisibility plus curvature" decomposition on real measurements.
• Determine the currently undetermined cases — sycophancy in the Qwen2.5 pilot, and the planted backdoor, which survives blind safety training at four of five sizes — and establish whether local slopes near 1 at the largest sizes indicate a transition.
• Establish whether the general results' hypotheses (additive burden for Corollary 1 and Proposition 2, positive mixtures of fixed-exponent power laws for Proposition 3, audits without false positives for Proposition 4) are violated by real training systems, since the paper says real systems may violate them and that categories which saturate or vanish at scale could reverse the bias direction.

Target Audience

Alignment and safety researchers designing evaluations and post-training recipes; scaling-law researchers interested in extending power-law analysis beyond loss; policy and governance analysts who need a precise statement of what "alignment scales" would even mean; and technically comfortable readers interested in the browser games at www.aisafety.fun, which let players run the toy model or the measured exponents. The paper states plainly that it does not tell the reader which world frontier assistants are in.

Authors’ abstract

Whether alignment gets easier or harder as models grow is often argued from isolated findings, as if alignment were one property. We treat it as a family of measurable scaling relations: for each risk category r, the alignment burden needed to hold a fixed safety target is modeled as B_r(N)=a_rN^alpha_r, with N a capability proxy; against a budget proportional to N, scaling helps if alpha_r&lt;1, keeps pace if alpha_r~1, and accumulates alignment debt if alpha_r&gt;1. We give three operationalizations of burden and distinguish observed, audited and true alignment. A toy model, in which corrections consume capability headroom, makes the consequences explicit. We prove that the largest exponent among corrected risks, not an average, sets the long-run regime; that above 1 any policy holding headroom above a floor must grow super-exponentially; that, for burdens that are positive mixtures of power laws, fits on small models underestimate large-scale exponents; and that an audit that uncovers hidden failures without false positives never underestimates true alignment. We propose a pre-registrable protocol and apply reduced versions of it twice. A preregistered reanalysis of public adversarial-training data for Pythia classifiers finds that the compute needed to bring attack success under 10% grows as N^0.60. A preregistered pilot on Qwen2.5 0.5B-72B finds exponents of -0.05 for truthfulness and 0.48 for stated dispositions (both scaling helps under its reduced rule, though local slopes approach 1 at the top; replicated on Qwen3 0.6B-14B), while sycophancy (0.89, or 0.83 with two seeds added at 72B) and a planted backdoor are undetermined: the backdoor is removed quickly when its trigger is known but survives blind safety training at four of five sizes. We release four browser games that play these laws (www.aisafety.fun). We make no claim about which regime holds for current frontier models.