Ethics & Governance

The Alignment Tax: The Real Cost of Making AI Safe — and Who Pays It

The alignment tax is real and it is collected from the wrong people: over-refused specialists, hourly raters, small firms. Updated for the EU's 16-month high-risk deferral and Anthropic's RSP rewrite.

Gabriele Masetti · 2026-08-20

The Bill Nobody Itemizes

Every safety measure bolted onto a large language model shows up somewhere else as a loss. That trade has a name — the "alignment tax" — and it dates to the paper that made modern chatbot alignment possible. When OpenAI researchers described InstructGPT in 2022 (Ouyang et al., "Training language models to follow instructions with human feedback"), they noted that reinforcement learning from human feedback (RLHF) could quietly degrade performance on standard NLP benchmarks even as it made the model more obedient and less toxic — a side effect they called the alignment tax.

Their fix, mixing pretraining gradients back into the RL objective (PPO-ptx), reduced but did not eliminate the regression. That paper is the reason "alignment tax" is now industry shorthand rather than an activist's slogan: the tradeoff was documented by the people selling the alignment, not by critics of it.

Four years on, the tax has metastasized well beyond a benchmark score. It shows up as models that refuse to answer immunologists, as image generators that quietly rewrite history, as safety teams that quit rather than watch risk get outrun by product cycles, as compliance line items only large companies can absorb, and as an annotation labor market now worth billions. The question worth asking isn't whether an alignment tax exists — that's settled.

It's who actually pays it, and whether that incidence bears any relationship to who benefits from the safety it buys. It does not. The people paying the steepest price are rarely the people the safety measures protect, and the actors most capable of causing catastrophic harm are, for now, the ones most able to dodge the bill entirely.

The Over-Refusal Users Are Subsidizing Everyone Else's Sense of Security

The clearest, most measurable form of the tax is over-refusal: a model declining a request that was never dangerous in the first place. Researchers formalized this in 2024 with OR-Bench, an 80,000-prompt benchmark of superficially sensitive but genuinely benign requests spanning categories like violence, harassment, and deception. The paper's central finding was a strong statistical relationship (a Spearman rank correlation around 0.88) between how aggressively a model refuses genuinely toxic prompts and how often it refuses harmless ones that merely sound similar.

Model families differed enormously in where they sat on that curve — some generations of Claude, GPT, and Llama models rejected large swaths of harmless prompts, while later versions pulled back over-refusal without abandoning safety — but the pattern held across nearly every family tested: safety and over-refusal move together, and almost no model breaks the trade-off cleanly.

The people who absorb that cost are not abstractions. In June 2026, Anthropic shipped a stricter safety classifier with Fable 5, released on June 9, after a third party (Amazon, per reporting) demonstrated a bypass technique; the company said plainly it had "made this safety margin much larger than in any prior launch," accepting that "many more benign requests would be blocked" as the price of closing the exploit.

Users who hit the wall were not aspiring bioweapons designers. Trade press reported security researchers and immunologists getting flagged for ordinary technical vocabulary, and coders getting blocked mid-debugging session on routine tasks. Anthropic's own account is unusually candid: the company designed the false-positive rate on purpose, judging that frustrating legitimate biology and security researchers was an acceptable cost of closing a jailbreak a small number of bad actors might have used.

That is a real allocation decision, made deliberately, and it falls on exactly the population — domain experts doing legitimate work — with the least ability to opt out of the product. Fable 5 has since been succeeded at the frontier by Claude Fable 5.1, released on September 1, 2026; the design choice it illustrates has not been.

Over-refusal also has a bias problem baked into how it's tuned. Research on refusal behavior has found that models frequently over-protect prompts that merely reference certain racial or religious groups while under-protecting prompts targeting disability — the safety margin isn't applied evenly across the population it's meant to protect, which means the tax isn't just regressive in aggregate, it's unevenly regressive by topic.

When "Safe" Overcorrects Into Wrong

Over-refusal is the tax paid in declined requests. A second, subtler version is the tax paid in wrong answers produced by a model straining to satisfy a safety objective. Google's Gemini image generator became the reference case in February 2024: after the underlying model produced historically inaccurate images — Black Founding Fathers, a female pope, interracial couples in place of the couple a user actually asked for — reporting traced the cause to instructions injected behind the scenes to increase demographic diversity in generated images, a bias-mitigation measure that had not been scoped to historical or factual contexts.

Google paused image generation of people entirely and told the press an improved version would follow. The episode is instructive precisely because the safety intervention wasn't malfunctioning by the metric it was built for — it was almost certainly reducing one measurable harm (stereotyped, non-diverse outputs) while introducing a different, more visible one (historical falsification). Nobody had priced that trade before shipping it; the market did the pricing after the fact, in the form of a global news cycle and a paused product line.

A Tax That Bad Actors Can Simply Refuse to Pay

Here is the part of the alignment tax story that gets least attention and matters most: the tax is not durable. Safety fine-tuning on open-weight models has repeatedly been shown to be cheap to remove. A 2023 paper demonstrated that LoRA fine-tuning could efficiently undo the safety training baked into Llama 2-Chat 70B, and a companion project, BadLlama, showed the same result on the 13B model — using a small amount of compute and a modest budget to strip safety behavior that Meta had invested real resources to instill.

The trained model, before stripping, refused the overwhelming majority of harmful prompts it was tested against; after a few hours of fine-tuning on commodity hardware, that refusal behavior collapsed. The researchers' point was not that Meta did a bad job — it's that safety fine-tuning, as currently practiced, is a shallow behavioral veneer on top of a capability that remains fully intact underneath, and open weights make that veneer removable by anyone with modest technical skill and a few hundred dollars of GPU time.

That matters for who bears the tax, because the burden turns out to be asymmetric by design intent rather than by outcome. Legitimate users and enterprises operating through official APIs pay the over-refusal cost in full, every day, on every ambiguous prompt. Actors who specifically want the capabilities the safety training was meant to withhold can, for open-weight models, remove the restriction for a trivial cost. The tax is close to voluntary for the population it's aimed at and mandatory for the population it isn't.

The Line Item Hiding in Latency and Compute

Alignment costs also show up as pure infrastructure overhead, and here the tradeoffs are unusually well quantified because they hit production systems directly. Lightweight, purpose-built guardrail classifiers can screen prompts and outputs with roughly 50–150 milliseconds of added latency, cheap enough to run on nearly every request.

But research comparing guardrail architectures has found that safety filters built from a large general-purpose model doing chain-of-thought reasoning about whether content is harmful can add multiple seconds of latency — in one comparison, up to roughly 8.6 seconds for models like Claude 3.5 Sonnet and Mistral Large used as judges — and that no configuration tested simultaneously achieved strong safety coverage, low false-positive rates, and low latency at once.

Specialized guardrail models close to that goal exist, but the more thorough the safety review, the more compute and time it costs, and that cost is either passed to the user as latency or absorbed by the provider as margin. Either way, it's a permanent tax on every single inference call a safety-filtered system makes, for the life of the product.

The Labor Market Built to Collect the Tax

Behind the classifiers and refusal behaviors sits a large, mostly invisible workforce whose entire job is generating the human feedback that makes alignment possible, and that workforce is now a serious industry. Surge AI, a data-labeling company specializing in RLHF, preference ranking, and red-teaming for frontier labs including OpenAI, Google, Anthropic, and Microsoft, had reached an estimated $1.2–1.4 billion in annualized revenue by 2025 while remaining privately profitable, built on a contractor network it has described as roughly a million people, with specialized technical or medical annotators reportedly billed anywhere from tens of dollars an hour into the hundreds. No comparable 2026 figure has been published, so read it as the last confirmed number.

The scale of money at stake became visible in June 2025, when Meta paid $14.3 billion for a 49% stake in Scale AI, another major RLHF vendor, valuing the company above $29 billion — and Google, an existing Scale customer, reportedly paused work with the firm within hours, unwilling to keep routing proprietary training signal through a company now partly owned by a competitor.

Alignment isn't merely an R&D line item; it has become a labor market with its own capital structure and its own competitive-conflict politics, one paid in real dollars to a supply chain of raters whose conditions and pay are set by a handful of buyers.

Regulation Adds a Second Tax With a Different Incidence

The EU AI Act layers a formal compliance tax on top of the informal capability tax, and its distribution is arguably worse. Estimates of the Act's aggregate cost to the European economy vary widely and are contested — one widely cited industry estimate put cumulative costs around €31 billion by 2025, a figure other researchers have argued substantially overstates the real burden — but the more defensible, granular numbers still show a consistent pattern: large enterprises face roughly €8–15 million in first-year compliance costs covering conformity assessments, documentation, and risk-management systems, while small and mid-size companies face costs in the tens of thousands to roughly half a million euros.

Those are not proportional numbers. A few hundred thousand euros is a rounding error for a company doing hundreds of millions in revenue and an existential threshold for a ten-person startup. The Act's stated purpose is to constrain the highest-risk uses of the most powerful systems; its practical effect, if the estimates hold, is to entrench incumbents who can absorb an eight-figure compliance line item while pricing out the smaller, more experimental developers who might otherwise challenge them. The safety tax and the competitive-moat effect are, here, the same policy.

Then Brussels moved the due date. The Digital Omnibus on AI, agreed provisionally on May 6, 2026 and in force since July 27, 2026, deferred the Act's stand-alone high-risk obligations under Annex III from August 2, 2026 to December 2027, and the obligations for high-risk AI embedded in regulated products under Annex I to August 2028. Only the Article 50 transparency duties — telling people they are dealing with an AI system, labeling synthetic content — began on August 2, 2026 as originally scheduled.

A sixteen-month deferral does not change the incidence argument; it changes when the bill arrives and who is still standing to receive it. A large firm banks the extra year and keeps building the compliance function it was always going to need. A startup that raised against a 2026 deadline has to explain to investors why the moat it priced in moved to 2027 — and the firms with the deepest lobbying benches are the ones who moved it.

The Companies Paying the Tax Are Now Quietly Refusing to Pay It

If the alignment tax were being priced correctly, the labs most exposed to competitive pressure would be investing hardest in cheaper, more surgical alignment techniques — and there has been real progress there, including preference-optimization methods like DPO that approximate RLHF's benefits with less compute overhead. But the more visible response from the two labs that built their public identities on safety has been to renegotiate the bill rather than pay it down.

In May 2024, OpenAI dissolved its Superalignment team within roughly a year of forming it, after its co-leads, Ilya Sutskever and Jan Leike, both resigned; Leike said publicly the team had been "struggling for compute" and that he disagreed with leadership's priorities, believing far more resources should go to safety relative to shipping new capabilities.

OpenAI later removed the word "safely" from its stated mission in a 2024 regulatory filing, swapping a commitment to ensure AI "safely benefits humanity" for language that simply commits to AI benefiting humanity — a change that tracked its restructuring toward a for-profit structure built to raise the capital a frontier-model race requires.

Anthropic, whose brand differentiation from OpenAI has largely been a more cautious posture, rewrote its own Responsible Scaling Policy on February 24, 2026. Version 3.0 was a comprehensive rewrite that replaced the old capability-threshold triggers with Frontier Safety Roadmaps and Risk Reports, and trade coverage that week read it as the company dropping the flagship pledge it had been built on, with several outlets tying the timing to US government pressure.

The story did not end there, and the update is less flattering to a clean narrative than the headlines were. On April 2, 2026 Anthropic published v3.1, which states: "We now clarify that, even if not required by the RSP, we remain free to take measures such as pausing the development of our AI systems in any circumstances in which we deem them appropriate." Three more revisions followed — v3.2 on April 29, v3.3 on May 26, v3.4 on July 8 — mostly procedural.

What changed, then, is not that Anthropic can no longer pause. It is that pausing moved from a stated trigger to management discretion. A commitment that fires automatically at a measured threshold is a thing a regulator or a customer can hold you to; a freedom to act "in any circumstances in which we deem them appropriate" is not.

Both companies are describing the same fact from different angles: the alignment tax is a cost you can choose to stop paying whenever a competitor forces the question, and in a two-or-three-lab race for market position, someone eventually will.

The Verification Tax Nobody Wants to Slow Down For

There is one more cost worth naming, because it's the one most directly connected to the catastrophic risks the alignment tax is ultimately meant to prevent: the time and expertise it takes to actually measure whether a frontier model is dangerous before releasing it. METR, an independent nonprofit that runs autonomy evaluations on frontier models before and after release, measures how long a task — expressed in the amount of time it would take a skilled human to complete it — a model can perform autonomously with a defined success rate.

That capability has been improving on a consistently exponential trend, with the length of task models can reliably complete roughly doubling every seven months between 2019 and 2024, and accelerating to something closer to a four-month doubling time since 2023. Running these evaluations, interpreting them, and building policy around thresholds derived from them is itself a substantial cost in researcher time and compute — a cost that exists precisely because it's the only mechanism giving labs, regulators, and the public any independent signal about autonomous-capability risk ahead of deployment.

It is the one piece of the alignment tax that functions as insurance rather than friction, and it's also the piece most vulnerable to being cut when a lab decides, as both OpenAI and Anthropic effectively have, that unilateral caution is a competitive liability it can no longer afford.

The Incidence Is the Scandal

Put the pieces together and a clear pattern emerges: the alignment tax is real, it is substantial, and it is being collected from the wrong parties. Domain experts and ordinary users pay it in refused prompts and degraded answers. A large, low-visibility labor market of contractors paid by the hour generates the training signal that makes it possible, at a cost now measured in billions of dollars a year.

Small companies and downstream developers pay a second, regulatory version of the same tax at a rate that large incumbents barely notice. Meanwhile, the actors capable of the worst misuse — those willing to fine-tune away safety behavior on open-weight models, exactly the risk profile the whole system is meant to guard against — can strip the tax for a few hundred dollars of compute.

And the two companies whose market position depends most on being seen as careful have both, within the same two-year window, converted explicit safety commitments into discretionary ones once competitors made the explicit version costly to keep.

None of this argues for abandoning alignment work; the InstructGPT lineage, OR-Bench-style measurement, and METR-style evaluation are exactly the kind of infrastructure that makes it possible to see the tradeoff clearly instead of guessing at it. But an industry that wants to keep calling this a "tax" rather than a subsidy scheme owes the public a tax code, not a tip jar.

That means investing the savings from techniques like DPO and targeted fine-tuning into lowering the over-refusal rate for legitimate specialist use rather than treating false positives as an acceptable rounding error; it means regulatory compliance costs scaled to actual risk and actual company size rather than a flat bar that happens to favor whoever can already afford outside counsel; and it means treating a Responsible Scaling Policy or a mission-statement pledge as a commitment that survives contact with a competitor's product launch, not a clause that gets rewritten into discretion the moment it becomes inconvenient. Until that changes, "who pays the alignment tax" will keep having the same answer: everyone except the people setting the rate.