Watermarks Without Verification: AI Text Watermarking After the EU AI Act

AI summary

Overview

Research area: AI governance and trustworthy machine learning — specifically statistical text watermarking, its measurement, and the institutional conditions under which vendor claims about it can be verified. The paper sits at the intersection of computer security, NLP evaluation, and technology regulation under the EU AI Act.

Technical level: Intermediate. Readers need a working sense of how large language models sample tokens and what a statistical detector does, but the paper is written for a policy-adjacent audience and defines its terms.

One-sentence scope: The paper argues that the real governance failure in the August 2026 watermarking controversy is not watermarking itself but the impossibility of verifying either the public's objections or the vendors' assurances, and it maps each unverifiable claim to a concrete institutional remedy.

What This Paper Is About

When Article 50 of the EU AI Act took effect on August 2, 2026, providers of generative AI systems became legally obligated to mark their output in a machine-readable way and to ensure it can be detected as AI-generated. Anthropic then disclosed that every Claude model released after that date embeds a SynthID-Text-based watermark in all generated text, on by default, with no user opt-out. The result was a public backlash built on mutually contradictory claims — that the watermark ruins code, secretly encodes identifying information, is trivial to strip, and is impossible to escape — met by vendor assurances that nobody outside the company could check. The paper's goal is to sort these contested assertions by what it would actually take to settle each one, to measure what can be measured on open models, and to show that the missing verification infrastructure, not the watermarking technique, is the substantive failure.

Key Contributions

1. A verification taxonomy of the debate. The authors reconstruct twelve contested assertions — four user claims, five vendor assurances, one vendor-acknowledged limitation, and two concerns raised by another provider — and classify each into one of three categories: (a) publicly testable with existing tools and open-weight models, (b) testable only if the vendor grants access, or (c) unverifiable until standards bodies and independent auditors exist.

2. A family-level empirical evaluation of SynthID-Text. Because no public tool can test the deployed Claude or Gemini systems, the authors run the open-source SynthID-Text implementation through the MarkLLM toolkit on two open-weight models (Gemma-2-9B and Llama-3.1-8B), measuring prose quality, code correctness, and detectability side by side under pre-registered equivalence bounds.

3. The argument that unverifiability is the governance failure. The paper distinguishes two kinds of gap: access that vendors withhold at essentially no cost to themselves, and institutions — agreed protocols, accredited auditors, cross-provider coordination — that no single vendor can supply.

4. A mapping from unverifiable claim to institutional requirement. Each category (b) and (c) assertion is paired with a specific remedy: release of matched watermark-on/watermark-off outputs, disclosure of the deployed configuration, accredited audits, a shared evaluation protocol, and interoperable detection.

Main Findings

• The two most prominent user claims cannot both be true. Claim 3 (the watermark is easily removable and therefore useless) and Claim 4 (it is effectively non-removable and therefore inescapable) appear side by side in the same discussion threads. Published results on edit robustness refute both: paraphrasing removes the mark, while light token-level edits and mixing watermarked spans into human text leave it detectable.

• On prose, the watermark's measured effect does not exceed changing the sampling seed. Across 500 open-ended prompts, the mean perplexity change was −0.3% on both models, well inside the pre-registered 5% equivalence band. The blinded judge returned win rates of 0.501 (Gemma) and 0.515 (Llama). On Llama, the seed control drifted +1.5% — farther than the watermark itself.

• On code, the quality cost is at most three points on one model and unmeasurable on the other. Gemma showed a +0.4 point difference in pass@1 (61.8% watermarked vs. 61.4% unwatermarked). Llama showed −3.1 points (60.7% vs. 63.8%), with a confidence interval of [−6.0, −0.3] that extends just past the five-point bound. The loudest objection of the August debate — that watermarking unavoidably degrades generated code — does not hold at the family level.

• Detection on code stays near chance, which reframes the code objection. At a 1% false-positive rate, prose detection reached true-positive rates of 39% at 200 tokens and 56–59% at 400 tokens, with AUROC of 0.78 and 0.82. Code detection stayed at AUROC 0.55 and 0.57 at every length. Code is disadvantaged twice: low per-step entropy leaves the key little to decide, and code passages are short — fewer than one sample in a hundred reached 200 tokens on Gemma. Between 25% and 31% of code samples were character-for-character identical with and without the watermark. The cost of low-entropy content falls not on the user's quality but on the detectability the provider legally owes.

• Assurance 3 is the one assertion testable on the deployment itself, and it holds. Hidden characters can be found by character-level inspection and token counts read from the public interface. Independent community inspection to date is consistent with Anthropic's claim of no hidden characters and no extra tokens. The authors present this as evidence that their taxonomy functions as an instrument rather than an indictment.

• Anthropic's assurances rest on evidence about the method, not the product. The claim that the watermark survives light editing but not a full rewrite matches family-level literature. Whether Anthropic's deployed thresholds, strength settings, and entropy cutoffs match the default configuration the authors tested is not externally observable.

• The most serious objection admits no external test. Users assert the watermark covertly encodes identifying information; the vendor denies it. The published scheme is zero-bit and carries no payload, but multi-bit schemes exist, and if a deployment assigned a distinct key per customer or organization, the operator could identify a text's source without adding anything to the text at all. No outside experiment can distinguish these configurations. Both sides are stuck in symmetric, unfalsifiable positions.

• Misattribution and false positives are the areas where evidence is thinnest. The mark records model involvement rather than authorship, so human-written text lightly edited by a model carries it. Detection thresholds are calibrated under the assumption that text carries no watermark at all; published error rates on mixed model-edited human text are sparse. Concern about disproportionate burden on non-native English speakers rests on findings about post-hoc classifiers, which score style rather than a key, so the finding does not transfer directly — but uneven detectability across languages is documented.

Methodology in Plain English

The authors could not test the deployed systems, so they tested the technique. They took the published, open-source SynthID-Text implementation and wired it into an open toolkit, then ran it on two openly downloadable instruction-tuned models. For each question or coding problem, they generated three outputs: one watermarked, one unwatermarked from the same random seed, and one unwatermarked from a different seed. The first pair isolates the watermark. The second pair is a control that measures how far two ordinary generations from the same model drift apart anyway — without that control, ordinary variation could be mistaken for watermark damage.

They used 500 open-ended questions and 364 programming problems from two standard code benchmarks, with ten samples per coding problem. Before running anything, they fixed the bounds that would count as "equivalent": perplexity within 5%, blinded pairwise win rate between 0.45 and 0.55, and code correctness within five percentage points. Prose quality was scored by a model from a third family, to avoid a model judging its own output. Preference was judged blindly, each pair twice with the order swapped. They also measured semantic similarity between responses using a sentence-embedding model.

For detection, they used the mean g-value detector from the published scheme and set the threshold at a 1% false-positive rate, re-estimating that threshold at each passage length so short and long passages would be compared fairly. To organize the broader debate, they reconstructed the strongest recurring forms of each public claim and each vendor assurance, then asked of each: what experiment or institution would settle this, and does it exist today?

Why This Matters

Impact on research. The paper separates evidence about a watermark family from evidence about a watermark deployment, a distinction it argues the public debate systematically collapsed in both directions. It also reframes the code-quality objection from a user-cost problem to a provider-detectability problem, which suggests evaluation protocols for low-entropy content — code, structured output, short factual answers — should measure detection strength rather than fluency alone. The finding that detection on code sits near chance under the exact configuration vendors report using is a concrete, reproducible result that the field can contest.

Real-world applications:

• Regulatory compliance under the EU AI Act. The paper gives providers and regulators a checklist of what Article 50 compliance would require in substance, not just in claim, and identifies where a transparency mechanism may be silently inactive — low-entropy outputs — while still nominally satisfying the obligation.

• Content provenance on platforms. Anyone building systems to label AI-generated material needs to know that detection on code and constrained formats is near chance, and that a text watermark can drift out of step with signed provenance metadata once a file is edited.

• Dispute resolution and appeals. A person whose human-written, model-edited text is flagged currently has no route to challenge the finding except through the tooling of the party that produced the flag. The paper motivates a scored detector and public calibration reporting as necessary to make such challenges meaningful.

• Procurement and model selection. Organizations choosing between vendors can distinguish assurances backed by releasable evidence from assurances that cannot be checked, and can ask for matched outputs and configuration disclosure as contract terms.

Industry relevance. Roughly 190 organizations signed the European Commission's voluntary Code of Practice on Transparency of AI-Generated Content, which recognizes watermarking as an acceptable marking technique and commits signatories to providing detection systems. Google has deployed SynthID-Text since 2024; Anthropic followed in 2026. The paper's point is that the Code specifies none of the infrastructure — shared protocols, audit access, calibrated scoring, cross-provider interoperability — that would make any of those commitments verifiable. It also notes the competitive asymmetry this creates: vendors that release matched outputs expose themselves to scrutiny that vendors who do not are spared, so verification cannot be voluntary.

Future Directions

• Establish a shared evaluation protocol. Existing benchmarks and toolkits differ in metrics, passage lengths, attack suites, and reporting thresholds, so results for different schemes are not comparable with each other or with vendor assurances. The paper calls for agreed operating points and a common attack suite as a precondition for meaningful comparison.

• Build institutions capable of accreditation. No body currently exists to conduct or accredit a text-watermark audit, and key management — how many keys exist, whether they are per-customer, how they rotate, who holds them — is precisely the kind of question that has established audit practice in other security-critical settings. Importing those practices is an open project.

• Solve cross-provider detection. Text that has passed through more than one provider's model currently has no shared checking mechanism, and the signed provenance metadata attached to a generated file can contradict the text watermark after editing. Interoperable detection across roughly 190 signatories remains unaddressed.

• Measure what cannot currently be measured. The misattribution rate on human text edited by a model, false-positive rates at population scale, and detection quality disaggregated by language and user group are all missing for every deployment. The authors argue these are cheap to produce relative to their governance value.

• Resolve the traceability question. Whether deployments use shared or per-customer keys is unanswerable from outside and determines whether the watermark is a provenance signal or a tracking mechanism. Some form of independent infrastructure audit is the only path the paper sees.

Target Audience

This paper is most useful to people who must act on watermarking claims rather than merely hold opinions about them: AI policy and regulatory staff implementing or enforcing Article 50; trust-and-safety and compliance teams at model providers who need to know which of their assurances are currently unfalsifiable; independent auditors and standards bodies deciding what to require; platform engineers building content-provenance or detection pipelines; and NLP researchers interested in a careful, pre-registered measurement of a deployed-at-scale watermark scheme that separates family-level evidence from deployment-level claims. It also serves journalists and technically literate readers who want a structured account of a controversy that circulated largely as rumor, since the paper's taxonomy explicitly sorts the loudest public claims into those that survive measurement and those that contradict each other.

Authors’ abstract

On August 2, 2026, the obligations of Article 50 of the EU AI Act took effect, requiring generative AI providers to mark the content their systems produce and ensure it can be detected as AI-generated. Days later, Anthropic disclosed that every Claude model released after that date embeds a watermark based on SynthID-Text in all generated text, enabled by default with no user opt-out; Google has deployed SynthID-Text in Gemini since 2024. Users objected that the watermark degrades quality, particularly for code, that it secretly encodes identifying information, and, in mutual contradiction, that it is easily removable and inescapable; the vendor answered with assurances of unchanged quality, no identifying information, and robustness to light editing. In this work, we argue that neither the objections nor the assurances can currently be verified and that this unverifiability, rather than watermarking itself, is the substantive governance failure. We sort the contested assertions by what it would take to settle each and evaluate the open-source SynthID-Text implementation on two open-weight models, because no public tool can test the deployed systems. On prose, the measured effect of the watermark does not exceed that of changing the sampling seed. On code, the cost is three points of correctness on one model and below measurement on the other, while detection remains near chance, a limitation of detectability rather than quality. The remaining gaps trace to withheld access or missing institutions and we map each to a requirement: release of matched outputs, configuration disclosure, accredited audits, a shared evaluation protocol, and interoperable detection.