Anchored Decoding: Provably Reducing Copyright Risk for Any Language Model

AI summary

Overview

Research area: Natural language processing, language model safety, inference-time decoding, and copyright/compliance risk mitigation.

Technical level: Intermediate. The method rests on a formal constrained-optimization argument (KL divergence budgets, Lagrange multipliers, root-finding), but the core intuition and the practical procedure are describable without deep mathematics.

Scope: The paper introduces a training-free decoding method that bounds how far any "risky" language model's generation can drift from a permissively trained "safe" model, and validates it on six model pairs against long-form copyright-copying and utility metrics.

What This Paper Is About

Large language models memorize spans of their pre-training data and can regurgitate them verbatim at inference time. When those spans come from copyrighted or sensitive sources, this creates compliance and legal exposure for developers and consent/compensation problems for creators, and retraining or filtering frontier models to fix it is prohibitively expensive. This paper proposes a decoding-time fix: combine the next-token distributions of a higher-utility "risky" model and a permissively licensed "safe" model so that the output stays within a provable information budget of the safe model, preserving fluency and factuality while suppressing verbatim copying.

Key Contributions

1. Anchored Decoding, an inference-time fusion method that solves a closed-form per-step optimization to keep generation within a user-chosen divergence budget (the K-NAF criterion) of a safe model. Local per-step constraints are proven to compose into a sequence-level guarantee.

2. Two budget-allocation refinements: a prompt-dependent prefix debt that reduces the available budget when the input appears to have already primed memorization, and an adaptive banking rule that rolls unspent budget from low-risk steps forward to cover high-risk spikes later in generation.

3. Anchored Byte Decoding, a byte-level variant built on the ByteSampler framework that removes the shared-vocabulary requirement, allowing fusion between models with incompatible tokenizers.

4. TinyComma 1.8B, a new permissively trained safe model (169.5B tokens from the Common Pile) that deliberately adopts the Llama 3.1 tokenizer, making direct token-level fusion with Llama-family models possible.

Main Findings

• Pareto-optimal risk-utility trade-off: Across six safe/risky model pairs and both token- and byte-level decoding, Anchored Decoding traces the Pareto frontier, closing up to 75% of the measurable copying gap between the risky baseline and the safe reference.

• Strong utility under high protection: At the "high-protection" operating point (normalized copying reduction ≥ 75%), Anchored Decoding achieves the best utility of any method tested. For {TinyComma 1.8B, Llama 3.1 70B} it reaches 0.53 factuality / 4.02 fluency, versus TokenSwap at 0.44 / 3.77 and RCAD at 0.37 / 3.38. At the byte level for {Comma 7B, Llama 3.1 70B} it reaches 0.52 / 4.23 versus CP-Fuse at 0.23 / 3.75.

• Modest inference overhead: Pairing a large risky model with a much smaller safe model keeps the extra cost low. The token-level pair {TinyComma 1.8B, Llama 3.1 70B} runs at roughly 1.1x the cost of the risky model alone, far cheaper than running two comparably sized models.

• Ablations confirm each component matters: Removing prefix debt, replacing the top-n log-likelihood-ratio statistic with a simple average, or switching from adaptive banking to a fixed cap or a lump-sum global budget all degrade the trade-off curve. The ∞-Rényi variant offers a worst-case guarantee and better fluency but slightly worse factuality than the KL formulation.

• Baseline comparison: System-prompt instructions barely reduce copying. MemFree and RCAD often fall short of the high-protection threshold, and when they cross it, they pay a substantial utility cost. CP-Fuse and TokenSwap reach high copying reduction but with worse utility than Anchored Decoding. The method also has weaker assumptions than prior two-model approaches, requiring neither a hand-curated seed-word list (TokenSwap) nor disjoint training shards (CP-Fuse).

• Per-step KL divergence is a useful risk signal: The gap between the risky and safe model's next-token distributions reliably identifies when generation has entered a copyright-sensitive regime, which is the intuition underlying the entire method.

Methodology in Plain English

The setup is simple in concept: you have a strong model you want to use but cannot fully trust, and a weaker, permissively licensed model you can trust. Rather than choosing one or the other, the method blends their next-token predictions at every step.

The blend is not arbitrary. The authors formalize a K-NAF constraint from prior work: the total divergence between the generated sequence's distribution and the safe model's distribution must stay under a budget K. Solving that constraint at the sequence level is intractable, so they decompose it into per-step constraints using the chain rule of KL divergence. Each step solves a small constrained optimization whose closed-form solution is a weighted geometric mean of the safe and risky next-token distributions. The mixing weight is found by solving a one-dimensional root-finding problem (a safeguarded Newton-Raphson iteration). Because the per-step budgets sum to K, the sequence-level guarantee holds automatically.

Two practical refinements make this work better. The prefix debt measures log-likelihood ratios between the risky and safe models over the prompt, takes the top-n largest positive values, and uses their average to pre-charge the budget. Prompts that look like they already trigger memorization thus get near-zero discretionary budget in early steps. The adaptive banking rule tracks actual divergence spent at each step and rolls the surplus forward, letting the model spend aggressively when the two distributions naturally agree and tighten when they diverge.

For cross-tokenizer settings, the byte-level variant performs the same computation over next-byte distributions derived by marginalizing token probabilities through a Valid Covering Tree traversal, so models with different vocabularies can still be fused.

Evaluation uses six copying metrics (ROUGE-1, ROUGE-L above a threshold, MinHash, word-level ACS, word- and character-level LCS) aggregated into a normalized copying reduction score, plus Prometheus-v2 for fluency on novel continuations and FActScore for factuality on biography generation.

Why This Matters

This work reframes copyright compliance from a training-data problem into a controllable decoding-time parameter. That is a meaningful shift: filtering or retraining frontier models is expensive and often degrades quality because copyrighted text is frequently high-quality, whereas a decoding-time constraint can be retrofitted onto any model with exposed logits. It also provides a formal guarantee rather than a heuristic, which matters for regulatory and legal contexts where "we tried to reduce copying" is weaker than "we can bound divergence to a trusted reference distribution."

Real-world applications:

• Deploying capable open-weight or closed-weight models in jurisdictions or contracts where verbatim reproduction of protected text creates liability.
• Content-licensing and creator-compensation frameworks that need a measurable, auditable bound on how much of a model's output derives from protected sources.
• Enterprise deployments where a permissively licensed safe model acts as a compliance anchor while a stronger risky model supplies capability.
• Cross-tokenizer or multilingual settings where safe and risky models come from different families and cannot be fused at the token level.

Industry relevance: Any organization serving a general-purpose language model to the public faces some copyright exposure. The method offers an inference-time knob that developers can tune per deployment, with a stated risk-utility curve and modest latency cost. It also motivates investment in permissively licensed safe models — the paper's own TinyComma 1.8B release is a step toward making these anchors widely available.

Future Directions

• Extending the K-NAF budget beyond copyright. The authors note the framework is tokenizer-, modality-, and domain-agnostic. Applying it to image, audio, or code generation under a trusted reference is an obvious next step.

• Better memorization signals. The prefix debt uses top-n log-likelihood ratios as a proxy for memorization likelihood. More accurate prompt-level or mid-generation memorization detectors could tighten budget allocation further.

• Non-literal copying. The paper's main metrics target near-duplicate and exact copying. Paraphrastic or stylistic infringement remains harder to measure and is only partially addressed; extending the guarantee to these cases is an open problem.

• Scaling and improving safe anchor models. The quality of the safe model directly caps the achievable utility. Larger, better permissively trained anchors — and the ecosystem of openly licensed corpora that support them — would raise the whole Pareto frontier.

Target Audience

Researchers and engineers working on language model safety, decoding algorithms, or data governance will get the most from this paper. It is also relevant to legal and policy teams evaluating technical mechanisms for copyright compliance, to practitioners deploying models under licensing constraints, and to anyone building or curating permissively licensed pre-training corpora. Readers with a background in information theory or constrained optimization will follow the formal arguments directly; others can rely on the intuition that per-step divergence between a risky and a safe model signals when generation is entering risky territory.

Authors’ abstract

Language models (LMs) tend to memorize portions of their training data and emit verbatim spans. When the underlying sources are sensitive or copyright-protected, such reproduction raises issues of consent and compensation for creators and compliance risks for developers. We propose Anchored Decoding, a plug-and-play inference-time method for suppressing verbatim copying: it enables decoding from any risky LM trained on mixed-license data by keeping generation in bounded proximity to a permissively trained safe LM. Anchored Decoding adaptively allocates a user-chosen information budget over the generation trajectory and enforces per-step constraints that yield a sequence-level guarantee, enabling a tunable risk-utility trade-off. To make Anchored Decoding practically useful, we introduce a new permissively trained safe model (TinyComma 1.8B), as well as Anchored$_{\mathrm{Byte}}$ Decoding, a byte-level variant of our method that enables cross-vocabulary fusion via the ByteSampler framework (Hayase et al., 2025). Across six model pairs on long-form metrics for copying risk and utility, Anchored and Anchored$_{\mathrm{Byte}}$ Decoding define a new Pareto frontier, preserving near-original fluency and factuality while closing up to 75% of the measurable copying gap between the risky baseline and a safe reference, at a modest inference overhead.