RegionMarker: A Region-Triggered Semantic Watermarking Framework for Embedding-as-a-Service Copyright Protection

AI summary

Overview

Research area: Copyright protection for Embedding-as-a-Service (EaaS) systems, at the intersection of NLP model security, watermarking, and model extraction attacks.

Technical level: Advanced. The paper assumes familiarity with text embeddings, dimensionality reduction (PCA), locality-sensitive hashing, distribution comparison tests (Kolmogorov-Smirnov), and adversarial attack models.

Scope in one sentence: The paper proposes and evaluates RegionMarker, a watermarking framework that triggers on semantic regions rather than trigger words, and reports that it resists three families of attacks (CSE, paraphrasing, and dimension-perturbation) where prior methods resist only a subset.

What This Paper Is About

Embedding-as-a-Service lets users send text to a provider and receive embeddings for a fee, but an attacker can query the service, collect the returned embeddings, and train a copycat model at low cost. Model providers try to defend against this by hiding a watermark inside the embeddings they return, so that a stolen model can later be identified. The problem is that existing watermarking schemes each survive only some attacks: trigger-word methods break under paraphrasing, and the linear-transformation method breaks under dimension perturbation, so an attacker who tries several attacks can usually find one that works.

Key Contributions

1. The paper surveys and compares existing EaaS watermarking methods (EmbMarker, WARDEN, EspeW, WET) against three attack types and reports in Table 1 that each prior method fails at least one attack, motivating a defense that covers all three.
2. It proposes RegionMarker, which defines trigger regions in a low-dimensional semantic space and injects a watermark embedding into any text embedding that falls inside those regions, using randomly selected regions and a secret dimensionality reduction matrix to make removal harder.
3. It designs a verification procedure that builds one backdoor corpus per watermark region plus one benign corpus, and reports cosine-similarity difference, squared L2 distance difference, and KS-test p-values, combined conservatively across regions.
4. It reports experiments on four datasets (SST-2, AG News, Enron, MIND) showing RegionMarker succeeds under all evaluated attacks, and includes an ablation on the necessity of dimensionality reduction and of using multiple watermark embeddings.

Main Findings

• Comprehensive defense on SST-2: RegionMarker is marked as successful copyright protection (p-value below 0.05) under no attack, CSE, paraphrasing with NLLB, paraphrasing with gpt-4o-mini, dimension-shift, and dimension-reduction, while WARDEN fails the two paraphrasing attacks, EspeW fails the two paraphrasing attacks, and WET fails the dimension-shift and dimension-reduction attacks.
• Baseline failures are specific, not general: WARDEN reports p > 0.30 (NLLB) and p > 0.25 (gpt-4o-mini) on SST-2 paraphrasing; EspeW reports p > 0.57 and p > 0.83 on the same two attacks; WET reports p > 0.46 under dimension-shift and has no reported results for dimension-reduction.
• Enron results follow the same pattern: WARDEN fails CSE on Enron (p > 0.05), EspeW fails both paraphrasing attacks (p > 0.49 and p > 0.28), and WET fails both dimension-perturbation attacks, while RegionMarker is marked successful in every row, with p-values ranging from < 10^-5 (no attack) to < 0.02 (dimension-reduction).
• Task performance is largely preserved: On SST-2, RegionMarker accuracy is 93.23 ± 0.36 with no attack and 93.29 ± 0.06 under dimension-reduction; the largest reported drop is under CSE, to 87.87 ± 0.73. On Enron, accuracy with no attack is 94.67 ± 0.18 and under CSE it is 95.55 ± 0.19.
• MIND results show lower absolute accuracy but consistent protection: RegionMarker reports 77.19 ± 0.10 accuracy with no attack and p-values below 0.05 for every attack, whereas WARDEN fails both paraphrasing attacks (p > 0.06 and p > 0.07), EspeW fails the gpt-4o-mini paraphrase (p > 0.39), and WET fails both dimension-perturbation attacks.
• Ablation on dimensionality reduction: On SST-2, RegionMarker without PCA still detects watermarks with no attack (p < 0.005) but fails under CSE (p > 0.5), compared with RegionMarker with PCA reaching p < 0.05 under CSE.
• Ablation on multiple watermarks: Assigning the same watermark embedding to all trigger regions reduces detection, and under CSE on SST-2 the watermark becomes ineffective (p > 0.08), whereas the multi-watermark version reports p < 0.05.
• Hyperparameter behavior: Detection performance improves as the watermark region ratio α increases, and the authors keep α at 20%. As the dimensionality after PCA increases, the cosine-similarity difference decreases without attacks but increases under attacks; the paper selects d = 4 and λ = 0.2.

Methodology in Plain English

The provider first compresses text embeddings into a small semantic space using dimensionality reduction such as PCA, because data in the original high-dimensional space is sparse and unevenly spread. In that compact space, the provider splits the space into 2^d regions with locality-sensitive hashing, so that semantically similar sentences land in the same region. A random subset of those regions, controlled by a ratio α, is secretly designated as the trigger set, and each trigger region gets its own watermark embedding, taken from the embedding of a target sample. When a user's text lands in a trigger region, the provider returns a blend of the original embedding and that region's watermark embedding, controlled by a strength parameter λ.

To check whether a suspect model was trained on the provider's outputs, the provider builds a verification corpus: sentences that fall in a trigger region on the provider's model, and sentences that do not. In a stolen model trained on watermarked embeddings, the trigger-region sentences should sit closer to the corresponding watermark embedding than benign sentences do. The provider measures this with cosine similarity and squared L2 distance, tests the difference with a Kolmogorov-Smirnov test, and takes the most conservative result across all watermark regions. If any region yields a p-value below 0.05, the provider treats the model as an infringement.

Experiments use the GPT-3 text-embedding-002 API as the provider's model and BERT as the stealer's model, trained with a learning rate of 5e-5, batch size 32, and the AdamW optimizer. CSE uses n = 20 and K = 50; WARDEN uses R = 2 and n = 20; paraphrase attacks generate five paraphrases per input and filter them at an 80% cosine similarity threshold.

Why This Matters

Impact on research: The paper reframes EaaS copyright protection as a coverage problem rather than a single-attack problem, arguing that a defense defeated by any one attack is effectively useless in practice. It also introduces semantic-region triggers as an alternative to both trigger words and global linear transformations, and shows through ablation that dimensionality reduction before partitioning is not cosmetic but necessary for surviving CSE.

Real-world applications:

• Commercial embedding APIs, such as the OpenAI text-embedding-3-large API cited in the paper, could embed provider-specific watermarks before returning vectors.
• Retrieval and recommendation services built on news data (AG News, MIND in this paper) could detect whether a vendor's cheaper substitute model was derived from a licensed embedding service.
• Spam and abuse classifiers (the Enron spam classification setting) deployed as hosted embedding services could carry ownership evidence.
• Sentiment or classification pipelines (the SST-2 setting) used by third-party developers could be audited for unauthorized extraction.

Industry relevance: Model extraction threatens the business model of any provider charging for embeddings, and the paper's verification procedure gives providers a statistical test they can run on a suspect model. The cost of the defense reported here is small in task terms: on SST-2 accuracy moves from 93.23 ± 0.36 without attack to 87.87 ± 0.73 under the strongest attack, and on Enron accuracy under CSE is 95.55 ± 0.19, so the protection does not obviously destroy the service's commercial utility.

Future Directions

• Determining how RegionMarker behaves when an attacker combines attacks, since the paper evaluates CSE, paraphrasing, and dimension perturbation separately and reports each row individually.
• Establishing whether the choice of d = 4, α = 20%, and λ = 0.2 transfers to embedding models other than GPT-3 text-embedding-002, which is the only provider model described in the implementation details.
• Investigating how the framework scales when the number of regions 2^d grows, since the paper notes that increasing PCA dimensionality grows the number of watermark embeddings while shrinking the sample size per region.
• Reproducing the reported advantage against attackers with knowledge of the defense, since the paper's threat model assumes the dimensionality reduction matrix and trigger regions remain secret.

Target Audience

Researchers working on watermarking, model extraction, and IP protection for machine-learning services; engineers building or operating commercial embedding APIs who need a detection mechanism that survives adversarial post-processing; and graduate students who want a concrete study of why single-mechanism defenses fail against adaptive attackers. Readers should be comfortable with embedding geometry, dimension reduction, hashing, and hypothesis testing.

Authors’ abstract

Embedding-as-a-Service (EaaS) is an effective and convenient deployment solution for addressing various NLP tasks. Nevertheless, recent research has shown that EaaS is vulnerable to model extraction attacks, which could lead to significant economic losses for model providers. For copyright protection, existing methods inject watermark embeddings into text embeddings and use them to detect copyright infringement. However, current watermarking methods often resist only a subset of attacks and fail to provide \textit{comprehensive} protection. To this end, we present the region-triggered semantic watermarking framework called RegionMarker, which defines trigger regions within a low-dimensional space and injects watermarks into text embeddings associated with these regions. By utilizing a secret dimensionality reduction matrix to project onto this subspace and randomly selecting trigger regions, RegionMarker makes it difficult for watermark removal attacks to evade detection. Furthermore, by embedding watermarks across the entire trigger region and using the text embedding as the watermark, RegionMarker is resilient to both paraphrasing and dimension-perturbation attacks. Extensive experiments on various datasets show that RegionMarker is effective in resisting different attack methods, thereby protecting the copyright of EaaS.