SkillMarker: A KGW-Based Watermarking Method for Agent Skill Ownership Verification

AI summary

Overview

Research area: AI security and provenance — specifically watermarking of large language model (LLM) output applied to "agent skills" (reusable plaintext instruction packages that agents load to specialize behavior). arXiv category cs.CR; work from Huawei Technologies Co., Ltd. and Harbin Institute of Technology, Shenzhen.

Technical level: Intermediate. Readers need some familiarity with LLM token generation, logits, and the KGW (Kirchenbauer et al., 2023) green-list watermarking scheme to follow Section 2 in detail, though the core idea is accessible.

Scope (one sentence): The paper proposes and evaluates SkillMarker, a KGW-based watermarking pipeline that rewrites agent skill text under strict preservation constraints so that ownership can later be verified and leaked or copied skill content traced, tested across four datasets for downstream task performance, copy-paste detection, localization, and skill merging.

What This Paper Is About

Agent skills package domain expertise and operational procedures into reusable instructions, but they are distributed as plaintext, which makes them easy to copy, redistribute, or fold into other skills. The paper asks whether a watermark can be embedded into a skill before distribution so that a developer can later verify ownership or trace which distributed version leaked — without breaking the skill's functionality.

The difficulty is twofold: skill instructions are structured and compact, so small edits to conditions, identifiers, or numerical thresholds can change agent behavior, and that same compactness leaves little room to build a strong watermark signal. SkillMarker targets both problems jointly.

Key Contributions

1. A KGW adaptation for skill text. SkillMarker adapts KGW watermarking through two components — quality-constrained rewriting and signal-oriented candidate search — explicitly aimed at preserving skill functionality while embedding a strong watermark under limited rewriting freedom.
2. Empirical functionality check across four datasets. Using agent-skills, awesome-skills, marketingskills, and skill-use-bench, the authors show watermarked skills change task performance by −2.01% to +0.93% relative to original skills, which they describe as negligible degradation on average.
3. Robustness evaluation under two reuse scenarios. Detection is measured for copy-paste (copied passages inserted into unwatermarked text) and skill merging (a target skill merged with zero, one, or three background skills), with localization reported for copy-paste.
4. A supporting toolkit. The authors develop a toolkit covering watermark embedding, quality auditing, and detection, supporting both full-text and window-based detection, with a project homepage at https://fuxi-ailabs.github.io/SkillMarker/.

Main Findings

• Task performance is largely preserved: Relative to original skills, watermarking changes performance by −2.01% to +0.93% across the four datasets, and watermarked skills retain their gains over the no-skill baseline (original skills improve performance over no-skill runs by 8.49–40.30%).
• Per-dataset pass rates: On agent-skills, pass rates are 97.46% before and 98.25% after watermarking; on skill-use-bench, 59.85% and 60.41%.
• Token and time costs are mixed: Mean total tokens increase by 14.1–29.3% on the first three datasets but decrease by 3.3% on skill-use-bench; median task time increases by 14.5%, 6.0%, and 16.4% for agent-skills, awesome-skills, and marketingskills, and decreases by 5.3% for skill-use-bench. The authors note these means are sensitive to long-tailed run token counts and that timing is reported for reference only.
• Copy-paste detection at 1,536 tokens: Window-based detection consistently identifies copied passages containing as few as 1,536 consecutive tokens, under both 1024-token and 4096-token unwatermarked backgrounds. In the longer background, full-text detection reaches only 55.6%, which the authors cite as the benefit of window-based detection.
• Localization works but is loose: The same 1,536-token passages are localized with 98.0–99.1% recall and 65.4–69.6% precision across the two background lengths, meaning detection windows recover nearly all copied content but also include surrounding text.
• Fragmentation hurts: Splitting the passage into eight fragments in the 4096-token background reduces detection to 77.8%.
• Merging dilutes but does not erase the watermark: With one, two, or four total input skills, multi-scale union detection achieves 91.7%, 72.2%, and 77.8% detection, versus 58.3%, 47.2%, and 44.4% for single-window detection.
• False positives rise with background noise: Both merge detectors show the same observed false-positive rates on unwatermarked controls — 0.0%, 11.1%, and 16.7% for one, two, and four input skills — which the authors say means reliable detection under greater background noise requires calibration.
• Ablation shows a quality–strength trade-off: SkillMarker reaches quality 70.97 with mean KGW z = 3.01. Removing fidelity constraints raises z to 3.37 but lowers quality to 67.09; removing beam search, signal-oriented selection, or logit bias drops mean z to 0.65, 0.69, or 0.83 while quality rises to 76.06, 75.50, or 73.06. Generated-token scoring stays close to SkillMarker (70.00 quality, z = 2.98), which the authors say does not by itself establish a substantial benefit from reconstructed-text scoring.
• Case study: For the skill observability-and-instrumentation, protected spans remain exactly unchanged and the saved full text yields nominal p = 4.62 × 10⁻⁹.

Methodology in Plain English

The watermarking idea. The method follows KGW: at each generation step, a secret key and the previous token determine a "green list" of vocabulary items, and the model's next-token probabilities are biased toward that list. Text produced this way contains more green tokens than chance, and a detector that knows the key can count green tokens and compute a z-score and a binomial upper-tail probability.

Step 1 — Protect what must not change. The original skill is parsed, and code, structural delimiters, recognized literals, and any region with unresolved structure are marked as protected and left untouched. The remaining prose is split into editable spans; spans shorter than a token-count threshold are left alone, and tokens straddling a protected boundary are protected too.

Step 2 — Rewrite with fidelity constraints. Each editable span is rewritten one at a time using a prompt that combines explicit preservation instructions with paired input/rewrite demonstrations, showing how to change wording while keeping facts, conditions, negations, numerical values, and technical identifiers. Preservation checks reject empty or incomplete candidates, unauthorized layout or syntax changes, and any change to protected content. The skill is then reconstructed with protected content in its original order.

Step 3 — Search for stronger signals. Because strict constraints can push the model to copy the input verbatim, beam search explores alternative expressions under the KGW-biased distribution. Candidates are scored on the reconstructed, retokenized text rather than on the raw generated sequence, so the measured signal matches what actually ships. Selection is lexicographic: higher green-token rate first, then lower perplexity (PPL) as a tie-break. If no candidate passes, the original span is kept and a fallback is recorded. Selection is local to each span, so it does not guarantee the best global green-token rate.

Step 4 — Avoid formatting damage. Token variants that differ only in leading spaces or periods are grouped so they share green-list membership from a representative, and formatting and control tokens are exempted from the bias, following soft KGW.

Step 5 — Detect. Detection needs only the tokenizer and watermark configuration, not model inference. It supports full-text scoring and window-based scanning (1024-token windows, stride 256, per-window threshold p ≤ 0.01 in the copy-paste tests, with a Bonferroni correction across windows), reporting the green-token rate and z-statistic. Because repeated contexts and shared green-list membership can violate the binomial null assumptions, the authors state these probabilities need independent calibration before being read as false-positive guarantees.

Step 6 — Audit separately. Quality auditing judges language, semantics, completeness, and usability without exposing watermark scores to reviewers, and downstream task performance is evaluated separately using DeepSeek-V4-Flash as the executing agent/judge.

Why This Matters

Impact on research. The paper reframes watermarking from a model-provider tool for attributing generated text into a publisher-side tool for protecting a distributed artifact. It shows that the preservation constraints that make skill watermarking hard (protected code and literals, exact thresholds, structured Markdown) can be handled by combining span-level protection with candidate search, and it contributes a robustness protocol — copy-paste with controlled fragment counts and backgrounds, plus skill merging — that other provenance methods can reuse. It also reports a trade-off the field needs to confront: stronger watermark signal came at a measurable cost in holistic quality score in the ablation.

Real-world applications (bullets):

• Skill marketplaces and internal catalogs: embed a keyed watermark before publishing a skill, then test a suspect text for the signal to establish ownership.
• Leak tracing across distribution versions: assign distinct keys to different distributed versions so a detected signal helps identify which version a suspect text came from, including when only fragments survive.
• Detecting copied instructions inside third-party skills: window-based scanning with localization supports finding a copied passage embedded in a larger, mostly unwatermarked document, which the paper tests with 1,536-token passages in 4096-token backgrounds.
• Pre-release quality gating: the accompanying toolkit's independent quality auditing and preservation checks act as a gate before a watermarked skill is shipped.

Industry relevance. The work comes from an industrial lab and targets a concrete commercial asset — proprietary agent skills. The reported costs are directly relevant to deployment: token overhead of 14.1–29.3% on three datasets (a 3.3% decrease on skill-use-bench), median time changes from −5.3% to +16.4%, and false positives of up to 16.7% on unwatermarked controls in the four-skill merge setting, which the authors flag as requiring calibration before operational use.

Future Directions

1. Calibrate the detection statistics. The authors state that nominal binomial probabilities require independent calibration before being interpreted as empirical false-positive guarantees, and that merge-detector parameters were selected on the same evaluation cohort without a held-out test set. A held-out, calibrated detector is the obvious next step.
2. Reduce false positives under merging. Reported control false-positive rates rise from 0.0% to 11.1% to 16.7% as input skills go from one to two to four; bringing these down without losing the 72.2–77.8% detection at higher background counts is an open problem.
3. Handle dispersed and fragmented reuse. Detection falls to 77.8% when a passage is split into eight fragments in the 4096-token background, and localization precision is 65.4–69.6% even for contiguous copies; better fragment-aware scanning and tighter localization are natural extensions.
4. Separate literal preservation from functional equivalence. The authors note in the appendix that literal preservation and detection do not establish semantic or functional equivalence, and their ablation uses a model-review quality score rather than execution-based correctness; closing that gap, and extending beyond the English prompt and frozen English release used here, would strengthen the claims.

Target Audience

Researchers and engineers working on LLM watermarking, AI provenance, and model/agent security will get the most from this paper, particularly those interested in KGW variants and in the practical constraints of watermarking structured, short, accuracy-critical text. It also suits platform and marketplace teams who distribute proprietary prompts, skills, or instruction packages and need an ownership-verification mechanism, and evaluation-focused readers who want a concrete robustness protocol (controlled copy lengths, fragment counts, and merge settings) to reuse or critique. Readers without background in token-level watermarking will need to consult the KGW reference for the detection statistic.

Authors’ abstract

Agent skills encode operational procedures and domain expertise as reusable knowledge assets, yet their distribution in plaintext exposes them to unauthorized copying and leakage. However, practical mechanisms for skill ownership verification and leakage tracing remain underdeveloped. Watermark embedding before distribution offers a potential solution, but structured, compact skill instructions impose stringent requirements for factual and procedural accuracy. To address these challenges, we introduce SkillMarker, a KGW-based watermarking method that jointly targets functionality preservation and strong watermark embedding. SkillMarker unifies quality-constrained rewriting with signal-oriented candidate search. The former separates essential knowledge from flexible expression to preserve instruction fidelity; the latter explores alternative expressions to obtain a stronger watermark signal under preservation constraints. We release an open-source toolkit that integrates watermark embedding, quality auditing, and detection to support the protection of agent skills. SkillMarker targets ownership verification and leakage tracing in distributed skill ecosystems, where content may be copied in whole or in part, or incorporated into other skills. Across four datasets, SkillMarker negligibly affects downstream task performance on average. In our copy-paste tests, window-based detection consistently identifies copied passages containing as few as 1,536 consecutive tokens. Even after merging with one or three background skills, SkillMarker retains watermark detection rates of 72.2--77.8\%.