Robust Watermarking on Gradient Boosting Decision Trees

AI summary

Overview

Research area: Machine learning security and intellectual property protection — specifically watermarking of Gradient Boosting Decision Tree (GBDT) models.

Technical level: Intermediate. The reader needs a working sense of gradient boosting, model fine-tuning, and classification metrics, but the paper's framing is largely conceptual and its equations are standard boosting updates.

Scope (one sentence): The paper proposes and empirically evaluates the first robust watermarking framework for GBDT models, embedding ownership signals through in-place fine-tuning of existing trees using four candidate-selection strategies.

What This Paper Is About

GBDT models are widely used — often outperforming neural networks on structured, noisy, or imbalanced data — yet watermarking research has focused overwhelmingly on neural networks. Watermarking neural networks relies on smooth, differentiable decision boundaries, which GBDT models do not have, and GBDT trees are built sequentially so that modifying one tree can cascade into accuracy loss in later trees. This paper asks whether hidden, verifiable ownership signals can be embedded into an already-trained GBDT model without meaningfully hurting its accuracy and without being erased by later fine-tuning.

Key Contributions

1. First robust GBDT watermarking framework with in-place updating. The authors state this is the first robust watermarking framework with in-place updating for GBDT, and the first work to focus on robust GBDT watermarking in general.
2. Four watermark embedding approaches. Wrong Prediction Flip, Outlier Flip, Cluster Center Flip, and Confidence Flip, each designed to minimize accuracy degradation while embedding robust watermarks.
3. In-place fine-tuning mechanism. An algorithm that adjusts the internal parameters of existing trees (recomputing gains and best splits, retraining subtrees where the split changes, and updating terminal node predictions) rather than adding new trees that could be pruned away.
4. Empirical evaluation across six public datasets. Demonstrating high watermark embedding success, limited accuracy impact, and robustness against further fine-tuning, compared against a random-selection baseline.

Main Findings

• Watermarking succeeds at high rates in most settings. In the setting where the candidate dataset equals the training set (D_cand = D_train, with duplication factor d_cand=train = 5), average success rates across datasets at watermark ratios 0.001, 0.01, and 0.1 were, for example, 0.792 / 0.980 / 0.999 for Cluster (Conf), 0.875 / 0.953 / 1.000 for Outlier (Dist), and 0.812 / 0.959 / 0.999 for Confidence (Dist).
• The random baseline is consistently worse. Under the same setting, Random (Conf) averaged 0.694 / 0.819 / 0.982 and Random (Dist) averaged 0.708 / 0.880 / 0.991, a significant gap from the proposed methods.
• Disjoint candidate data improves success. When the candidate dataset is separate from the training data (D_cand != D_train, without watermark duplication), Confidence Flip had the highest average success rates (1.000 / 1.000 / 0.947 for Conf. (Conf) and 1.000 / 1.000 / 0.927 for Conf. (Dist)), followed by Wrong Prediction Flip where it was valid. The Random baseline reached 1.000 / 0.913 / 0.800.
• Wrong Prediction Flip is highly accurate but rarely applicable. It depends on samples the initial model misclassifies, so its entries are often reported as "—" (not available). The authors note this limits how many watermarks it can support, especially when D_cand = D_train, because GBDT tends to be very accurate on its training data.
• Cluster Flip and Wrong Prediction Flip preserve accuracy best in the disjoint setting. The paper reports that these two generally yield the highest adjusted model accuracy (A_model' = A_model × A_wm) when sufficient incorrect predictions are available, because Cluster Flip anchors watermarks near existing boundaries via nearest-neighbor heuristics and Wrong Prediction Flip only touches already-misclassified samples. Outlier Flip and Confidence Flip rely only on the implicit assumption that outliers or low-confidence points sit in sparse regions.
• The accuracy gap is less visible when candidates come from training data. In that setting a gap versus the baseline is still visible, especially at watermark ratios 0.01 and 0.1.
• Watermarks survive post-deployment fine-tuning. The proposed methods generally show good robustness compared with the baseline, measured as the proportion of correctly embedded watermark samples that remain intact after fine-tuning on a held-out dataset.
• Relying on misclassified samples carries an ambiguity risk. Figure 3 compares the proportion of incorrect predictions by the initial model that are also incorrect under models trained with other GBDT libraries on the optdigits dataset; the high similarity indicates these may simply be "hard" samples that unrelated models also mispredict, which is why Wrong Prediction Flip uses the second-most-probable incorrect class as the modified label.
• Embedding is not theoretically guaranteed. The authors show analytically that, depending on the duplication multiplier r and the model's probabilities, the gradient for the watermark class is not strictly guaranteed to be negative, so watermark embedding is not assured.
• In-place updating is motivated by removability. The paper argues that the common practice of adding trees during fine-tuning is problematic for watermarking because low-contribution trees can be pruned away.

Methodology in Plain English

The workflow starts with a base model trained on D_train, which is then used to predict on a separate candidate dataset D_cand. From those predictions, the method builds a set of n watermark candidates C using one of four rules:

• Wrong Prediction Flip: pick samples the model gets wrong, choosing the n lowest-confidence ones, and relabel them with the second-most-probable incorrect class rather than the model's own (often hard-case) prediction.
• Outlier Flip: run k-Means (choosing the cluster count m that maximizes the silhouette score) and pick the n correctly predicted samples farthest from any cluster centroid, then relabel with the highest-probability incorrect class.
• Cluster Center Flip: pick the points closest to each of n cluster centroids as watermarks, relabel them, and include their l nearest neighbors with their original correct labels to anchor the boundary around a small localized "hole." The centroid group is duplicated once during watermarking to counteract the diluting effect of the neighbors.
• Confidence Flip: pick the n correctly classified samples with the lowest confidence, which tend to sit near decision boundaries where predictions are easier to shift.

From the candidate set C, a subset W of size k is selected using one of two strategies: Lowest Confidence (rank by prediction confidence and take the smallest) or Maximum Distance (maximize separation between watermarks, solved greedily since the exact maximum-diversity problem is NP-hard).

The selected samples are relabeled and the model is retrained by fine-tuning on them. Crucially, fine-tuning uses in-place updating rather than adding trees: for each boosting iteration and class, the algorithm computes pseudo-residuals, recomputes gains and best splits for each non-terminal node in depth-first top-down order, retrains the subtree when the best split changes, and updates terminal-node predictions. When candidates are drawn from the training set, they are duplicated five times (d_cand=train = 5) so their gradients dominate updates the model has already seen.

Evaluation uses six public datasets: Avila, Image Segmentation, Letter Recognition, Wine Quality, optdigits, and pendigits. Watermark ratios of 0.001, 0.01, and 0.1 are tested, all watermark bits are set to 1 as a stress test, all models are trained for 200 iterations in all phases, and candidate set size is set to |C| = 2|W|. Two scenarios are used: watermarking with D_cand = D_train (internal watermarking) and with D_cand != D_train, where the data is split into D_train, D_cand, D_test, and D_fine using 80:20 splits. Four metrics are reported: watermarking effectiveness A_wm, general accuracy A_model on the test set, adjusted model accuracy A_model' = A_model × A_wm, and fine-tuning robustness.

Why This Matters

Impact on research: The paper opens a research direction that has been largely neglected. Prior work on tree watermarking either targeted random forests, where trees are independent, or focused on fragile (weak) watermarking for integrity authentication in boosted trees, which is a different goal from robust ownership verification. GBDT's non-differentiability and sequential tree construction had been practical blockers, and this work offers a concrete route around them.

Real-world applications:

• Licensing and ownership verification for commercial models built on structured data, where GBDT is often the model of choice.
• Post-hoc watermarking by model purchasers or third parties, the scenario the authors explicitly simulate with disjoint D_cand and D_train.
• Healthcare and privacy-sensitive deployments, domains the paper identifies as heavy GBDT users, where provenance and compliance evidence matter.
• Outsourced or externally maintained models, where the party deploying the model is not the party that trained it.

Industry relevance: GBDT libraries such as XGBoost and LightGBM are standard in industrial pipelines. The finding that watermarks survive further fine-tuning matters because deployed models are frequently retrained on new data — the watermark needs to persist through exactly that process to be useful as evidence of provenance.

Future Directions

• Addressing candidate scarcity for Wrong Prediction Flip. The method performs well where applicable but is often unavailable; the paper does not report a solution beyond noting the limitation, leaving open how to generate or substitute suitable candidates.
• Testing robustness beyond fine-tuning. The paper evaluates resilience to further fine-tuning only. Removal attacks such as pruning, or the tree-pruning concern that motivated in-place updating in the first place, are not evaluated here.
• Questioning the distributional assumptions. Outlier Flip's accuracy preservation and resilience depend on similarity between the fine-tuning data distribution and D_cand, which the authors acknowledge may not always hold. How to make the method robust to distribution shift is unresolved.
• Turning the reported strategic guidance into general rules. The paper states it synthesizes its findings into recommendations for when each embedding strategy performs best, but the content of those recommendations is not included in the material provided, and the theoretical embedding guarantee remains open given the gradient analysis showing no strict guarantee exists.

Target Audience

Researchers and practitioners in machine learning security, model intellectual property protection, and watermarking; engineers deploying GBDT models in commercial or regulated settings who need ownership verification; and readers already familiar with gradient boosting who want an accessible entry point into tree-model watermarking, since the paper contrasts its approach directly with the much larger body of neural network watermarking work.

Authors’ abstract

Gradient Boosting Decision Trees (GBDTs) are widely used in industry and academia for their high accuracy and efficiency, particularly on structured data. However, watermarking GBDT models remains underexplored compared to neural networks. In this work, we present the first robust watermarking framework tailored to GBDT models, utilizing in-place fine-tuning to embed imperceptible and resilient watermarks. We propose four embedding strategies, each designed to minimize impact on model accuracy while ensuring watermark robustness. Through experiments across diverse datasets, we demonstrate that our methods achieve high watermark embedding rates, low accuracy degradation, and strong resistance to post-deployment fine-tuning.