Online Shift Detection and Conformal Adaptation for Deployed Safety Classifiers

AI summary

Overview

Research area: Machine learning safety — online distribution-shift detection for deployed LLM safety classifiers, conformal prediction under covariate shift, and adversarial evasion of score-based monitoring.

Technical level: Advanced. The paper combines sequential testing theory (martingales, confidence sequences), conformal prediction, density-ratio estimation, and adversarial attack evaluation, with a pre-registered factorial experiment.

Scope (one sentence): The paper evaluates whether sequential statistical tests on the outputs of deployed safety classifiers can detect distributional shift online, whether weighted conformal prediction can restore coverage after such a shift, and under what conditions score-disagreement "canaries" detect gradient-based evasion.

What This Paper Is About

Safety classifiers sit between a language model and its users, and they are deployed under the assumption that tomorrow's inputs look like the inputs they were calibrated on. When that assumption breaks — through adversarial attack, paraphrasing, code-switching, long-context composition, or emerging harm categories — the classifier can degrade silently, with no error signal until ground-truth labels arrive. This paper builds and stress-tests an online monitor that watches the distribution of classifier scores, raises an alarm when it drifts, and then tries to recover coverage with weighted conformal prediction. The paper also asks a second question: when a "canary" classifier disagrees with the deployed classifier, does that disagreement reliably flag gradient-based evasion, and can an attacker who knows about the canary suppress it?

Key Contributions

1. Factorial drift benchmark. A pre-registered 800-cell evaluation (4 classifiers × 5 shift types × 20 seeds × 2 windows) establishing detection baselines. It reveals that detection difficulty is dominated by a classifier × shift interaction (η² = 0.185), so monitoring must be tuned per classifier–shift pair. A calibration-free scan martingale replaces empirical threshold tuning, reducing false-alarm-rate spread from 5× to <2× across classifiers.

2. Conformal failure mode (density-ratio collapse). A diagnosis of why weighted conformal prediction — the standard post-shift coverage recovery method — fails silently on decoder classifiers: logistic density-ratio estimation achieves perfect source/target separability in 3584–4096-dimensional space, clipping all importance weights to the floor. Projecting to ≤32 dimensions restores coverage (up to +33pp; reported as +21–33pp in the contributions list).

3. Adversarial canary characterization. An empirical security characterization of score-disagreement monitoring against gradient-based evasion. The paper falsifies three assumptions about score-disagreement canaries, derives a confidence-gated equilibrium bound (gap = 1/(2λ), validated within 95% CI at n = 20), and provides a deployment blueprint across 35 frontier models with Wilson CI guarantees (≥71% detection at <1.5% FPR; N = 1000 FPR validation on 10 Pareto-dominant models).

4. Reasoning-token budget starvation. A methodological finding that reasoning models used as safety monitors silently fail under standard token budgets because adversarial inputs require 3.3× more reasoning tokens than benign inputs to produce valid safety scores.

Main Findings

• Reasoning-token budget starvation. Adversarial inputs require 3.3× more reasoning tokens than benign inputs to produce valid safety scores. For o3, T₅₀,adv = 154 tokens versus T₅₀,benign = 46 tokens. The steepness of the response sigmoid differs by 5.8× (k_benign = 0.173 versus k_adv = 0.030), so adversarial inputs degrade more gradually as budget shrinks. The paper's empirical rule is T_r ≥ T₅₀(hardest class) + 2.2/k for >90% coverage, which for adversarial safety monitoring with o3 means T_r ≥ 227 tokens — well above the 16-token default that caused 11 models to appear "ceiling-clipped" in the initial screen. Insufficient token allocation produces silent scoring failures that default to maximum-risk scores, inflating apparent detection rates.

• Threat landscape. Template jailbreaks fail at 99% in the paper's evaluation (1% success rate), leaving gradient-based evasion — GCG-optimized suffixes — as the residual operational threat class, which flips encoder decisions reliably.

• Overall detection rate. The system detected shift in 693 of 800 cells (86.6% valid detection rate, 95% Wilson CI [0.841, 0.888]), with empirical false alarm rates of 2–10% across classifiers. Raw TPR across alarming cells is 91.5%, with a 5.8 pp gap attributable to false alarms on negative controls.

• Crossover interaction. Encoders detect paraphrase drift in 28 steps but miss adversarial suffixes for 37; decoders show the opposite pattern. Paraphrase is easy for encoders (28–35 steps) but hard for decoders (69–85 steps); adversarial suffix is hardest for DeBERTa (36.6 steps) but easiest for Llama Guard (27.8 steps). The slowest cell is Llama Guard × code-switch at 93.4 steps [81, 107].

• False alarm rates. Text-Moderation 2.0% [0.8%, 5.0%] < Llama Guard 3.0% [1.4%, 6.4%] < ShieldGemma 8.5% [5.4%, 13.2%] < DeBERTa 9.5% [6.2%, 14.4%]. The 5× spread reflects differences in null score distribution shape (ShieldGemma σ = 0.14, Llama Guard σ = 0.14).

• Window size trade-off. w = 100 detects 7 steps faster on average (39.5 [37.0, 42.4] versus 45.4 [42.6, 48.3]; paired difference −7.0 [−8.6, −5.3], n = 314 pairs) at the cost of slightly higher false alarm rates.

• Failure concentration. Of 107 invalid cells: 68 alarmed before shift onset, 46 had dirty negative controls, 7 exhibited both. Failures concentrate in DeBERTa × adversarial-suffix (13/40 invalid, 32.5%) and Llama Guard × adversarial-suffix (11/40 invalid, 27.5%). The DeBERTa failure is explained by GCG suffixes pushing DeBERTa's scores into the safe region; the Llama Guard failure is the opposite — adversarial tokens produce such a strong signal that the detector fires during calibration.

• Density-ratio collapse is shift-agnostic and systematic. For Text-Moderation, Llama Guard, and ShieldGemma, effective sample size (ESS) sits at or near the maximum of 300 (all calibration weights floor-clipped to 1/C) across temporal, paraphrase, and adversarial suffix shifts. The mechanism: logistic regression assigns P̂(target | x) ≈ 0 to every calibration point, so all weights clip to the floor 1/C = 0.1, eliminating data-driven reweighting.

• Residual recoveries at ESS ≈ 300 are a formula artifact. The test-point weight w(X_{n+1}) = 1.0 raises the effective quantile level from 90.3% to 93.3% at n_cal = 300, ε = 0.1 — not adaptation.

• DeBERTa is the sole classifier with meaningful weighted-conformal recovery, with a gradient across shift types: paraphrase (ESS = 46, recovery = +0.390) > temporal (ESS = 88, recovery = +0.160) > adversarial suffix (ESS = 206, recovery = +0.020). On adversarial suffix, 92.7% of DeBERTa's weights sit at the floor with max weight 0.82. ESS is the continuous predictor of recovery: ESS = 46 yields +0.390; ESS = 88 yields +0.160; ESS ≥ 200 yields ≤ +0.020.

• Dimensionality reduction as diagnostic. PCA to 32 dimensions breaks the collapse for both generative classifiers on temporal shift: Llama Guard recovers 33.0 pp (coverage 0.555 → 0.885, ESS drops from 300 to 19.6) and ShieldGemma recovers 20.5 pp (0.690 → 0.895, ESS drops from 300 to 85.0). At 32 dimensions, 82–91% of embedding variance is retained. At 64 dimensions, ShieldGemma re-collapses (the provided content is truncated at this point).

• Scan martingale. The scan martingale achieves FAR ≤ 1% across all four classifiers with no per-classifier null-threshold calibration, versus 2–10% under empirical KS calibration. It uses ε = 0.3 and M = 50 concurrent sub-martingales (validated at M ∈ {50, 100}). ε = 0.3 achieves 100% detection for encoders but 57–77% for decoders with wide null distributions.

• Ramped-onset adversary. Against DeBERTa with a 50-step ramp and small per-sample shift: at 15% mixing KS detects 3% versus scan 37% (+34 pp); at 20% mixing KS 7% versus scan 67% (+60 pp); at 25% mixing KS 10% versus scan 87% (+77 pp); at 30% mixing KS 47% versus scan 100% (+53 pp). The martingale's advantage is specific to ramped or accelerating onsets.

• Residual blind spot. Against a constant sub-threshold adversary (sustained low-rate injection at ≤12% mixing without ramp), neither KS nor the scan martingale reliably detects.

• Canary assumptions falsified. Three assumptions are falsified: that architectural diversity drives detection (false), that canaries are generic OOD detectors (false; GCG-specific, p < 10⁻¹²), and that an adaptive attacker can suppress the signal (false while the canary is confident).

• Corpus validation. Paraphrase: approximately 14–20% of examples became LLM refusals. A filtered ablation removing the 9.4% of examples identified as refusals (n = 5 seeds × 2 classifiers) shows negligible effect: DeBERTa latency 38.0 → 37.8 steps, Llama Guard 66.6 → 60.8 steps, both 5/5 detected. Code-switch: all 50 examples confirmed as authentic Singlish by a native speaker; 20–30% became refusals. Compositional: 20/20 correctly placed harmful content. Temporal: 20/20 reviewed examples were genuine jailbreak prompts; zero false positives. Adversarial suffix: 20/22 showed correct suffix concatenation with confirmed score flips (original ≥0.95 → attacked ≤0.01); one example was excluded post-validation.

• Martingale honesty note. Under instantaneous step-onset shift at full mixing, per-condition calibrated KS thresholds match or exceed the scan martingale's detection rate. The martingale's contribution is operational: exchangeability-valid FAR ≤ α with no per-classifier null-threshold tuning.

Methodology in Plain English

The monitor watches a classifier's output stream, which consists of an input text, an unsafe-class probability score s ∈ [0,1], and the classifier's penultimate-layer representation vector. Before monitoring begins, the system records a reference sample (n_ref = 500 records) and freezes the reference score distribution; nothing about the reference is updated while monitoring, which is what keeps the statistical tests valid.

Two detection channels are built. The first is a Kolmogorov–Smirnov (KS) detector: it keeps a sliding window of the most recent w scores and measures the largest pointwise gap between the window's empirical distribution and the frozen reference distribution. This differs from the KSWIN method, which compares two adaptive sub-windows, in that the reference here is frozen. The second channel is an MMD detector on embeddings, with a Gaussian kernel whose bandwidth is fixed at calibration time using the median heuristic and a threshold set by a 1000-permutation bootstrap on the reference embeddings. The factorial evaluation uses the KS detector only; the MMD channel is evaluated separately on a representative subset.

Alarm thresholds are set empirically. Fifty negative control streams containing only reference data are pushed through the entire pipeline, and the alarm threshold is set at the 97th percentile of the maximum statistic observed in those runs. This accounts for the sliding-window correlation structure, window size, and stream length, which a theoretical Hoeffding bound treats conservatively. Because that empirical approach produced a 5× spread in false alarm rates across classifiers, the authors also implement a conformal test martingale: each incoming score is converted into a two-sided conformal p-value against the frozen reference CDF, a betting-based wealth process accumulates evidence, M = 50 sub-martingales are run concurrently, and an alarm fires when any one exceeds its threshold. Ville's inequality plus a union bound gives an anytime-valid false-alarm guarantee.

When an alarm fires, an abstention layer switches on. In unweighted mode it is standard split-conformal prediction: compute nonconformity scores on calibration data, take the appropriate quantile, include classes whose score is below that threshold, and abstain when the prediction set has more than one class (or is empty). Under shift, exchangeability breaks and coverage degrades. In weighted-on-alarm mode, the paper follows Tibshirani et al. (2019): a logistic regression is trained to distinguish source from target embeddings to produce density ratios, these are clipped to [1/C, C] with C = 10, and a weighted quantile replaces the ordinary quantile. The paper tested whether PCA before density-ratio estimation restores the method.

For the factorial experiment, each of the 800 cells runs the full pipeline end-to-end, and each cell includes a parallel negative control. A cell counts as a valid detection only if detection latency is ≥ 0 and the negative control does not alarm. The design, including all hyperparameters (α = 0.05, calibration percentile = 97, reference size = 500), was committed before execution (commit be630f3). The stated deviations from pre-registration are limited to dropping w = 500 because it left insufficient post-shift observations with 300 shifted examples. Compute was 120 hours total across a Mac Studio (M3 Ultra, 96GB) and a MacBook Pro (M3 Max, 128GB).

Statistics: BCa bootstrap CIs on means (10,000 resamples, seed = 42), Wilson score intervals on rates at 95% confidence, Clopper–Pearson exact intervals on coverage proportions, permutation tests (10,000 permutations for pairwise comparisons, 1,000 for ANOVA), and Holm–Bonferroni correction across 8 highlighted pairwise comparisons, all of which survive correction (weakest adjusted p = 0.044). Variance decomposition uses a two-way fixed-effects ANOVA with classifier (4 levels) and shift type (5 levels), reporting η².

Why This Matters

Impact on research. The paper argues that any study using reasoning models as judges, monitors, or evaluators must verify that its token budget exceeds T₅₀ for its hardest input class — reporting that a 16-token default made 11 models appear ceiling-clipped. It also documents a failure mode in weighted conformal prediction that operates on exactly the classifier architectures (8B+ decoders) most widely deployed, and shows that the standard remedy can silently do nothing while appearing to run. The factorization of the detection benchmark yields a classifier × shift interaction (η² = 0.185) that single-classifier studies structurally cannot see.

Real-world applications.

• Production content moderation, where a safety classifier silently degrades and labels arrive too late to catch it.
• Automated red-teaming pipelines, where GCG-style suffixes are the residual threat that template jailbreaks do not represent.
• Multilingual platforms, where code-switching is the slowest-detected shift condition in the benchmark (Llama Guard × code-switch, 93.4 steps).
• Long-context systems, where compositional attacks chain benign components into harmful sequences.

Industry relevance. The paper offers a concrete deployment recipe: (1) choose a canary model with independent training data or architecture; (2) estimate benign disagreement on held-out traffic; (3) set a threshold for a desired false-positive rate; (4) alert on sustained disagreement; (5) periodically rotate the canary to limit adaptive attacker knowledge. The stated guarantee is ≥71% detection at <1.5% FPR at N = 1000. The paper is direct that the practical value is as a non-adaptive tripwire whose failure modes can be measured before deployment, and it explicitly declines to claim novelty for two-classifier disagreement, KS tests, or martingale testing.

Future Directions

• Online conformal prediction without exchangeability. The paper cites Gibbs & Candès (2021) as extending conformal prediction to the fully online setting and notes this as future work.
• Broader martingale hyperparameter sweeps. Results are validated at M ∈ {50, 100}; the paper states that broader window sweeps are deferred, and that ε and M remain deployment hyperparameters whose settings trade the Type I guarantee against Type II detection delay.
• Handling long-horizon, non-exchangeable benign drift. For deployments where benign drift may violate exchangeability, the paper recommends a rolling reference window or sub-sampling the stream at session-level boundaries — measures it reports but does not fully evaluate as a study.
• Residual blind spots. The constant sub-threshold adversary (≤12% mixing without ramp) defeats both KS and the scan martingale, and the MMD channel was only evaluated post-factorial on a representative subset, leaving the embedding-based detection path less characterized than the score-based one.

Target Audience

Machine learning safety engineers and researchers who deploy or evaluate safety classifiers, particularly those responsible for monitoring in production where labels do not arrive in real time. It is also relevant to researchers working on conformal prediction under covariate shift, sequential testing and change detection, and adversarial robustness of guardrail models. Practitioners who use reasoning models as automated judges or evaluators should read the reasoning-token budget section regardless of their interest in the rest

Authors’ abstract

We present an online monitoring system for distributional shift in deployed safety classifiers, using calibrated sequential statistics to detect when a classifier has moved out of distribution. Upon detection, a conformal abstention layer adapts decision thresholds to recover a target error rate epsilon=0.1. In a pre-registered factorial evaluation (4 classifiers x 5 shift conditions x 20 seeds x 2 window sizes, 800 cells), the system achieves 86.6% valid detection (693/800, 95% CI [84.1%, 88.8%]) with mean latency of 39.5 steps. Detection holds across three ground-truth regimes: synthetic onset (86.6%), real temporal jailbreaks (85%, 17/20), and GCG adversarial attacks. Weighted conformal prediction recovers up to 39 pp of lost coverage for DeBERTa (ESS=46/300) but collapses for all other classifiers (ESS~300): logistic density ratio estimation achieves perfect source/target separability in high-dimensional embedding spaces, clipping all importance weights to the floor. DeBERTa shows a gradient from effective correction (paraphrase, ESS=46) to near-total collapse (adversarial suffix, ESS=206). PCA to 32 dimensions breaks the collapse, recovering 33 pp for Llama Guard and 21 pp for ShieldGemma. Variance decomposition reveals classifier (eta^2=0.243), shift type (eta^2=0.237), and their interaction (eta^2=0.185) all contribute substantially to detection latency variance (all p&lt;0.001), indicating per-classifier monitoring profiles are necessary.