The Impact of Off-Policy Training Data on Probe Generalisation

AI summary

Overview

Research area: Interpretability and AI safety — specifically activation-based probing of large language models (LLMs) for behaviour monitoring, and how the way training data is generated affects a probe's ability to generalise.

Technical level: Intermediate. Some familiarity with linear classifiers, AUROC, and the idea of reading a model's internal activations will help, but the paper's core argument is conceptual and readable without deep mathematical background.

Scope: A systematic study of how four response-generation strategies (on-policy natural, on-policy incentivised, on-policy prompted, off-policy) affect probe generalisation across eight LLM behaviours, multiple probe architectures, and multiple models.

What This Paper Is About

Probes — classifiers trained on a language model's internal activations — are a promising way to detect unwanted behaviours such as deception, but natural labelled examples of such behaviours are rare and ambiguous, so researchers fall back on synthetic or off-policy data. The paper asks how much that choice of training-data generation strategy actually costs, and whether there is a cheap way to predict in advance which probes will fail when deployed on real, on-policy model behaviour.

Key Contributions

1. A systematic evaluation of four response strategies (on-policy natural, on-policy incentivised, on-policy prompted, and off-policy) across eight distinct behaviours, two probe architectures (linear and attention), and several LLMs, showing that off-policy data suffices for "text-evident" behaviours but harms generalisation for "text-ambiguous" ones.
2. A proposed diagnostic: success on on-policy incentivised data (where the model was indirectly coerced into the behaviour) correlates strongly with success on on-policy natural data, giving a proxy test when natural data is unavailable.
3. An empirical demonstration that domain shift (changing the input dataset) degrades probe performance more than response-strategy shift (changing how responses were generated), yielding a practical heuristic for choosing training data.
4. A prediction, based on these results, that current deception and sandbagging probes trained on existing datasets may not generalise to real-world monitoring scenarios.

Main Findings

• Results depend heavily on the behaviour. For same-domain training, the refusal, lists, metaphors, and science probes generalised successfully to on-policy natural data under all response strategies, while sycophancy and deferral-to-authority probes trained on on-policy prompted or off-policy data performed worse.

• Poor cross-strategy scores reflect distribution shift, not task difficulty. All probes achieved more than 0.7 AUROC (for deferral-to-authority) and more than 0.9 AUROC (for all others) when tested on the same response-strategy distribution they were trained on.

• Domain shift is the dominant problem. Shifting training domain degraded performance by 0.162 AUROC on average on the on-policy natural test set (p < 0.001), while response-strategy effects were small and non-significant (all p > 0.05). The reported Cohen's d values are 1.42 for domain shift versus 0.60 for response-strategy shift.

• A concrete heuristic follows. When same-domain on-policy data is unavailable, same-domain off-policy data (0.85 ± 0.039 AUROC on average) is better than different-domain on-policy data (0.74 ± 0.051 AUROC), while same-domain on-policy data scores 0.90 ± 0.019 AUROC.

• Text-ambiguous behaviours suffer most. In the OLS interaction model, domain-shift penalties were small for text-evident behaviours (metaphors −0.045, refusal −0.088, science −0.099) but far larger for deception (−0.416), sycophancy (−0.283), and deferral-to-authority (−0.222). Response-strategy effects were largely non-significant, with exceptions for lists × incentivised (−0.064, p < 0.05) and sandbagging × prompted (−0.111, p < 0.05).

• Incentivised test data is a useful proxy. The line of best fit comparing generalisation gaps on on-policy incentivised versus on-policy natural test sets achieves a Pearson coefficient of 0.793 for off-policy training. The equivalent correlation for prompted training data is weaker, at 0.639.

• Predicted failures for deception and sandbagging. Based on the incentivised proxy, deception (trading) and sandbagging (WMDP) probes trained on off-policy data are predicted not to generalise to on-policy natural test sets. Sandbagging (multichoice) had an AUROC below 0.75 even when trained on the same distribution as the test set, suggesting the behaviour is inherently hard to classify.

• Attention probes score higher than linear ones. For same-domain on-policy results on Llama, attention probes averaged 0.946 (±0.015 SEM) AUROC versus 0.903 (±0.019 SEM) for linear probes. Linear probes averaged 0.923 (±0.014 SEM) on Gemma and 0.901 (±0.023 SEM) on Ministral.

• Larger models can be more domain-sensitive. With Gemma-3-27B-it, on-policy natural scores clustered tightly, while different-domain scores for sycophancy and deferral-to-authority were noticeably lower.

• Mixing data helps only marginally. Other response strategies only approach on-policy natural performance when practically all natural training examples are added; adding only positive or only negative natural examples did not improve test performance. Training on two domains and testing on a third only slightly reduced the generalisation gap.

• Text-ambiguous behaviours resist LLM-as-judge labelling. GPT-5-Nano achieved near-random AUROC (0.518 and 0.493) on sycophancy test sets, while linear probes exceeded 0.9 AUROC.

Methodology in Plain English

The researchers pick a target language model and collect activations — the model's internal numerical states at some chosen layer — for prompts and responses. They build training sets for the same behaviour in four ways:

1. On-policy natural: the target model answers the original prompt freely. This is the baseline.
2. On-policy incentivised: the prompt is augmented with an indirect incentive that makes the behaviour instrumentally useful, and the target model responds.
3. On-policy prompted: the prompt is augmented with a direct instruction to exhibit the behaviour.
4. Off-policy: the response is written by a different model (or a human) and simply fed through the target model to extract activations.

For prompted and incentivised data, the added instruction is removed before activations are computed, so the probe cannot just learn to recognise the prompt. Labelling is done in several ways: counterfactual comparison (generating a control response and a nudged response, and labelling a change as the behaviour), comparing public versus private answers, the HarmBench classifier (a Llama-2-13B model fine-tuned to detect harmful compliance), and a GPT-5-Nano autograder that keeps only high-confidence (≥ 7) and strong-judgment (≥ 8 or ≤ 3) labels. Agreement with Claude-4.5-Haiku gave Cohen's Kappa scores of 0.71 to 0.86.

Datasets are balanced to equal numbers of positive and negative samples. For the first six behaviours, each strategy uses 3500 training, 500 validation, and 1000 test samples; for sandbagging and deception, 2500 training, 500 validation, and 500 test samples. Two probe architectures are trained: linear probes (scikit-learn logistic regression on sequence-averaged activations) and attention probes (trained with AdamW on binary cross-entropy). Evaluation uses AUROC with 95% confidence intervals from the analytical standard error formula of Hanley and McNeil (1982), always testing on on-policy natural data — except for deception and sandbagging, where on-policy incentivised data stands in as a proxy.

The primary activation model is Llama-3.2-3B-Instruct, with additional experiments on Gemma-3-27B-it, Ministral-8B-Instruct-2410, and Qwen3-30B-A3B-Instruct-2507 (the last excluding sycophancy and deferral-to-authority).

Why This Matters

Impact on research. The paper turns a common but under-examined practice — training probes on synthetic or externally-generated responses — into a measurable variable. It introduces a conceptual split between text-evident and text-ambiguous behaviours, and provides a testable proxy (on-policy incentivised data) for diagnosing generalisation risk before deployment. It also extends the preliminary single-behaviour comparison of Goldowsky-Dill et al. (2025) to eight behaviours and multiple architectures.

Real-world applications:

• Deployment monitoring: safety teams deciding whether a deception or sandbagging probe is trustworthy in production cannot assume it transfers from a benchmark dataset.
• Data-collection planning: when natural examples are scarce, the paper's heuristic (same-domain off-policy beats different-domain on-policy) directly informs how to spend labelling effort.
• Evaluating synthetic safety data: the finding that stated intent in chain-of-thought may not produce the same activation signature as genuine strategic reasoning warns against taking surface-level text similarity as evidence of a valid training signal.
• Probe pipeline hygiene: removing instruction prompts before extracting activations prevents probes from overfitting to the prompt rather than the behaviour, a simple implementation change with measurable benefit.

Industry relevance. Any organisation deploying activation-based monitors on LLMs — for content moderation, agent oversight, or red-teaming — faces the distribution-shift problem this paper quantifies. The results argue that monitoring tooling needs to handle domain shift, not just changes in how training data was elicited.

Future Directions

• Characterising the gap between off-policy and on-policy distributions with formal distributional distance measures, ideally with guarantees strong enough to inform deployment decisions.
• Testing whether the findings hold for other probe architectures, especially "deep probes" with many layers and non-linearities, as well as black-box monitors.
• Applying techniques from domain adaptation or invariant risk minimisation to improve probe robustness to distribution shift.
• Extending the experiments to much larger models (100B–1T parameters) and their future generations; the paper notes that SOTA LLMs sometimes reason in their own language, which may make on-policy natural data more important for monitoring reasoning traces.
• Investigating minimal contrastive pairs (as used by MacDiarmid et al., 2024) rather than naturalistic scenarios, and using more diverse multi-domain training distributions to reduce the domain-shift gap.

Target Audience

AI safety and interpretability researchers who train or evaluate probes; practitioners building production monitors for LLM deployments; and evaluation engineers deciding how to source labelled data for rare behaviours such as deception and sandbagging. Readers interested in distribution shift, robustness of classifiers over internal representations, or the methodological question of what makes synthetic safety data valid will also find it relevant.

Authors’ abstract

Probing has emerged as a promising method for monitoring large language models (LLMs), enabling cheap inference-time detection of concerning behaviours. However, natural examples of many behaviours are rare, forcing researchers to rely on synthetic or off-policy LLM responses for training probes. We systematically evaluate how off-policy data influences probe generalisation across eight distinct LLM behaviours. Testing linear and attention probes across multiple LLMs, we find that training data generation strategy can significantly affect probe performance, though the magnitude varies greatly by behaviour. The largest generalisation failures arise for behaviours defined by response ``intent'' (e.g., strategic deception) rather than text-level content (e.g., usage of lists). We then propose a useful test for predicting generalisation failures in cases where on-policy test data is unavailable: successful generalisation to incentivised data (where the model was coerced) strongly correlates with high performance against on-policy examples. Based on these results, we predict that current deception probes may fail to generalise to real monitoring scenarios. We find that off-policy data can yield more reliable probes than on-policy data from a sufficiently different setting. This underscores the need for better monitoring methods that handle all types of distribution shift.