The Pulse
OpenAI Details Monitors Trained on Models’ Internal Activity
OpenAI says activation classifiers monitor internal model activity alongside text classifiers as part of its safety system. The probes can assess response prefixes as they are generated, but must be retrained for each new pretrained base mo
AI.info Team · Written with AI · fact-checked ·
Text checks meet internal signals
OpenAI’s existing text classifiers run on user prompts and assistant completions; its activation classifiers instead read patterns inside the model as it produces a response. The company describes the two approaches as layers in a “defense in depth” safety system, not as substitutes for one another. Starting from GPT-5.6, OpenAI says it added activation classifiers to the stack for all assistant completions. The company reports deploying probes on internal and external traffic for its most capable models.
OpenAI says probes can make streaming decisions and block a policy-violating model turn midway. The company also says probes are more efficient to run and more robust to jailbreaks than significantly bigger text classifiers. Its post cautions that the figures in the report come from internal experiments and may not represent classifiers in deployment.
Each probe is tied to its model
A probe takes a vector of residual-stream activations—typically from a layer around the middle of the model—and produces logits for a binary classification task, such as whether a completion violates a policy. During inference, the probe computes a score for each message prefix as the assistant response is sampled. This allows it to flag a message as soon as its score crosses a threshold.
The design also creates a constraint: probes depend on the activations of the model they monitor. OpenAI says a probe trained on one model’s activations will generally behave differently on another model with different weight initialization, so it needs to be retrained for each new pretrained base model. Even within a model family, performance can decline as weights move away from the checkpoint used to train the probe, prompting periodic retraining.
Training uses labeled conversations and repeated trials
OpenAI trains probes on activations collected from a mix of synthetic and de-identified conversations. The company typically uses up to 1 million training messages and says classification improvements continue up to that scale. Its most capable reasoning models supply the labels, acting as judges of whether examples fit the categories being monitored.
Probe performance varies with random initialization, according to the post, so OpenAI typically trains at least 32 initializations for each hyperparameter configuration. The team selects a seed using a validation dataset, then checks the chosen configuration against a separate held-out evaluation set. Training uses AdamW for fewer than five epochs, and the company says it can train tens of probes together on an accelerator in nearly the time required for one by vectorizing and fusing the probe weights during training.
Streaming can stop a response and its tools
OpenAI’s probes run directly on the inference engine and return one logit per sampled token. The system logs classifier logits jointly with each sampled token. Whenever a logit is above a threshold, it stops streaming the completion to the user and escalates the message to a reasoning model for final judgment. Tool calls are also blocked until the call is marked safe.
OpenAI researchers Alex Makelov, Xiaojun Xu, Sundaram Ananthanarayanan, Ethan Mcdonald and Tom Dupre la Tour, the authors, write:
Probes run in a streaming fashion, directly on the inference engine (keeping only a minimal buffer for token pooling), and return one logit per sampled token.
For deployment decisions, the team measures recall on a small, carefully labeled dataset and separately measures the escalation rate on a larger dataset meant to represent the intended use. It tests resistance to jailbreaks through automated and manual red-teaming. OpenAI says its experiments found probes overall more robust to jailbreaks than content-based classifiers, while warning that real-world attackers can adapt to signals from the safety system.
Better results do not remove the limits
The authors report that probe performance improves with the capability of the model being monitored, and that performance keeps increasing on frontier models. They also find excellent transfer across many post-training checkpoints, but more limited transfer across pretraining checkpoints. Those results sit alongside the operational limits the post describes: probes cover only labeled, measurable forms of undesirable activity, and a probe trained on one base model cannot simply be carried over to a different one.