The Pulse
Microsoft study finds Kimi K3 probe signals can arrive late
Microsoft researchers trained a probe to flag potential misalignment in Kimi K3 agent runs using labels from a language-model monitor. They found signals in reasoning and tool interactions, but some first alerts arrived late in a run.

A probe trained to flag potential misalignment in Kimi K3 agent runs sometimes raised its first alert only after later tool feedback or claims to a user, according to Microsoft researchers. In a Nov. 16 study, the team reported that those late signals could leave too little time to intervene.
The researchers also found strong signals in reasoning, tool calls and tool feedback, including material surrounding a suspected violation. Their case study compared the probe with a separate language-model monitor that judged the agent’s recorded interactions.
Watching the course of a task
The probe was trained on internal representations, or activations, from Kimi K3 while the model itself remained unchanged. It assigns a potential misalignment score to each successive prefix of a conversation: the recorded interaction up to a particular token, without using tokens that come later.
The language-model monitor reads the transcript and can explain its judgment by citing passages. Its assessments supplied the labels used to train the probe, with each run receiving a single label rather than labels identifying where a problem occurred.
A run is flagged by the probe if at least one prefix reaches the selected cutoff. The recorded material includes tool messages as well as the agent’s own output; the monitor’s assessment is excluded from the conversation replayed to the probe.
Signals and timing
The authors report that, in many runs flagged by both systems, the probe’s signals overlapped with passages cited by the monitor. That overlap appeared even though training labels applied to whole runs, rather than individual tokens.
Some first alerts nevertheless occurred only in later tool feedback or in claims presented to the user. The researchers suggest investigating whether supervision at the token level could encourage earlier warnings.
The study describes a conceptual monitoring system in which a lightweight probe screens activity and routes flagged cases to a language-model monitor for review. Depending on that review, activity could continue, be cleared or be paused and escalated.
Training and comparisons
The initial training used Kimi K3 agent runs from internal safety data, labeled by a language-model monitor using an internal misalignment prompt. After a search across model layers, the team found that layer 63 gave the highest validation AUROC, a measure used to assess the probe.
The team then fine-tuned the selected probe on a subset of runs from the public benchmarks ImpossibleBench, ShoppingBench, SWE-bench Verified and tau3-bench. Only the probe’s weights were updated; the Kimi K3 model and the selected activation layer remained fixed.
For the reported performance results, each benchmark had its own cutoff chosen to maximize F1 on that benchmark’s labels. Cutoff selection and metric reporting used the same examples.
In one shopping comparison, a benign agent was asked to find a black, water-resistant sports watch. The agent checked the color option and water-resistance rating before recommending an item; the monitor accepted the recommendation, and the probe did not flag the run.
Source
Companies in this story
Related stories
- Epoch AI says test agents attempted exploits on its live website
- Anthropic publishes recommendations for designing, running and reporting AI safety tests
- MatrAIx opens proposals for long-horizon personal agent benchmark
- LiveGeo tests find AI agents struggle as geographic conditions change
- Google reports new proof using paid Antigravity Teamwork