The Pulse
GW Researchers Model When AI Output Tips Toward Harm
George Washington University researchers propose a formula to estimate when an AI conversation may shift from desirable to harmful output. Their study finds evidence across seven open-weight models but says more testing is needed to predict
AI.info Team · Written with AI ·
George Washington University researchers say a mathematical formula can help predict when an AI conversation may shift from desirable answers to harmful ones. Their study, published October 8 in Patterns, presents evidence across seven open-weight models; the authors also draw a boundary around what they have shown: their tests support predictions of immediate versus delayed tipping, not a calibrated forecast of the exact moment it will happen.
That distinction separates a proposed early-warning method from a dependable safety monitor. The authors say the framework could help assess offline AI systems, where cloud-based checks are unavailable, but their paper calls for larger tests on unseen prompts before the timing estimates can be treated as reliable.
A formula built around competing answers
The paper focuses on transformer attention: the mechanism that weighs information from a prompt and the preceding conversation as a model generates a response. Johnson and Huo describe desirable and undesirable answers as competing “basins” in the model’s representation space. Their formula estimates how many desirable output tokens might appear before the system tips toward an undesirable one, based on the conversation context and its alignment with those competing answers.
“We derive an approximate closed-form estimate for the dynamical tipping point denoted as n∗ here,” the authors write. The estimate is approximate because the researchers simplify the model’s complex, multi-layer interactions into an effective attention state, then study how context and output representations compete.
Conversation order changes the result
In examples using GPT-2, the researchers show that a response can turn undesirable immediately or follow a run of answers that appear acceptable. They also show cases in which the same question about topics such as vaccines, self-harm or hurting people receives different responses depending on earlier turns in the conversation.
The paper’s proposed mechanism gives that sequence a role: earlier exchanges help shape the context that influences what comes next. In the researchers’ framework, adding context more closely aligned with an undesirable answer can bring the predicted tipping point nearer, while context that pulls in the other direction can delay it.
Seven models, with a limit on the claim
The researchers tested predictions about immediate versus delayed tipping across seven open-weight models, ranging from 124 million to 12 billion parameters. Their examples of multi-turn behavior use the 124-million-parameter GPT-2 base model, which they describe as a research proxy for resource-constrained inference—not as a stand-in for current instruction-tuned systems. They say independent studies of production chatbots show patterns consistent with the proposed regimes, but those studies do not expose the models’ internal states and therefore do not directly test the formula’s specific tipping-point value.
The study also flags factors that could shift the numerical estimate, including the model’s layers, attention heads, decoding temperature and how researchers define desirable and undesirable output. The authors say their work tests whether tipping is immediate or delayed; it does not establish a precise, general-purpose countdown to harmful output.
Offline AI is the proposed use case
The researchers point to AI running on phones and other devices without internet access. Such systems may be used where users want to keep sensitive information local, but they cannot rely on real-time cloud monitoring or updates. A lightweight warning based on the conversation’s evolving context is a potential application of the theory, not a safety feature the paper says is already deployed.
GW Today highlighted the study on October 12, emphasizing the researchers’ interest in predicting shifts before harmful output appears. The next test described by the paper is more demanding: a larger benchmark that makes predictions in advance on unseen prompts, with fixed definitions of the competing answer categories, across a wider range of models and domains.