The Pulse
New risk benchmark finds newer AI models tend toward neutrality
Researchers tested how language models choose between certain and uncertain outcomes across different contexts. They found newer models generally made more stable, consistent and risk-neutral choices, with exceptions.

A new set of tests measures how language models choose between certain outcomes and uncertain bets, and whether those choices change with the task. In a preprint, researchers report that newer models generally made more stable, consistent and risk-neutral decisions than older ones, although some models were exceptions.
Source arxiv.org
The paper was submitted on Nov. 12 by Luhan Mikaelson, Robert Adragna, Emmett Brosowsky and Hayley Clatterbuck of Rethink Priorities. Its Risk Toolbox combines evaluation protocols with an analysis pipeline for building profiles of model risk behavior.
Differences between models
The main evaluation covers nine models: Claude Opus 5, Claude Sonnet 5, GPT-5.6, Gemini 3.8 Flash, Llama 4 Maverick, Qwen3-8B, DeepSeek V4 Pro, Kimi K3 and GLM-5.2. The researchers also examined older models in a generational case study.
The researchers found that newer models were more likely to choose according to expected value, which they say appears related to their greater stability and consistency. Many state-of-the-art models treated scenarios largely as numerical prospects, with little sensitivity to details such as the goods at stake or legal constraints.
Opus 5 was an exception among the state-of-the-art models, showing more risk aversion and sensitivity to context. Opus 5.5 showed more extreme risk aversion and high rates of refusal, the paper reports. The authors argue that the differences suggest a shift toward risk neutrality is not an inevitable consequence of scaling.
Choices under uncertainty
The tests adapt methods used in behavioral economics. Models face a series of choices between a certain outcome and a riskier bet; each risky bet offers a 0.5 probability of a lower outcome and a 0.5 probability of a higher one.
The tests estimate a certainty equivalent: the point at which a model is indifferent between the certain and risky options. A risk-averse model accepts a certain payoff below the expected payoff of the gamble, while a risk-seeking model chooses the gamble even when its expected payoff is lower than the certain option.
To measure stability, the researchers varied details of hypothetical financial choices, including wording, the order of options and whether responses were multiple choice or free-form. For consistency, they made larger changes, such as varying the size and type of payoff, the beneficiary and whether choices appeared as abstract bets or realistic vignettes.
Profiles and limits
The profiles describe a model’s average position relative to risk neutrality, its sensitivity to different domains and whether it switches sharply or gradually between choices. An initial study using numerical gambles over dollars and tokens provided a baseline for examining how added context changed behavior.
The authors caution that consistency is not always desirable: different risk attitudes may be appropriate in different contexts. The paper also identifies limits to the method, including its focus on bets with even odds; a fuller assessment would examine additional framing and probability variations.
Sources
Related stories
- Ai2’s XYBench tests whether AI answers address users’ real goals
- Sakana AI researchers release Japanese finance benchmark in preprint
- Stanford Study Finds 44% of Tested LLM Claims Expired
- BEAKER Preprint Finds Multimodal Models Struggle to Ground Actions in Chemical Labs
- Study finds swearing in a minority of AI models' benchmark runs