The Pulse
Ai2’s StateBench Finds Long-Context Tracking Still Trips Up Models
Ai2’s StateBench benchmark finds that frontier models struggle with harder state-tracking tasks as inputs grow, while recurrent and hybrid models generalize farther when trained directly on the tasks. The results vary by task difficulty and
AI.info Team · Written with AI · fact-checked ·
Recurrent and hybrid models can generalize much farther than transformers when trained directly on StateBench, Ai2 says, while frontier models still falter on harder tracking problems as inputs grow. The contrast is sharpest when a task requires carrying changes forward rather than locating the latest update. Ai2 introduced StateBench on November 3, 2026, as a benchmark for measuring how models track evolving information across long documents.
“Tracking the state of the world across a long context is necessary for understanding discourse, documents, and code.”
The paper’s authors, Jackson Petty, William Merrill and Ashish Sabharwal, write that sentence.
Four kinds of updates, from fixed values to swaps
StateBench divides state tracking into four levels, each allowing more complex changes to variables. R-trivial tasks use immutable assignments: a model must recover a value that never changes. Aperiodic tasks allow a variable to be reassigned, so the latest value replaces earlier ones.
Soluble tasks add self-referential updates, in which each new value depends on the current one, as with a running score. General finite tasks allow one variable to take another’s current value, creating dependencies across updates. Ai2 compares that hardest level to tracking how repeated swaps rearrange objects under five cups.
To make the levels comparable, Ai2 derives three simpler versions from a general finite sequence. The versions keep the same number of variables and distinct values, with assignments at the same points; only the assignment type changes. StateBench also presents the same underlying problems in multiple formats, because the format itself affected model scores.
Open pretrained models struggled on harder tasks
Ai2 tested open pretrained models on r-trivial, aperiodic and general finite tasks. The group included the 7-billion-parameter Olmo 3 transformer and Olmo Hybrid, which adds Gated DeltaNet layers, as well as smaller models with 275 million and 2.7 billion parameters. None had been pretrained on contexts longer than 64,000 tokens, and Ai2 tested most on inputs longer than that.
Almost all the pretrained models did well on r-trivial tasks, lost accuracy on aperiodic tasks as inputs grew, and performed at chance on general finite problems. Format mattered: the 275-million-parameter hybrid and Gated DeltaNet models scored near the ceiling on RULER-style r-trivial problems but near chance when the same tasks appeared as spreadsheet edits.
Frontier models fell to 13% or lower on the longest general finite problems
Ai2 also evaluated Gemini 3.8 Flash, Claude Opus 5.5, GPT-6 Sol and GPT-6 Luna. All four performed well on r-trivial and aperiodic problems, but their accuracy declined on soluble and general finite tasks as inputs lengthened. On the longest general finite problems, accuracy reached 13% or lower.
Direct training widened the gap between architectures
To test whether the weakness came from architecture or from pretraining, Ai2 trained 275-million-parameter models from scratch on StateBench. The set included two transformer variants, two hybrids and two Gated DeltaNet variants. Training sequences reached 256 assignments; evaluation sequences extended to 64,000 assignments, 250 times the longest training sequence.
As the test sequences grew, Gated DeltaNet and hybrid models generalized almost perfectly on r-trivial, aperiodic and soluble tasks out to tens of thousands of assignments. Transformers learned the easier levels but degraded on longer sequences, and did not learn general finite problems. On that hardest level, the Gated DeltaNet variant with a more flexible memory update stayed near perfect to roughly 10 times its longest training sequence.
The contrast also depended on training: Ai2 says directly trained recurrent and hybrid models learned levels that their pretrained counterparts could not. The transformers still failed on the hardest level after direct training.
StateBench still assumes a fixed set of variables
Ai2 says its proof for Gated DeltaNet with non-negative eigenvalues covers all aperiodic tasks and certain periodic tasks, including tracking whether a repeated input has occurred an even or odd number of times, under fixed-precision arithmetic with rounding. The architecture’s exact position in the hierarchy remains open, including whether it can solve all soluble tasks and how numerical precision or attention layers affect its abilities.
The benchmark also assumes a fixed number of variables, while real conversations can introduce new people and objects that a model must begin tracking. Ai2 says its current framework does not cover that situation. For now, StateBench isolates a narrower question: how well models follow changing state when the entities and update sequence are defined.