The Pulse
StateBench Finds Language Models Lose Accuracy on Harder State-Tracking Tasks as Documents Grow
allenai introduced StateBench, a benchmark for measuring how language models track changing information across long documents. The authors report that accuracy on harder state-tracking problems fell as sequences grew longer, with results di
AI.info Team · Written with AI · fact-checked ·
allenai introduced StateBench to test how language models track information about entities as they change across long documents. The benchmark evaluates problems ranging from finding the latest update to following chains of updates where a missed step can change the answer. allenai says the tested frontier models still break down on harder forms of state tracking as inputs grow longer.
Four levels of state tracking
StateBench groups its tasks into four levels, each allowing more complex updates to a variable. In r-trivial problems, a value never changes, so the model needs to recover its initial value. Aperiodic problems allow reassignment, making the latest value the one that matters.
Soluble problems add self-referential updates, such as adding to a running score, so a model must carry each change forward. General finite problems add assignment by reference: one variable can take another variable’s current value, creating dependencies across updates. The general finite level can be illustrated by the S₅ state-tracking problem, in which successive swaps rearrange five objects.
The authors write that the four levels also represent distinctions rooted in Krohn–Rhodes theory, which describes how state-tracking problems can be decomposed into atomic primitives. To compare levels, the researchers derive three easier versions from a general finite sequence; the resulting tasks use the same number of variables and distinct values, with assignments at the same points in the sequence.
Almost all pretrained models lose accuracy on aperiodic problems as inputs grow
For its pretrained-model tests, allenai evaluated open models on the r-trivial, aperiodic and general finite levels. The set included two 7-billion-parameter models, Olmo 3, a transformer, and Olmo Hybrid, which adds Gated DeltaNet layers, as well as models with 275 million and 2.7 billion parameters.
All of those models were pretrained on the same data, and none had seen contexts longer than 64,000 tokens during training. allenai tested most of them on inputs beyond that length. Almost all performed well on r-trivial tasks, lost accuracy on aperiodic tasks as inputs grew, and stayed at chance on general finite problems.
Performance also varied with task format. The 275-million-parameter hybrid and Gated DeltaNet models scored near the ceiling on RULER-style r-trivial tasks but near chance when the same problems were framed as spreadsheet edits. That gap is one reason StateBench tests the same underlying problem in several formats.
Frontier-model accuracy falls sharply on soluble and general finite problems as inputs grow
allenai tested Gemini 3.8 Flash, Claude Opus 5.5, and GPT-6 Sol and Luna by asking them to read sequences of variable assignments and produce each variable’s final value. All four performed well on r-trivial and aperiodic problems. On soluble and general finite tasks, however, accuracy fell sharply as inputs grew longer, reaching 13% or lower on the longest general finite problems.
allenai identifies sensitivity as one likely factor in the harder tasks: more steps can affect the final answer, leaving less room for a missed update.
Models trained directly on StateBench
To test whether the harder tasks expose architectural limits or gaps in pretraining, allenai trained 275-million-parameter models from scratch on StateBench. The experiments included two transformer variants, two hybrids and two Gated DeltaNet variants with different memory-update constraints. The models trained on sequences of up to 256 assignments and were tested on sequences as long as 64,000 assignments.
Gated DeltaNet and hybrid models generalized almost perfectly on r-trivial, aperiodic and soluble problems through the longest sequences tested. Transformers learned the easier levels but degraded on longer sequences, and never learned general finite problems. On that hardest level, the Gated DeltaNet variant with a more flexible memory update stayed near perfect out to roughly 10 times the longest training sequences.
Open questions and the release
allenai says the exact placement of Gated DeltaNet with a more limited memory update in the hierarchy remains unresolved, including whether it can solve all soluble tasks and how numerical precision or combining it with attention layers affect its capabilities. StateBench also assumes a fixed number of variables, while real tasks often introduce new entities over time. The current framework does not yet cover that setting.