StateBench Tests Whether Language Models Can Track State, Not Just Retrieve It

The Allen Institute for AI has published StateBench, a benchmark built to measure something that standard long-context tests mostly skip: whether a language model can track entities whose values keep changing, rather than simply find a fact planted somewhere in a document. The work, described in a Hugging Face blog post and an accompanying technical report by Jackson Petty, William Merrill, and Ashish Sabharwal, frames the task as variable reassignment and spreadsheet-style edits playing out over long sequences.

StateBench organizes its problems into four tiers drawn from Krohn-Rhodes theory, an algebraic framework for decomposing finite-state automata into resets and permutations. The tiers run from r-trivial problems with immutable values, through aperiodic and soluble cases, up to general finite problems that include assignment by reference, the variable-swap logic that trips up naive tracking. Most benchmarks marketed as testing state tracking skip this kind of formal task taxonomy, which is what makes the tier structure here worth noting on its own.

The headline result is blunt. Frontier models, reported as Gemini, Claude Opus, and GPT-variant systems, handle the easy tiers well and fall to 13 percent accuracy or lower on the longest general finite problems, according to Ai2's writeup. That drop happens even though the underlying documents are not especially long by current context-window standards. The limit shows up in reasoning, not in memory.

Ai2 did not stop at testing existing models. The team trained six 275-million-parameter models from scratch, two transformers, two hybrid architectures, and two Gated DeltaNet variants, to separate architecture effects from whatever these models pick up during pretraining. The recurrent and hybrid models generalized well past their training length. One Gated DeltaNet variant stayed near-perfect at roughly ten times the sequence length it was trained on. The transformers never learned the hardest tier, even when trained directly on it.

A second finding complicates the picture for models that already exist in the wild. Pretrained open models, including Ai2's own Olmo 3 and an Olmo Hybrid variant, score well on state-tracking problems phrased as retrieval questions. The same models fall to near-chance accuracy when an identical logical task gets reframed as a spreadsheet edit. The paper also includes a proof that fixed-precision Gated DeltaNet with non-negative eigenvalues can solve all aperiodic tasks and some periodic ones, which lines up with the empirical split the authors observed between architectures.

That format sensitivity is the detail that matters most for anyone reading model cards instead of leaderboards. It suggests the pretrained models' failures trace to what kind of state-tracking language appeared in their training data, not to a hard ceiling on what the architecture can represent. A model that already handles the underlying logic in one phrasing and fails it in another has a data-coverage problem. That is a different problem than a model that cannot represent the logic at all.

StateBench did not appear out of nowhere. Merrill, Petty, and Sabharwal published The Illusion of State in State-Space Models at ICML 2024, arguing that state-space models such as S4 and Mamba sit in the same complexity class as transformers and cannot solve permutation composition. That paper also proposed a fix: a minimal change to linear SSMs, closer to the update rule used in DeltaNet, that would let a single layer learn permutation composition the way a recurrent network does.

StateBench reads as the large-scale test of that 2024 prediction. Gated DeltaNet and hybrid architectures, built on the corrective mechanism the ICML paper proposed, now outperform pure transformers on the hardest state-tracking tier, matching what the theory predicted. The benchmark adds a condition the theory paper did not specify: the advantage shows up when a model is trained directly on the task. It mostly disappears under ordinary pretraining, where the format-sensitivity results take over.

Related theoretical work from the same group, A Little Depth Goes a Long Way, ties transformer depth to state-tracking capability, showing that log-depth transformers can recognize regular languages that shallower ones cannot. The Gated DeltaNet architecture itself traces back to the December 2024 paper on Gated Delta Networks, which modified Mamba2 with a delta rule update.

StateBench joins a crowded field of recent state-tracking benchmarks. Microsoft released its own STATE-Bench in May 2026 focused on agent memory in enterprise tasks. Other entrants from 2025 and 2026 include LEAPS, StateMemBench, a benchmark called MCST presented at an ACL student workshop, and a Cardiff NLP benchmark built around tasks like LinearWorld and HandSwap. State tracking has become one of the more active evaluation categories in the field over the past two years. Ai2's entry distinguishes itself mainly through its algebraic task taxonomy and the from-scratch training comparison, not through being first.

One naming collision is worth flagging for anyone searching for this work later. A PyPI package also called statebench exists for fact-invalidation and timeline testing across 1,400 timelines. It has nothing to do with Ai2's benchmark and shares only the name.

Ai2's own writeup names a limitation that its result tables do not show: StateBench assumes a fixed, closed set of entities throughout each problem. It doesn't test what happens when new entities get introduced partway through a document, a scenario closer to how state actually accumulates in long real-world logs, contracts, or codebases. That gap marks where the next version of this benchmark would need to go.