Ai2's StateBench Finds Frontier Models Lose Track of Changing Facts in Long Documents

Ai2 published the work on Hugging Face along with a technical report, the dataset and the code.

What The Benchmark Measures

State tracking means following how information changes over time. Examples include reading code where a variable is assigned and then reassigned, or following a story in which characters move around and objects change hands. Ai2 argues that these problems vary a lot in difficulty. Some only require finding the most recent update. Others require following a chain of changes, where one missed step leads to a wrong answer.

StateBench sorts problems into four levels, described in terms of variable assignment. At the easiest level, called r-trivial, a value never changes. At the aperiodic level, values can be overwritten, so the model needs the latest one. At the soluble level, each update builds on the current value, as with a running score. The hardest level, general finite, adds assignment by reference: one variable can take on another's value, as when items are repeatedly swapped. The researchers say the levels come from Krohn-Rhodes theory, a branch of algebra that breaks state-tracking problems down into basic components. All four versions of each problem use the same number of variables and update points; only the type of assignment changes.

Architecture Versus Training

The work is partly a test of model design. Transformers, the architecture behind most large models, have known theoretical limits on the hardest state-tracking problems. Recurrent and hybrid designs such as Gated DeltaNet, which carry a running memory forward, should in principle handle all four levels.

Ai2 first tested open pretrained models whose training data and context lengths are documented, including its 7-billion-parameter Olmo 3 transformer and Olmo Hybrid, which adds Gated DeltaNet layers. Almost all of them did well on r-trivial problems, lost accuracy on aperiodic problems as inputs grew, and stayed at chance on the general finite level. How a problem was framed also mattered: some small models scored near the ceiling on one format of an easy problem but near chance when the same problem was framed as spreadsheet edits.

The researchers then trained 275-million-parameter models directly on the tasks, using sequences of up to 256 assignments and testing on up to 64,000. According to Ai2, the Gated DeltaNet and hybrid models generalised almost perfectly on the first three levels out to tens of thousands of assignments. The transformers learned the easier levels but degraded on longer sequences and never learned the hardest level. On the general finite level, a Gated DeltaNet variant with a more flexible memory update stayed near perfect out to about 10 times its longest training sequences. Ai2 concludes that the weak results from pretrained hybrid models reflect how they were trained rather than a limit of their architecture.

Frontier Models Fall Short

The four frontier models did well on the r-trivial and aperiodic levels. On soluble and general finite problems, accuracy fell sharply as inputs got longer, reaching 13% or lower on the longest general finite problems. Ai2's post does not give scores for each model. The researchers suggest one reason is that at the harder levels more steps affect the final answer, so a single missed update is more likely to cause an error.

For businesses using AI on long contracts, codebases, ledgers or case files, the results add to evidence that a long context window does not guarantee the model follows every change inside it. They also suggest that hybrid architectures may become more relevant for this kind of work.