Stanford Study Finds 44% of Widely Cited LLM Research Findings Have Already Expired

The pair retested 50 well-known claims about LLM behaviour on 20 models from eight developers, released between July 2024 and September 2026. They found that 44 percent of the findings had "expired", meaning the original effect had largely disappeared. Another 34 percent had weakened significantly. Only 16 percent still held, and 6 percent had become stronger.

For the 22 findings where the researchers could estimate an expiry date, the average lifespan was 15.7 months. Sixteen percent of findings had expired after one year and 39 percent after two.

Weaknesses Fade, Tendencies Persist

Claims about things models cannot do expired fastest. Five of the nine reasoning-related findings had expired, and none had held up or strengthened. They include well-known failures: counting the letters in "strawberry", doing arithmetic in unfamiliar number systems, losing information placed in the middle of a long context, and dropping correct answers when users push back. The researchers suggest these errors have a clear right answer, which gives developers an obvious target to train against.

Findings about preferences and tendencies lasted longer. Of the 11 findings that held or strengthened, three concern models acting as evaluators. Models prefer their own answers, rate accepted and rejected academic papers similarly, and are swayed by spin in medical abstracts. Three more involve sycophancy toward users. A left-leaning political bias, near-identical answers to open-ended questions and uneven distributions when generating random numbers also persisted.

Social biases were mixed. Gender stereotypes and implicit association biases weakened. Bias against speakers of African American English reversed direction in the trend data.

Hallucinated Citations

In a case study, the researchers reran prompts from a 2023 study by Walters and Wilder, which found GPT-3.5 invented 55 percent of its scientific references. GPT-6.1 Sol fabricated none and Claude Sonnet 5.5 fabricated 1 percent. Qwen3.7-Flash, another 2026 model, still fabricated 69 percent.

The researchers warned against treating this as an all-clear. On two-hop questions, where a model must combine two facts it knows, Claude Sonnet 5.5 still failed in about 10 percent of cases where it answered both individual questions correctly. The original paper reported 43 percent for ChatGPT.

Models released in the same window also differed widely. Among those released between July and September 2026, the number of expired findings ranged from 11 for DeepSeek V4 Pro to 30 for GPT-6.1 Sol. Release date, developer and model size did not reliably predict the pattern, the study found.

What It Means for Buyers and Policymakers

Queen and Zou argue that empirical findings about LLMs should be treated as observations about specific models at a specific time, not as general truths. They recommend that papers record the model, version and date, and publish prompts and analysis code so the results can be retested on later models.

The point matters beyond academia. Companies choosing models, and regulators writing rules, often lean on published research about model failures and biases. The study suggests evidence that is two years old may describe systems that no longer behave that way. Wide gaps between models from the same period also mean results from one model say little about another. Some weaknesses, notably the tendency to fabricate references, depend heavily on which model is in use.