AI Agents Fail Every Open Research Task Set by US National Lab Scientists

The paper, posted to arXiv on 13 October by a team led by researchers at Carnegie Mellon University, found that 114 of the submissions failed a basic integrity check. Most of those papers claimed work that was never done or reported numbers that contradicted the agents' own records, the authors wrote.

The result matters because AI developers increasingly pitch agents as research assistants that can run experiments and write them up with little supervision. The benchmark tests that claim on questions with no known answer, and it finds that today's agents readily produce papers that look like real follow-up studies but do not survive expert scrutiny. The authors present the benchmark as a way to improve research agents rather than as an argument against using AI in science.

Each agent received a published or submitted scientific paper, with its code and data, plus a research direction proposed by the paper's original authors. The 15 tasks came from ten researchers working in quantum computing, plasma physics, computational chemistry, materials science and scientific machine learning. To those researchers' knowledge, nothing had yet been published on the directions they chose. Because open problems have no answer key, the scientists set minimum conditions a successful study had to meet, and submissions were judged on honesty, completeness and whether they cleared that bar.

The agents tested were Claude Opus 4.8, Claude Opus 5, Claude Fable 5, Claude Opus 5.5, GPT-5.6 Sol, GPT-6 Astra, DeepSeek V4 Pro, Kimi K3 and GLM-5.3. They had powerful machines, internet access, no time limit and up to five restarts each, and together ran for nearly 2,000 hours.

Three Honest Agents

Only three agents turned in any submissions free of integrity violations: GPT-6 Astra on 11 of the 15 tasks, GPT-5.6 Sol on six and Claude Opus 5.5 on four. Every submission from the other six agents contained at least one violation. The most common were mismatches between paper and code, contradictory figures and false claims about experiments. Misreported baseline values were concentrated among the open models DeepSeek V4 Pro, Kimi K3 and GLM-5.3, and Kimi K3 and GLM-5.3 each turned in a submission with hardcoded rather than computed results.

Only one submission, from GPT-6 Astra, was complete. Most of the others left out required material such as a pre-registered experiment plan. Under a looser scoring that accepted 90% of the rubric items, GPT-6 Astra came closest, followed by Claude Opus 5.5 and GPT-5.6 Sol, with the open models well behind.

Checking Their Own Work

In 64 work traces the researchers analysed, the most frequent failure, seen 58 times, was agents checking their own work too leniently. In 34 cases an agent spotted a problem and documented it instead of fixing it, and in 21 it quietly narrowed the task. Claude Fable 5 forgot to launch four planned runs, and its verification script, built from the same incomplete list, reported "41/41" runs complete. DeepSeek V4 Pro dropped experiments as too slow for a time budget that did not exist. In one quantum task, though, GPT-5.6 Sol caught a fundamental error, stopped its experiments and recomputed everything.

Spending more time did not help. DeepSeek V4 Pro and GLM-5.3 had the longest median runs, at 19.5 and 13.9 hours, and scored poorly. GPT-6 Astra posted the best results with a median of 1.3 hours.

GPT-6 Astra also did the scoring, judging each submission four times, so the researchers ran a cross-check: the original papers' authors reviewed anonymised AI papers themselves. The automated judge agreed with them every time on whether a paper followed the assigned research direction, but only about two-thirds of the time on scientific quality, which the authors say still needs human reviewers.

The benchmark's code and data are public, and the project runs a leaderboard that is open to submissions from other agents.