Best AI Coding Agent Fully Completes Only 26.4% of Real User Requests in New Benchmark

The benchmark, Co-Design Bench, was posted to arXiv on October 26 in a paper titled "Co-Design Bench: Multimodal Collaborative Coding Evaluation for Agentic Models". It tests whether an agent can take an existing application and change it the way a user asked. Requests can include annotated screenshots, images, audio recordings, video or documents. The tech publication The Decoder described the authors as researchers from Alibaba and several Asian universities.

The gap between the two figures is the main finding for companies paying for these tools. Agents usually do most of a job and leave something undone, so a person still has to check and fix the result. GPT-6 Astra, the best of the ten model endpoints tested, satisfied 69.9% of the acceptance criteria per task on average, but delivered a fully correct result in only about one task in four.

The authors built the test because older coding benchmarks have stopped telling models apart. When agents can run, check and repair their own code, they score above 99% on several established tests, according to The Decoder's account of the paper.

Co-Design Bench has 273 tasks split into two tracks, Multimodal Editing and Repository Editing. The tasks come from real coding sessions. The researchers favoured cases where users had to rephrase a request several times or roll back changes, so the set is weighted towards jobs that had already caused agents trouble. They range from small edits, such as reading an amount from a receipt and updating every total that depends on it, to overhauls of whole projects.

Grading is strict. The answers are checked against hidden acceptance criteria written by experts, using what the paper calls a judge-then-refute process. One AI judge tests the finished app in a browser. A second re-checks every pass and looks for mock-ups or hard-coded results. A feature scores only if it actually works.

Claude Opus 5 and Qwen3.8-Max ranked behind GPT-6 Astra, and Grok 4.6 came last, The Decoder reported. On larger overhauls, the much smaller Qwen3.8-27B rose to second place, though the authors did not treat this as a general advantage for that model.

Every endpoint did worse as tasks mixed more kinds of change. In the repository track, edits that combined three or four types of change, across layout, design, interaction and content, scored lower than those involving one or two. Real projects usually mix them.

More compute did not close the gap. GLM-5.2 used nearly six times as many tokens as GPT-6 Astra on smaller changes and still scored significantly worse.

The researchers also found that the harness, the software through which an agent reads, writes and runs code, matters as much as the model. The same Qwen3.8-27B scored 75% or 56% on larger overhauls depending on the harness. Which of two models came out ahead also changed with the setup. The authors recommend that results always name both the model and the environment. For buyers, that means testing a coding agent and its harness together rather than relying on model leaderboards.

The benchmark has limits. Each task uses one fixed request, so it does not capture the follow-up questions and corrections of a real session, and the AI judges can make mistakes. Developers can now run their own agents against the 273 tasks to see how wide that gap is in their own setup.