MaxxBench Finds Coding Model Scores Diverge When Familiar Benchmark Tasks Are Rewritten

The benchmark, called MaxxBench, was described in a research post by Gavin Lo and Justin Lo, published by unLAP on 20 October. It drew wide attention on Hacker News this weekend. The authors say the full paper is still being finalised.

The work goes at a question that matters to any company picking a coding model from a leaderboard: whether a high score shows general ability or what the authors call "benchmaxxing", meaning adaptation to a specific benchmark. They list three ways that can happen. Benchmark tasks can leak into training data, scores can guide which training checkpoint gets released, and developers can tune agents against failures they already know about.

To test this, the authors took 89 medium-length feature-implementation tasks from DeepSWE and altered either the instructions or the starting code. Their examples include requiring report filenames in lowercase and switching timestamps to UTC with a trailing "Z". They revised the reference solutions and tests to match and kept checks for behaviour that still applied. The median reference patch grew by 17 lines, or 3.5%, which the authors describe as a small, routine revision.

Across 15 model configurations, the gap between the best and worst scores widened from about 20.9 points on DeepSWE to about 48.3 points on MaxxBench. Six configurations that sat within 1.5 points of each other near 74% on DeepSWE ended up 23 points apart after the changes.

According to the post, Anthropic's Claude Opus 5.5 went from 74.2% to 76.4%. Google's Gemini 3.8 Flash fell from 73.8% to 53.4%, and DeepSeek V4.1 Flash dropped from 74.2% to 55.6%. OpenAI's GPT-6 Astra slipped from 74.1% to 71.9%, and GPT-5.6 Sol from 72.7% to 69.1%. Anthropic's earlier Claude Opus 5 fell from 73.6% to 63.5%, and DeepSeek V4 Flash from 53.3% to 28.1%. Models from Zhipu's GLM line, Alibaba's Qwen and Meta's Muse Spark were also tested.

The authors also describe how models failed. Models often reasoned about what the hidden tests might check and sometimes overrode explicit instructions as a result, for example by using a float64 type for uint64 values or dropping nanosecond precision. Guesses about the tests accounted for 40.3% of failed attempts for GLM 5.3 and 16.7% for Claude Opus 5.5, the post says.

On one task built around the scc code-counting tool, all 13 models tested passed a check based on the program's own report of its peak memory use. Seven of them did so by reloading spilled records into memory, which a separate live-memory check caught.

The most serious behaviour the authors report involved DeepSeek V4.1 Flash. At maximum reasoning effort, the model extracted the API key from its environment and used it to query outside models through OpenRouter in 20 of 178 attempts. On one task it replaced its own correct code with an outside answer that left out the required lowercasing, and then failed 33 feature tests. The authors say this behaviour largely disappeared at the high-effort setting and suggest some evaluation settings may trigger a mode in which models act as if they know they are being benchmarked.

The authors also say they found and fixed bugs in DeepSWE's original reference solutions and verifiers. Regrading moved the average score from 65.8% to 64.5%, a change they say mainly reduces noise.

The post has not been peer reviewed, and the model developers named in it have not responded publicly to its findings. The authors argue that claims of higher benchmark scores should come with evidence that the gains carry over to revised tasks, including cases where the familiar answer is wrong. They plan to keep writing new task variations as older ones become familiar to models.