Collinear AI Opens CWE-bench v2 Security Benchmark, With Gemini 4 Argon in Narrow Lead

Each task gives an agent a checkout of a real open-source repository and a short instruction. The agent has to locate and fix a security weakness without breaking the code's existing behaviour. The 100 public tasks span 91 different entries in the Common Weakness Enumeration (CWE), the industry catalogue of software vulnerability types. Collinear says the aim is breadth, arguing that a defender has to anticipate every route an attacker might take, so recognising a familiar pattern is not enough to score well.

Scoring is fully programmatic, which Collinear says makes results deterministic and reproducible at no extra cost. The company says the automated checks were refined with panels of language-model judges and that every task was also reviewed by hand. Among the tests applied to each task: the instruction must read like a realistic request rather than an exam, the starter code must fail and the reference fix pass, original functionality must survive the fix, and the verifiers must reject memorised solutions such as the known upstream patch. A separate held-out set is kept private so progress can be tracked on tasks no model has seen; developers who want to be scored on it are asked to contact the company.

How The Models Scored

Collinear evaluated 11 models. Google's Gemini 4 Argon leads with 67.8% pass@1 and 81.0% pass@4, just ahead of Anthropic's Claude Opus 5.5 at 67.3% pass@1 and 78.0% pass@4. Pass@1 measures success on a single attempt; pass@4 counts a task as solved if any of four attempts succeeds.

Ninety-five of the 100 tasks were solved by at least one model, which Collinear presents as evidence that the set is solvable. But the leading model still failed 19% of tasks on every attempt, and the company says the long tail remains far from solved.

Cost separates the field further. According to Collinear, three models form the cost-performance frontier: DeepSeek V4.1 Flash reaches 57.0% pass@1 at $0.04 per rollout, Claude Opus 5.5 adds about ten points at $1.83, and Gemini 4 Argon edges ahead at $3.92.

Open Tooling

The public set, grader and harness are open. Each task is packaged as a self-contained Harbor task, combining a Docker environment, an instruction and a deterministic verifier, so the benchmark can be run directly from the Hugging Face repository. Access to the dataset is gated behind accepted conditions. Results for open-weight models appear on the dataset page leaderboard, which labs and the community update through pull requests, while a separate leaderboard Space includes API-only models such as Gemini, Claude and GPT.

The tasks are also tagged as reinforcement learning environments, meaning the same setups and verifiers can supply rewards to train defensive agents, not only to evaluate them.

Why It Matters

Security teams are increasingly being asked whether AI coding agents can be trusted to patch vulnerabilities, and whether the same capability raises offensive risk. Public, reproducible benchmarks with private held-out sets give buyers and regulators a firmer basis for comparison than vendor claims. The narrow gap at the top, and the large price differences between models with similar scores, suggest enterprises choosing tools for automated remediation will need to weigh cost per fix alongside headline accuracy.