DeepSeek's Open Model Nears OpenAI's Restricted Cyber Model on Network Attack Test

Vals' CyberOffense benchmark measures whether AI agents can break into realistic enterprise networks without human help. It ranked GPT-6 Sol first, run through OpenAI's restricted Daybreak programme, with about 81% of objectives completed across two test networks. DeepSeek V4.1 Flash 1017 came second with about 73%. On one of the two networks, the DeepSeek model did as well as the OpenAI model for under $3 a run, against about $22 for GPT-6 Sol Daybreak, Vals said. It added that open-weight models ran at about a tenth of the cost.

The two models reach users in very different ways. OpenAI offers GPT-6 Sol for cyber work only through Daybreak, a programme with controlled access. DeepSeek's model weights can be downloaded and run by anyone, with no provider able to watch how they are used. Results like these put a number on how close freely available tools are to the most capable models built for offensive and defensive security.

The benchmark uses two simulated networks. Nova is a single large Windows network made up of several connected Active Directory domains, with 21 objectives. Lighthouse has 31 objectives spread over two separate corporate networks, and the second can only be reached using information found inside the first.

On Nova, DeepSeek V4.1 Flash completed an average of 16.3 objectives over three runs, with a range of 15 to 18. GPT-6 Sol Daybreak averaged 15.7, with a range of 12 to 19. On Lighthouse, GPT-6 Sol Daybreak completed 27 objectives and DeepSeek 21. The overall score gives the two networks equal weight.

The results among other models were uneven. Meta Wonder completed 28 Lighthouse objectives, more than any other model, but only 7 on Nova. Moonshot's Kimi K3.1 Pro scored 13 and 20, xAI's Grok 4.7 scored 12 and 16, Kimi K3 scored 9 and 10, and GLM 5.3 Flash completed 5 on each network.

How the Test Was Run

Vals ran each model in the open-source Pi agent loop on an isolated Kali Linux machine that could reach only the model provider and the target network. Every run started on a fresh copy of the network, with a 48-hour time limit and a budget of 500 million tokens, counting input, output, cached input and reasoning tokens.

Objectives were checked against hidden markers in a private answer key, matched to recorded command output. A model's own claims of success earned no credit. Vals said it reviewed every run and found no access to the answer key and no shortcuts.

Limits of the Results

The firm noted several limits. Grok 4.7, Kimi K3 and GLM 5.3 Flash stopped on their own before their budgets ran out, so their scores show where they got stuck, not where compute ran out. DeepSeek completed nothing in its first 25 million tokens and made most of its progress after 100 million. According to Vals, that means tests with tighter token caps would understate the model.

The page was last updated on October 23. Only GPT-6 Sol Daybreak and DeepSeek V4.1 Flash were run three times on Nova, and the other figures are single runs. No model completed the final Lighthouse objectives, and only one run reached the final stage of Nova.