Open Weight Models Are Capable Autonomous Network Attackers
We ran three of the latest open weight models (Kimi K3, Qwen 3.8 Max, and GLM 5.2) on our cyber range benchmark MHBench and found a substantial increase in capability compared to the open weight models tested in the original Incalmo paper. Qwen 3.8 Max leads in performance across all harnesses on our benchmark. Qwen 3.8 equipped with the Incalmo harness had the highest overall performance.
Each cell holds the mean percentage of goals achieved for that model/harness pair, averaged over every environment and trial. Qwen 3.8 Max leads on all three.
Why open weight models?
Closed-weight frontier models have demonstrated increasingly strong offensive cyber capabilities, as shown by the gamut of cyber benchmarks like CyBench, CyberGym, and Incalmo’s MHBench. This has prompted a question we often hear from security practitioners and policy experts: can open-weight models do the same?
How did we test them?
We tested Kimi K3, Qwen 3.8 Max, and GLM 5.2 as autonomous offensive security agents, using 3 different agentic harnesses.
| Harness | Agent structure | Interface to the LLM | Execution mechanism |
|---|---|---|---|
| Incalmo | Single agent | High-level action library (LateralMove, ExfiltrateData, etc.) | High level actions are translated into a series of bash commands, executed via C2 server/agents |
| Artemis | Multi-agent | Supervisor that spawns dynamically-prompted sub-agents | Shell (via spawned sub-agents), coordinated through a shared note/TODO system |
| Bash Shell (baseline) | Single agent | One tool: a bash shell | Commands are executed directly through shell, with no higher-level abstraction |
These agents are evaluated on the MHBench cyber ranges, a set of network environments each with at least one multi-stage exploit. Further detail on the construction and scoring criteria of each of these cyber ranges can be found here.
What did we find?
When we first ran experiments with open weight models for Incalmo a year ago, we observed that “[Open weight] models do not follow instructions and are unable to execute shell commands correctly.” Open weight models have come a long way from when we first evaluated them for offensive cyber tasks.
We ran three of the latest open weight models on our cyber ranges (Kimi K3, Qwen 3.8 Max, and GLM 5.2). Qwen 3.8 Max currently leads on our cyber range benchmark MHBench, and with the Incalmo harness we see dramatic performance improvements and reduced cost.
Each point is the average goals achieved for one model/harness/environment experiment setup. Each highlighted region spans one standard deviation around its group's mean cost and mean goals achieved. Results using the Incalmo harness are highlighted with an outline for their higher performance and lower cost.
There is significant variability in all 3 of the open weight models’ performance across environments with the baseline shell harness, as well as in the cost of a run.
With the Incalmo harness, we see the performance of these runs be significantly higher, and the cost of each run is on average lower than both the baseline shell harness and the multi-agent pentesting harness ARTEMIS.
Deep Dive
To make these results easier to explore, the heatmap below breaks performance down by model, environment, and harness.
Each cell below is one model/environment pair, colored by percent of goals accomplished for the harness selected in the dropdown. Switch the statistic dropdown to see the mean, median, best trial, worst trial, or standard deviation across each cell’s trials instead of the default mean.
We also provide the cost for these experiments, in the interest of transparency.
To analyze even more dimensions of this data, check out the raw dataset here!
What does this mean for security practitioners?
Open-weight models are now a practical alternative for red team and penetration testing assistance when frontier models refuse offensive-security tasks. On-prem hosted open-weight models also let organizations keep sensitive network data private while integrating agentic workflows into pentesting.
What does this mean for security policy?
We now have visibility into the abilities of cyber-offensive capable models without provider-enforced guardrails. Open-weight models are already extremely capable at executing real, multi-stage network attacks. Sophisticated offensive security capabilities will become more widespread, beyond the control of centralized model providers.
What’s next?
Open weight models have proven themselves capable of complex offensive security tasks in complex network environments, and we are excited to see more innovation leveraging the specific advantages of open weight models for applications in this space, such as penetration testing.
Considering the success of open weight models in our benchmark set, we have already started exploring how we can make these challenges more discriminatory to assess the limits of open weight models’ offensive capabilities, as well as their defensive capabilities.
For more information on the work we’re doing and the open source systems we publish to enable it, take a look at our Github!