Last month, we published a blog post series (Part 1, Part 2) about EthiBench, our evaluation protocol for realistic evaluation of Pentesting Agents. The results in those posts came from experiments we ran a few months ago, so we thought you guys deserved some updated results using recent frontier models.
Setting things up
Hackian’s Pentesting Agent is not a “one LLM for everything” type of harness. It is built from many components that work together towards a specific objective: finding valid vulnerabilities. This includes planning, orchestration, reconnaissance, exploitation, validation, guardrailing, summarization, and more. Some of these components can be isolatedly tested and specifically optimized for their individual tasks. Others can only be properly assessed once they are integrated into the full environment.
During our time researching this field, we’ve noticed that not every LLM is equally good at every task. In particular, there is a fundamental distinction between a good Strategizer (responsible for planning and orchestration) and a good Executor (responsible for reconnaissance and exploitation). These are the two roles we’ll focus on here, studying how different frontier LLMs behave when incorporated into Hackian’s harness.
We’ll evaluate 6 different frontier models: Claude Opus 5, Claude Sonnet 5, Deepseek V4 Pro, GPT 5.6 Sol, GPT 5.6 Terra, Kimi K3 and Mistral Medium 3.5.
Can they strategize and execute?
Let’s start by seeing which models can do both. The plot below shows both average and cumulative results across 3 pentesting executions on EthiBench. Average results are computed by averaging the metrics from each individual run, with the standard deviation shown as well. Cumulative results are obtained by aggregating the findings from all 3 runs into a single combined result.
Average and cumulative F1, Recall, and Precision across 7 frontier LLMs. Claude Opus 5 leads overall. DeepSeek V4 Pro trades Precision for Recall.Looking at the average results, Claude Opus 5 has the overall lead, combining a very strong Precision (~95%) with the highest Recall, resulting in the best F1 score. DeepSeek V4 Pro is an interesting outlier: it achieves a very good Recall, but at the cost of much lower Precision. In other words, it finds many vulnerabilities, but also reports a lot of false positives. The GPT 5.6 Terra model sits at the bottom, mainly dragged down by poor Recall.
The cumulative results amplify some of these trends. High-precision models like Claude Opus 5 benefit the most from multiple runs, with its F1 jumping from ~80% to ~90%. Low-precision models gain less – with DeepSeek V4 Pro actually decreasing its F1 in this setting – as additional runs also introduce more noise. Sonnet 5 and GPT 5.6 Sol are interesting cases: their cumulative F1 jumps significantly compared to their average result – due to a significant raise in Recall – hinting at a higher degree of stochasticity across runs.
As we said before, not all LLMs can do both. So, does the fact that the highly regarded GPT 5.6 models don't perform well in this setup mean they aren't suited for pentesting? Not necessarily. Our experience with previous OpenAI models actually shows that they can be great Executors, but often struggle as Strategizers. So let’s see how these models perform when backed by a good Strategizer.
Let’s focus on execution now
For the next experiment, we’ll keep Claude Opus 5 as the Strategizer and test all the other models as Executors.
Average and cumulative metrics with Claude Opus 5 fixed as Strategizer across 5 Executor models. Performance gaps narrow significantly compared to standalone results, confirming that strategy, not execution, was the bottleneck.With Claude Opus 5 handling the strategy, the performance gap between models shrinks dramatically. GPT 5.6 Terra shows the biggest improvement, confirming that it was being held back by poor strategy rather than poor execution. In the cumulative results, pairings such as Claude Opus 5 + Kimi K3 and Claude Opus 5 + GPT 5.6 Terra stand out, approaching the performance of Opus 5 running both roles.
None of these pairings actually beat Claude Opus 5 doing both jobs alone. So why bother splitting the roles? Because performance isn't the only axis that matters – cost does too.
Let’s talk about cost-effectiveness
Cumulative F1 vs. relative cost across all setups. Claude Opus 5 + Kimi K3 and Claude Opus 5 + GPT 5.6 Terra deliver near-top F1 at roughly half the cost of Claude Opus 5 alone.This is where the Strategizer + Executor split really pays off.
Claude Opus 5 alone sits in the top-right corner: best F1, but also very close to the highest cost. Pairing it with Kimi K3 or GPT 5.6 Terra gets you within a few F1 points of the best performance at roughly half the cost. Meanwhile, Kimi K3 and Claude Sonnet 5 running solo are the most cost-effective options, achieving respectable F1 scores at just 25-30% of the relative cost.
The bottom line: a smart Strategizer + Executor split can get you 95%+ of the top performance at roughly half the cost.
Wrapping up
The best pentesting agent isn't necessarily the one powered by the most capable LLM. It’s the one that puts the right model in the right role.
Separating planning from execution is not just architecturally cleaner: different models can excel at different parts of the pentesting process, and a strong Strategizer can make a huge difference in how efficiently those capabilities are put to use.
Strategy is key.
By the way, these experiments were evaluated using an updated version of the EthiBench ground truth, which we’ll be publishing soon, so keep an eye out for our upcoming updates. Also, EthiBench is open-source, and we’d love your help making it better! Check out the repository and contribute here: https://github.com/ethiack/ethibench.