Autonomous Offensive Security done right requires more than isolated capabilities. It depends on the seamless coordination of reconnaissance, asset enumeration, guardrails, security controls, scalable deterministic detection, and a wide range of operational concerns. In the AI era, though, one more component has become essential to any cutting-edge system: agentic AI. That is why a core part of Hackian, our comprehensive Autonomous Offensive Security system, is our Pentesting Agent.

Building such an agent, however, immediately raises a fundamental question: how do you know it is actually ready for real-world use? As is often the case with cutting-edge technology, advancing the state-of-the-art also means confronting new challenges. One of the most important is evaluating these systems in a way that reflects how they perform outside the lab. How can we reliably say that Agent A is better than Agent B for use in real applications, under real-world conditions? We looked into the state-of-the-art; we were not satisfied with the answers. The evaluation frameworks we found lacked fundamental characteristics that matter in realistic scenarios, such as the need for agents to explore noisy targets and validate findings without predefined success conditions. Improvements achieved in controlled benchmark settings were often not reflected when our agent operated with real clients. So the answer became clear: we needed our own evaluation framework.

In this two-part blog series, we want to contribute to the security community by presenting our evaluation protocol for AI Pentesting Agents, with the goal of bridging the gap between controlled testing environments and real-world scenarios. In this first part, we focus on the main characteristics of our evaluation methodology and how it differs from the current state of the art. We’ll also share results and comparisons using this evaluation setup. In the second part, we’ll go beyond standard performance measurement and discuss additional properties that a proper evaluation protocol should have if it is to be reliable, robust, and scalable enough for real production needs, like handling stochasticity, evaluating cumulative and temporal behavior, and defining cost-effective target subsets.

If you want more detail about our evaluation framework, we also wrote a paper about it. And if you want to get your hands dirty and evaluate your own Pentesting Agent with it, here is the open repository.

What is the state-of-the-art lacking?

Existing public work on Pentesting Agent evaluation has produced some interesting benchmarks, but most of them are task-specific and do not provide a broad, robust assessment of real-world pentesting capabilities.

A large portion of these benchmarks is based on Capture-the-Flag (CTF) challenges [link, link]. They mostly focus on controlled tasks where flag capture is used as the measure of success. We understand why: flags provide a simple and cheap way to assess success. But there are no flags in real scenarios. These benchmarks can be useful for measuring isolated offensive capabilities, yet they do not reflect realistic pentesting, where agents must explore noisy targets, decide where to focus, and distinguish valid findings from non-actionable leads.

Of course, not all benchmarks are CTF-based. For example, TermiBench replaces flags with remote code execution (RCE), which gives a stronger notion of exploit validation, but still ties success to a very specific objective. CVE-Bench moves closer to realism by instantiating real web-based CVEs, but each target is still evaluated through a predefined attack goal. BountyBench and CyberGym also provide stronger exploit validation by using a vulnerable-versus-patched logic, but they still evaluate bounded tasks rather than end-to-end pentesting scenarios.

Ultimately, by measuring only bounded capabilities on simpler targets, existing benchmarks do not reward the behaviors that matter most in realistic pentesting: open-ended exploration, planning, prioritization, strategy, and validation.

That is why our evaluation framework shifts the focus from task completion to what actually matters in the wild: finding valid vulnerabilities in complex targets. Rather than defining yet another fixed benchmark, we present a methodology that can be adapted to different target classes depending on the intended real-world application and the operational context in which the system will be used.

Our evaluation framework

Overview of the EthiBench evaluation framework, from target testing to final 1:1 finding-to-ground-truth matching and metric computation.

To prioritize realism over the convenience of closed-world benchmarks, our evaluation protocol is built around four core ideas:

• Real-world: Using realistic enough targets with a diverse set of functionalities, to force exploration, prioritization, validation and strategic decision-making.

• Explicit: Evaluating findings through a structured pipeline that pairs LLM-as-a-judge matching with bipartite resolution to turn finding reports into reliable metrics.

• Adaptable: Treating ground-truth as a resource that is continuously maintained, rather than as a fixed answer key.

• Robust: Accounting for stochasticity through repeated and cumulative evaluation under realistic conditions.

Why are realistic targets hard to evaluate on?

Realistic targets are easy to find. There are plenty of open-source realistic applications that can be easily deployed to test pentesting capabilities. So why are these not the standard for evaluating Pentesting Agents? Because a difficult challenge appears: how do we autonomously measure success?

This challenge can be broken down into two main issues:

1. How do we actually know which vulnerabilities are present in those targets in order to establish a ground-truth?

1. Even if we know, at least approximately, all vulnerabilities present in a target, how do we autonomously measure how successful a Pentesting Agent is at reliably finding them?

Let’s start with the second issue.

Building a robust finding-to-ground-truth pipeline

Our proposal for assessing a Pentesting Agent’s performance on a complex target, with multiple and diverse vulnerabilities, can be broken into three fundamental steps: (i) constructing a structured ground-truth, (ii) matching agent-reported findings to ground-truth entries using an LLM-as-a-judge approach, and (iii) resolving ambiguous correspondences through maximum bipartite matching.

Structuring the ground-truth

Once we know which vulnerabilities are present in a given target, they need to be structured in a way that allows reliable correspondence between them and the agent’s findings. For this, we build a structured ground-truth with multiple entries, where each entry can include fields such as “name”, “description” and “additional info”.

Some care is needed when creating these entries. If a ground-truth entry is too generic, it becomes difficult to determine whether a finding matches that specific vulnerability or only the broader vulnerability class. On the other hand, if an entry is too specific, it may encode assumptions about how the vulnerability should be discovered or described, causing valid findings to be incorrectly rejected when the agent reaches the same issue through a different path or reports it using different evidence. So, when creating these entries, we should aim for a level of specificity that supports accurate matching without over-constraining what a correct finding looks like.

This phase can also be used to define the scope of the evaluation, for example by limiting it to vulnerabilities above a chosen severity threshold or to specific categories of interest.

LLM-as-a-judge matching

A few years ago, given the complexity of a field like Offensive Security, it would have been nearly impossible to accurately match multiple pentesting findings to ground-truth vulnerability descriptions in a fully autonomous way. Fortunately, in the LLM era, we now have a powerful concept for exactly this type of task: LLM-as-a-judge.

The cybersecurity knowledge and semantic understanding of modern LLMs make them suitable judges for deciding whether an agent-reported finding matches a structured ground-truth entry. At this stage, we do not enforce a one-to-one correspondence: a finding may match multiple ground-truth entries, and multiple findings may match the same entry. This matters because, in realistic settings, both findings and ground-truth descriptions are imperfect; a single report may plausibly refer to multiple vulnerabilities, and the agent may also produce duplicate findings. For that reason, the LLM-as-a-judge stage is intentionally permissive and preserves plausible correspondences for later resolution.

Distinguishing duplicates from true positives with bipartite matching resolution

Because the LLM-as-a-judge phase can produce many-to-many matches, the relationship between findings and ground-truth entries naturally forms what we call a bipartite graph. That means matched findings cannot be directly interpreted as true positives.

To obtain reliable detection counts, we resolve these ambiguities by computing a maximum bipartite matching between findings and ground-truth entries. This ensures that each ground-truth entry is matched at most once, and that repeated or overlapping reports are not incorrectly counted as additional true positives. We solve this using the Hungarian algorithm, which gives us a reliable count of true positives while preventing duplicate reports from inflating the results.

Maximum bipartite matching resolves many-to-many LLM-as-a-judge matches into unambiguous 1:1 finding-to-ground-truth correspondences.

Is this pipeline really reliable?

Short answer: yes. Long answer: it depends heavily on the quality of the ground-truth and the LLM-as-a-judge system. We did not arrive at a solid finding-to-ground-truth pipeline on the first attempt. It took multiple iterations to define the best structure and level of detail for the ground-truth, as well as the right model and prompt for the LLM judge.

To make sure our version of the pipeline actually works, security researchers manually triaged multiple findings from our experiments, from which we selected 50: 25 that matched different ground-truth entries, meaning true positives, and 25 labeled as false positives. In the figure below, this expert-annotated labeling is compared against our finding-to-ground-truth matching system using different LLMs as judges.

LLM-as-a-judge classification results vs. human annotations for true positives and false positives across five model backends.

Because this stage involves a large number of finding-to-ground-truth comparisons across multiple experiments, we favored smaller and cheaper models for this purpose. From the results, we can see that both Gemini 3 Flash and GPT 5.4 Mini showed reliable classification performance, with the latter, on average, misclassifying only around one finding, typically as a duplicate instead of a true positive. Based on these experiments, GPT 5.4 Mini was the model we selected for the finding-to-ground-truth matching stage.

Expert maintenance of the ground-truth

Remember the earlier challenge of how to know which vulnerabilities are present in a target in order to establish a ground-truth? This is where we address it.

The starting point is relatively straightforward. Whether through manual testing, agentic testing with manual triage, or vulnerable applications with public information about known vulnerabilities, an expert security researcher can build an initial ground-truth file. The problem is that, unlike closed benchmarks with exhaustively enumerated answers, realistic pentesting targets are inherently open-ended. The full set of vulnerabilities may simply be unknown at evaluation time. In practice, some agent-reported findings that do not match any existing ground-truth entry may be labeled as false positives, even though they correspond to real vulnerabilities that were missed during target annotation. As a result, the reported metrics depend on how complete and accurate the current ground-truth is.

To make evaluation reliable in realistic settings, Pentesting Agent testing must be coupled with periodic expert review of unmatched findings and continuous maintenance of the ground-truth files. Besides adding new entries, the same review process can also be used to refine entries that are too vague, too restrictive, or consistently matched to multiple findings, which improves matching quality and helps recalibrate the LLM judge when necessary.

That said, this should not turn evaluation into a fully manual process, since that would defeat the purpose. We recommend periodic review focused on new unmatched finding types and on ground-truth entries that are repeatedly matched to multiple findings, followed by automatic re-evaluation with the updated ground-truth. As targets mature under this process, the need for manual intervention will become increasingly rare.

Evaluation metrics

The finding-to-ground-truth pipeline described above allows us to go beyond a binary “success or failure” style of evaluation. Under this framework, we can count true positives, false positives, false negatives, and duplicates. With these counts, the evaluation can approximate metrics commonly used in classical detection settings, such as Recall, Precision, and F1 score. Precision, for example, is a highly relevant metric for real-world application, but is not present in most available benchmarks because they do not have a way to assess it. In addition, if the ground-truth is properly labeled, we can derive other security-specific metrics, such as Severity Score (a non-linear point system based on ground-truth CVSS scores) and CWE Coverage.

To illustrate our evaluation protocol, we compare multiple agentic systems with pentesting capabilities: Strix, PentAGI, and Claude Code and our own Hackian’s Pentesting Agent. Since these systems are highly configurable, our results should not be read as a definitive ranking of either the agents or the underlying models. Instead, the goal is to show how our protocol enables multi-level comparisons across different agentic setups.

We evaluated Strix and PentAGI with multiple LLM backends, including Claude Sonnet 4.6, GPT 5.4, DeepSeek v3.1, and Qwen 3.6 Plus. Claude Code is evaluated only in its native setup with Claude Sonnet. We used three open-source targets: vuln-bank, paygoat (xben-090), and blog, together with our own ground-truth annotations (available in our EthiBench open repository). These targets are included to illustrate the evaluation protocol and should not be considered a complete benchmark for pentesting evaluation. Unlike vuln-bank and paygoat, blog originates from XBEN benchmark, where evaluation follows the usual CTF-style binary criterion: whether the agent obtains the flag. In our setting, however, we treat it as a realistic pentesting target rather than a flag-retrieval task, by applying our evaluation methodology with structured ground-truth. This helps illustrate an important motivation for our protocol: even simple CTF-style scenarios can contain multiple relevant vulnerabilities, including unintended ones, that would be missed by flag-based scoring alone.

The ground-truth was curated by security researchers, initially based on disclosed vulnerabilities in the targets’ repositories, then augmented and refined through iterative testing and manual analysis of the applications. In total, the dataset contains 108 expert-annotated vulnerabilities: 60 for vuln-bank, 28 for paygoat, and 20 for blog.

The figure below shows the metrics produced by our evaluation protocol, averaged across three independent runs for each agent setup. As you can see, performance varies quite a bit depending on the configuration, both in terms of finding vulnerabilities and validating them correctly.

We typically use the F1 score as our main decision metric, since it balances Recall (how many vulnerabilities are found) and Precision (how many of those are actually correct). But looking at F1 alone can be misleading. Breaking it down into Recall and Precision gives a much clearer picture of what’s going on. For example, in large-scale scenarios, an agent with low Precision can quickly become unusable, even if its Recall is high enough to keep the F1 score competitive. The opposite trade-off is also evident in our experiments: the Strix agentic system achieves high Precision, but at the cost of a significantly lower percentage of valid findings being reported.

On top of that, security-specific metrics like Severity Score and CWE Coverage add another layer of insight; they help capture not just how many issues are found, but how impactful they are and how broadly the system explores different classes of weaknesses. Putting all of this together gives a more complete view of a pentesting agent’s capabilities, and leads to better-informed decisions when deploying these systems in practice.

Going beyond average scores

Our evaluation metrics already go beyond simple success criteria, allowing for a broader view of performance differences. However, a comprehensive evaluation framework needs to be more than just a way of looking at average system performance, even if that performance is measured from multiple angles.

For that reason, our next blog post will cover additional, equally important challenges in evaluating Pentesting Agents for real-world use, such as dealing with stochasticity under realistic constraints, analyzing the cumulative behavior of agentic detections, temporal evaluation, and selecting representative subsets for sustainable experimentation. We will also show additional comparisons of Hackian’s Pentesting Agent against other solutions. Stay tuned!