"Confidence that turns out to be a big lie": The perils of AI pentesting

For most of the history of penetration testing, coverage was the enemy. Security teams worked against a simple and frustrating constraint: you cannot find what you do not look at, and human testers can only look at so much.

Engagements were scoped tightly not because organizations wanted limited visibility but because unlimited visibility was never really on the table. Time, cost, and human capacity set the ceiling, and the ceiling was low.

AI appeared to change that. And in one important respect, it genuinely has. Modern AI-assisted pentesting tools can surface vulnerability candidates at a scale and speed that no human team could match in a comparable timeframe.

Attack surface coverage that would once have taken weeks can now be compressed into days. For organizations operating under regulatory pressure or managing large, complex environments, that acceleration is meaningful.

The assumption that followed was reasonable enough: if coverage was the problem, and AI solves coverage, then AI solves pentesting.

That assumption is now running into the data.

The bottleneck didn't disappear 

Our research, drawn from a survey of 158 security practitioners who use AI-assisted vulnerability assessment tools in their day-to-day work, illustrates where the assumption breaks down.

Only 20.3% of respondents said they have a workflow capable of triaging 500 or more AI-generated vulnerability candidates from a single engagement. A further 38.6% said that volume would strain their team, while 29.7% said it would be unmanageable outright.

These are not practitioners who are unfamiliar with AI tooling. They are using it regularly, in live engagements, and they are still being overwhelmed by the output it produces. The reason is straightforward: AI has accelerated one part of the penetration testing workflow while leaving everything downstream of it essentially unchanged.

READ MORE: “Unmonitored agents at scale invite systemic disaster": New approaches to AI governance

Reproduction, validation, contextualization, prioritization, remediation planning: all of these remain human activities, running at human speed. When discovery outpaces validation by the margins the survey suggests, the time savings at the front of the workflow do not materialize as time savings overall. They materialize as a backlog.

Among the 147 practitioners in the survey who had used AI for finding generation, 87.8% encountered findings that required significant manual validation.

Practitioners described tools generating hundreds of findings across a single engagement, many of which proved to be duplicates, non-exploitable vulnerabilities, fabricated exploits, and in some cases, AI-generated CVEs that did not exist. The teams then spent days working through the output before they could act on any of it.

The false positive problem has been externalized

This is the part of the AI pentesting conversation that has not been named clearly enough. When a tool generates a finding, its job is done. Whether that finding is valid, exploitable, or worth the time of a senior security engineer to investigate is not a problem the tool bears. It is a problem the team bears.

Traditional pentesting had false positives too. But the human tester who produced a false positive was also the person who had to stand behind it in a debrief, answer for it in a report, and absorb the reputational cost of getting it wrong.

There was a built-in incentive toward accuracy that the current generation of AI tooling does not share in the same way. The cost of an invalid finding has been quietly transferred from the tool to the practitioner, and the practitioners are starting to feel it.

The survey captures this in a way that goes beyond volume. Respondents noted that hallucinated exploits and fabricated vulnerabilities were not just wasted effort in isolation. They eroded confidence in subsequent findings, forcing practitioners to verify even legitimate results more carefully than they otherwise would.

One respondent, a Security Manager at a mid-market company, described the experience as: "Confidence that turns out to be just a big lie." When a tool's output cannot be trusted, the verification burden extends across everything it produces, not just the findings that turn out to be wrong.

Where AI actually struggles

Beyond the volume problem, the survey data points to something more fundamental about the current limits of AI in security testing. Respondents were asked to identify the area where AI pentesting tools struggle most.

Business logic understanding ranked above exploit chaining and creativity, which is a more significant finding than it might initially appear.

Technical vulnerability detection is, in many respects, a pattern-matching problem. SQL injections, misconfigured headers, exposed endpoints: these have known signatures, and AI is increasingly competent at recognizing them. Business logic is a different category of problem entirely.

It requires understanding not just what an application can be made to do, but what it is supposed to do, and where the gap between those two things creates exploitable conditions.

Respondents described AI tools that could identify injection points but could not reason about a discount coupon that should only apply once per customer, or a cart function where adding a negative quantity resulted in free purchases, or a user ID in a URL that, when modified, exposed another customer's account.

READ MORE: The EU has “kill switches” in its critical systems. Can digital sovereignty fix them?

These are not obscure edge cases. They are the kinds of vulnerabilities that cause material harm when exploited, and they require an understanding of business intent that current AI tooling does not reliably have.

This is where the ceiling for AI-assisted pentesting currently sits, and it is a meaningful ceiling.

Organizations running ecommerce platforms, financial services applications, or any system where business rules carry real-world consequences need security testing that can reason about those rules.

Coverage of the attack surface is not sufficient if the tools doing the covering cannot understand the logic the surface is built around.

What the next advance actually looks like

The conversation around AI in pentesting has been almost entirely focused on discovery: how much can a tool find, how quickly can it find it, how broadly can it sweep.

Those are reasonable metrics if coverage is the constraint. But the survey data suggests that coverage is no longer where organizations are losing time or confidence.

The more useful measure is what happens after the scan. How many of the findings can a practitioner act on without spending days verifying them first? How many are grounded in a genuine understanding of how the application is supposed to behave?

How much manual work does the tool create downstream of its output, and is that work less than the manual work it replaced?

AI pentesting tools that move the needle on those questions will represent a different order of advance from what has come so far.

Faster discovery, at this point, produces diminishing returns if the findings it generates cannot be trusted at scale.

The industry has spent considerable energy solving the coverage problem. The validation problem is where the work now is.

Daniel Bechenea is Security Manager at Pentest-Tools.com 

Follow Machine on LinkedIn