AI Can Find Bugs, But Human Knowledge Still Proves Them – CYBERDEFENSA.MX
Artificial intelligence (AI) is changing offensive security, but it has not changed the standard that matters most: a finding has to be proven before it becomes useful. AI-assisted tools can read code quickly, generate payloads, summarize attack surfaces, explain unfamiliar APIs, and run repetitive testing workflows at impressive speed. That is a real advantage for security teams. It also creates a new kind of pressure, because the industry can now produce more vulnerability-looking output than ever before.
The problem is that output is not the same as evidence. A generated report can sound polished, include a severity rating, and even contain a proof-of-concept that looks reasonable at first glance. None of that proves the bug exists in the deployed environment. None of it proves exploitability, impact, or risk. In offensive testing, the hard part has never been writing something that sounds like a vulnerability report. The hard part is demonstrating what is actually true.
That distinction is becoming more important as AI becomes more common in security workflows. AI can accelerate discovery, but validation still depends on knowledge: knowledge of systems, protocols, application behavior, identity boundaries, memory corruption, business logic, and all the implementation details that separate a plausible theory from a real exploit. The future of offensive security will not belong to people who merely produce the largest number of findings. It will belong to people and teams that can prove what matters.
The Industry Is Already Seeing the Cost of Shallow AI Output
The warning signs are already visible. Bug bounty programs and maintainers have been dealing with a surge of low-quality AI-generated reports, often submitted with thin evidence, templated language, and little meaningful validation. Bugcrowd publicly addressed this pattern in its policy changes around AI-generated submissions, describing a class of reports that looked polished but created unnecessary triage burden rather than a useful security signal.
This is not just a bug bounty problem. It is a preview of what happens anywhere AI is used to create security findings without enough human judgment behind them. If a tool can generate a convincing write-up in seconds, organizations will receive more reports, more alerts, and more claims. Unless those claims are validated, the result is not better security. It is a larger queue.
Security teams are already overloaded with scanner output, dependency alerts, cloud configuration issues, and compliance findings. Adding AI-generated speculation on top of that does not help unless the quality bar goes up at the same time. A finding should answer basic questions clearly: what happened, how it was reproduced, what the attacker controls, which boundary was crossed, and what the demonstrated impact is. Without that, the report may be interesting, but it is not ready to drive engineering action.
“Looks Vulnerable” Is Not the Same as Vulnerable
One of the most dangerous habits in offensive testing is confusing a suspicious pattern with a validated vulnerability. AI can make that habit worse because it is good at explaining why something might be bad. A model may see user input near a database query and describe SQL injection. It may see a URL fetch and suggest SSRF. It may see a dangerous API in a code path and describe remote code execution. Sometimes the model is pointing at a real issue. Other times, it is missing the conditions that decide whether the issue matters.
A tester still has to prove reachability. Does the attacker-controlled input actually reach the dangerous operation? Is authentication required? Is authorization enforced somewhere else? Is the vulnerable feature enabled? Does the production configuration expose the code path? Does the application normalize, encode, sanitize, or reject the payload before it matters? Does the issue cross a trust boundary or merely affect an internal-only path with no practical security impact?
These questions are where real offensive security begins. They are also where shallow automation often breaks down. AI can generate hypotheses quickly, but hypotheses are not findings. A good tester treats AI output as a lead to investigate, not a conclusion to forward.
Why Knowledge Still Matters
The best offensive security practitioners are valuable because they understand systems, not because they can run tools. Tools have always been part of the job, but tool output has never been enough. A web scanner may identify a parameter that reflects input. A static analyzer may flag a dangerous function. A fuzzer may produce a crash. A language model may describe a plausible attack path. In every case, someone still needs to understand what the signal means.
That understanding is usually earned through repetition. Senior researchers spent years doing the work manually: tracing requests, reading source, reverse engineering binaries, debugging crashes, writing exploit code, breaking authentication flows, and learning how real systems fail. That process builds memory and instinct. It teaches a practitioner when a finding is probably real, when a tool is being misled, and when a small bug may become serious if chained with something else.
This kind of knowledge is hard to fake. It shows up in the questions a tester asks. It shows up in the way a report is written. It shows up in whether the tester can explain the exploit path without hiding behind generic language. Most importantly, it shows up when the first attempt fails. A person who understands the system can adapt. A person who only accepts the tool’s explanation is often stuck.
AI Can Make Good Testers Faster, but Can Also Make People Rusty
There is a real concern among experienced practitioners that overdependence on AI can make people rusty. This is not an anti-AI argument. It is a human learning argument. When a tool answers every question instantly, it becomes tempting to stop remembering details. When it writes the first version of every script, it becomes tempting to stop practicing. When it explains every code path, payload, crash, and error message, it becomes tempting to stop building the mental model yourself.
That convenience has a cost. Offensive security rewards depth, pattern recognition, and technical recall. The hardest findings often come from recognizing that a behavior in one area violates an assumption somewhere else. They come from knowing how parsers, frameworks, allocators, identity providers, and authorization systems have failed before. They come from seeing the connection between small details that do not look important in isolation.
If practitioners stop exercising those muscles, they lose some of the very skill that makes them effective. The risk is not that AI makes security professionals useless. The risk is that people let AI do too much of the thinking too early, then mistake fluency for competence. Prompting is useful, but it is not a replacement for judgment.
Most AI-Assisted Testing Still Uses Familiar Techniques
A lot of AI security marketing can make it sound as if machine learning is discovering vulnerabilities through some entirely new kind of reasoning. Sometimes models do surface patterns a human might miss, especially across large and unfamiliar codebases. That is useful. But in many practical offensive testing workflows, the underlying techniques are still familiar: enumerate endpoints, inspect parameters, trace data flow, compare authenticated and unauthenticated behavior, generate payloads, run fuzzers, observe responses, and determine whether the application state changed in a security-relevant way.
In other words, many AI-enabled systems are orchestrating known testing techniques at scale. They can plan, execute, observe, and iterate faster than a human doing everything by hand. That is a meaningful improvement, but it does not remove the need to understand the result. If the system reports an authorization flaw, someone still has to know whether the object relationship matters. If it reports a memory corruption bug, someone still has to reason about reachability, crash context, mitigations, and exploitability. If it reports an API weakness, someone still has to determine whether the observed behavior violates the application’s trust model.
The most valuable use of AI is not to replace those decisions. It is to reduce the mechanical work around them so skilled testers can spend more time on analysis and validation.
What Good Validation Looks Like
A validated offensive finding should be specific, reproducible, and tied to impact. It should not require the reader to guess why the issue matters. The report should make the exploit path clear enough that an engineer can reproduce it and a security leader can understand the risk. That does not mean every issue needs a dramatic exploit chain or a movie-style proof-of-concept. It means the evidence should support the claim.
For AI-assisted testing, teams should draw a sharp line between leads and validated findings. A lead is something worth investigating. A validated finding is something that has been tested and proven. Mixing those categories creates confusion and wastes time. A good workflow can absolutely use AI to generate leads, but the promotion from lead to finding should require evidence.
Practical Validation Checklist
A practical validation standard does not need to be complicated. Before a lead becomes a reported finding, the tester should be able to answer questions like these:
- What specific behavior was observed, and where did it occur?
- What attacker-controlled input, identity, or state was required?
- What security boundary was crossed, such as authentication, authorization, tenancy, trust, privilege, or memory safety?
- What exact steps reproduce the behavior in the target environment?
- What is the demonstrated impact, not just the theoretical worst case?
- What evidence shows that the issue is reachable and relevant in the deployed configuration?
- What would a fix need to change, and how can the team confirm that the fix works?
This kind of checklist helps keep AI in the right role. It can help produce candidates, suggest test ideas, and speed up reproduction. It should not be allowed to skip the step where a human verifies the claim against reality.
The Human Role Is Still Technical
One of the underappreciated realities of AI security platforms is that human validation remains deeply important behind the scenes. That should not be surprising. Offensive security has always required judgment, and judgment is especially important when findings become consequential. The person reviewing the evidence has to decide whether the exploit path is realistic, whether the environment matters, whether the issue is isolated or chainable, and whether the severity claim is justified.
This is not just an administrative quality-control function. It is technical work. Authorization flaws often depend on business logic and object relationships. API vulnerabilities may require understanding how roles, tenants, and resources interact. Memory corruption requires reasoning about crash state, control, mitigations, and exploit primitives. Cloud findings depend heavily on identity, trust policies, and service-specific behavior. AI can assist with all of this, but it does not remove the need for someone who knows what they are looking at.
The higher the impact of a finding, the more important the human role becomes. Organizations do not want a confident guess when the result may affect engineering priorities, customer trust, compliance obligations, or executive risk decisions. They need proof.
Avoiding Exaggerated Impact
AI-generated reports can also overstate severity. Reflected input is not cross-site scripting until script execution is demonstrated. A URL fetch is not meaningful SSRF until the tester can show access to something the attacker should not reach. A dangerous function is not remote code execution unless reachability, control, and execution can be proven. These mistakes are not just embarrassing; they erode trust between security teams and engineering teams. It happens quite often that a finding will get a rating of CVSS 9.8, when in fact it might not even be a finding at all.
Experienced researchers are careful with impact because they know it has to be earned. A bug in an admin-only feature does not carry the same risk as an unauthenticated internet-facing bug. A crash may be a denial of service, a path to code execution, or simply an unexploitable reliability issue, depending on the context. A missing check in one code path may be serious, or it may be protected by a control somewhere else. The only way to know is to validate.
Good validation prevents both underreporting and overreporting. It helps testers avoid crying wolf, but it also gives them the evidence needed to make a strong case when the issue is genuinely serious. Tenable also recently brought up challenges in this space, including how there are often critical contextual combinations that are also missed.
How Teams Should Use AI Without Losing Skill
The right goal is not to avoid AI. The technology is too useful for that. The right goal is to use it in a way that strengthens offensive testing instead of weakening the people doing it. AI should help testers move faster, explore more hypotheses, and reduce repetitive work. It should not become a substitute for learning how systems behave.
Security leaders can encourage that balance by setting expectations around evidence and training. Junior testers should still learn fundamentals before they outsource too much of the process. Senior testers should use AI as a force multiplier, not as an authority. Teams should review not only whether a finding was generated, but whether the tester can explain and reproduce it. That explanation is where real understanding becomes visible.
A healthy AI-assisted offensive testing program should reward validated impact over volume. It should measure signal quality, not just finding the count. It should preserve manual practice in areas like request manipulation, code review, debugging, exploit development, threat modeling, and impact analysis. It should also use AI as a teaching tool: when the model suggests an issue, the tester should ask why, test the claim, and learn from the result.
The Standard Has Not Changed: Prove It
AI will continue to improve. Agents will become better at navigating applications, reading code, generating payloads, and documenting results. Some of this progress will be genuinely impressive, and security teams should take advantage of it. But offensive security cannot become a volume game where every plausible theory becomes someone else’s triage burden.
The core standard of the field is still simple: prove it. Prove the bug exists. Prove the attacker can reach it. Prove the impact. Prove the business risk. Prove the fix works. AI does not lower that standard. If anything, it raises the importance of enforcing it, because convincing but unproven output is now easier to produce than ever.
The best researchers and teams of the next decade will not be the ones that reject AI. They will be the ones who combine automation with technical judgment, using the machine to accelerate the work without handing it the final say. Knowing when to stop, inspect, test, and think will remain a competitive advantage. Knowledge still matters because validation still matters, and in offensive security, validation is the difference between noise and truth.
I will be expanding on this topic in SEC660: Advanced Penetration Testing, Exploit Writing, and Ethical Hacking at SANS Network Security 2026. Our course update blends together manual understanding of complex topics, such as exploit writing, and instructs how to leverage AI to assist in automating specific tasks.
Register for SANS Network Security here.
Note: This article has been expertly written and contributed by Stephen Sims, SANS Fellow.
