AI Patch Study Keeps Humans In Vulnerability Reviews

AI-generated security patches closed vulnerabilities cleanly only about one quarter of the time in a 1Password Off-by-1 Labs test, leaving most outputs with incomplete remediation, changed application behavior or fresh security risk.

The Register reported that researchers tested 6,080 patches produced by ChatGPT 5.5 at "medium" effort and Claude Opus 4.8 at "high" effort across six recently disclosed CVEs.

Keith Hoodlet, 1Password's director of security research, said the average success rate for a patch that fully resolved the vulnerability without materially changing application behavior was 26.0 percent.

That result puts autonomous remediation closer to a security review workload than a replacement for engineers who understand the vulnerable code.

The failure modes were spread across several categories.

Another 20.1 percent of the generated patches fixed the original issue but changed application behavior.

Some 2.3 percent fixed the issue while introducing new security problems.

Nearly half, 49.3 percent, failed to close at least one existing exploit path, while 2.2 percent both missed the vulnerability and introduced a new exploit path.

Even the outputs that appeared to work were not all durable repairs.

Among patches rated either cleanly successful or successful with application behavior changes, more than a third were judged fragile because the adjusted code guarded against a specific vulnerability pattern without fixing the underlying weakness.

The research paper by Axel Mierczuk, Spencer Michaels and Hoodlet calls the pattern FLAWED, short for Fix-Like Artifacts With Embedded Defects.

The authors concluded that the expected value of a fully LLM-generated, non-human-reviewed patch is "a net-negative by a considerable margin."

Guidance changed the results sharply, but not in a way that removes the need for supervision.

Correct guidance raised the LLM fix-success rate to 65.0 percent, compared with 50.4 percent with no guidance.

Incorrect guidance pushed the success rate down to about 15.2 percent, showing how easily automated patching can follow a bad premise into plausible but incomplete code.

Human developers also need initial direction when tackling a vulnerability, but the authors argue they have a better chance of catching misleading information as they reason through the code.

That distinction matters because a patch that looks right can still leave an exploit path open or alter the application in ways that create operational risk.

The economics are tempting in isolation.

The average successful, clean patch cost $6.74, including the cost of failed attempts.

But the paper argues that organizations have to count the expert time required to review large numbers of similar, subtly different and often incorrect patches before any of them can be trusted in production.

The Off-by-1 Labs team released a FLAWED patch evaluation harness for organizations that want to test the effectiveness of security fixes.

For now, the reported result leaves autonomous LLM-driven patching dependent on human review, targeted validation and an engineer who remains responsible for deciding whether the vulnerability has actually been removed.