Benchmarking Endor Labs AI SAST: 2.6x more real vulnerabilities found than frontier models

Introduction

AI now writes a large and growing share of production code, and that code inherits the same vulnerability patterns as everything before it, often faster than human review can keep up. The labs behind those models suggest that if you point a capable enough model at your code it will surface the rare, long-dormant flaws no other tool can. We benchmarked it and found that Claude and Codex aren't the strongest at finding flaws, and missed many of the hard-to-find ones, though we didn’t have access to Mythos-class models for these results.

The reason comes down to how the two common approaches work. Prompted to scan a repository on its own, a frontier model reasons well about the code it surfaces but leaves most of it unexamined. A traditional pattern-based scanner is the reverse: it matches known-bad patterns across the whole codebase but can't reason about how an application behaves, so it flags false positives while missing real flaws.

Endor Labs AI SAST found 192 real vulnerabilities, more than double any other tool in the test: 2.6x what Claude (Opus 4.7) caught, 3.5x what Codex (GPT-5.5) caught, and 2.4x what Semgrep OSS found. Endor Labs was not only able to find the most true positives, it also found the most vulnerabilities no other tool could find. 63 of its findings were caught by no other tool in the test, with the next-highest at 18 (Claude Opus 4.7). And Endor Labs AI SAST covered the widest range of weakness types of any tool, detecting 64 of the 106 CWE types in the benchmark, with the next highest at 36 types (Claude Opus 4.7).

Endor Labs built its reachability engine for software composition analysis, where call-graph and dataflow analysis cut noise by prioritizing only the vulnerabilities that can be reached and exploited in the context of your application code. AI SAST points that same engine at first-party code as part of a multi-agent system, building a unified view across the different projects, services, and infrastructure running around the code itself.

How we measured this

We ran Endor Labs AI SAST against four traditional SAST tools (Semgrep OSS, OpenGrep, Bearer, and CodeQL) and two frontier models prompted to find vulnerabilities directly (Claude Opus 4.7 and Codex GPT-5.5). The corpus is 8 open-source projects spanning Java, Python, JavaScript, TypeScript, C# and Go, chosen as a mix of purpose-built security benchmarks and projects that resemble real applications: real frameworks, real dependency graphs, and control flow that crosses files and services.

• anonymous-github
• demo-netflicks
• doublestar
• OWASP BenchmarkJava
• OWASP Juice Shop
• OWASP WebGoat
• Shopizer
• XBOW Validation Benchmarks

Our security researchers established the ground truth by pooling every tool's true positives, de-duplicating them by hand, adding issues found through manual review, and verifying each one. A vulnerability in that set that a tool failed to report counts against its recall. Each tool is scored only against the vulnerabilities in the languages and projects it supports, so denominators differ by tool. Because AI-based tools vary between runs, we ran them multiple times and report the aggregate.

How it compares

True positive detection

Endor Labs AI SAST out-detected every other tool in the test, surfacing 2x more real vulnerabilities compared to Semgrep OSS and several times what Claude (2.6x) and Codex (3.5x) managed. 

It did that with far less noise than the pattern scanners that found the most: about 60% fewer false positives than Semgrep OSS and OpenGrep, which flagged over 460 each. That aligns with industry benchmarks estimating 68-78% of SAST findings are false positives. It also concentrated its finds where risk is highest, catching 4x more high-severity vulnerabilities as the strongest traditional tool (72 vs 17) and leading on critical (46 vs 38).

File coverage

On file coverage, the tools fall into two camps, each with a blind spot. Traditional SAST tools are systematic, scanning every supported file in the repo, but limited to known rule patterns, so they only find problems they were already taught to look for. 

LLM agents like Claude and Codex work the other way: flexible enough to surface issues no rule anticipated, but they explore a narrow, shifting slice each run. That gap widens as repositories grow. On a large enterprise Java codebase, the security-relevant code Claude and Codex examined dropped below 10%. 

Endor Labs AI SAST balances both. Every function is triaged to judge whether it's worth a closer look, and only the interesting ones get an in-depth AI analysis. You get the breadth of traditional SAST and the flexibility of the agents, without the blind spot of either.

Recall, precision, and F1 balance

We also measured recall, the share of real vulnerabilities a tool finds, precision, the share of its findings that are real, and F1, a balance between the two. Endor Labs led the test on both recall (0.435) and F1 (0.465), while the model-only tools were precise but limited: Codex (0.859) and Claude (0.718) rarely raised a false alarm, but their recall sat near the bottom of the field, so most real findings slipped past. The noisiest pattern scanners were the mirror image, precision near 0.15 from hundreds of false positives apiece, and even the more balanced traditional tools like CodeQL topped out at 0.211 recall.

What the other tools missed

The unique findings covered by Endor Labs were concentrated in the CWE categories where pattern matching falls short: broken access control and IDOR, authentication failures, and business-logic flaws, all of which depend on how an application behaves.

Buffa by Anthropic

On buffa, Anthropic's Rust protobuf library, AI SAST traced untrusted wire data into an unbounded heap allocation reachable from the default decode APIs—a flaw that amplifies a small input into a ~22x memory blow-up and OOM-kills the process (CVE-2026-55407). It's a clean example of source-to-sink data flow analysis earning its keep on a memory-safe language, where there's no overflow to grep for.

You can read more about CVE-2026-55407 this in the blog post from our security team.

Shopizer by Shopify

In Shopizer, a production open-source e-commerce platform, Endor Labs flagged an insecure direct object reference in the product-image delete endpoint. The handler deletes an image by its client-supplied ID without checking that the image belongs to the caller's store, so one merchant can delete another's images. The vulnerability is the missing ownership check spanning the controller and the service, not a string anywhere in the code.

WebGoat benchmark

In WebGoat, Endor Labs caught an authentication bypass that no other tool in the test found. The account-verification check confirms the number of submitted answers and that any recognized keys are right, but it never requires those keys to exist, so two arbitrary keys pass and both real checks are skipped. Catching it means following the request data through the verification logic and reasoning about what the guards actually enforce.

How it works

Finding the vulnerabilities that depend on how an application behaves takes more than a capable model. It takes a map of how the whole system fits together and the dataflow analysis to trace how data moves through it. Endor Labs AI SAST is built on that architecture, the same call-graph and reachability infrastructure that has powered our SCA, now pointed at first-party code, run by specialized security agents. Here are the five steps from first-party code to a confirmed finding, with each step building on the last.

1. Index the code and build the code graph. During indexing, code search and code navigation work together to build the code graph, which spans your entire codebase including any services and repositories. Code search creates a searchable database of your code and understands what the code means, so it recognizes when a function is renamed but behaves the same, and when its behavior actually changes. Code navigation reads the structure of that graph, tracing how data flows through the code and which functions call a vulnerable function. We index for maximum coverage to limit false negatives.
2. Add framework and infrastructure context. Endor Labs learns how and where your code runs, including the frameworks it uses, the protections already in place, and whether a service is exposed to the internet, so the focus stays on what's actually at risk.
3. Identify security vulnerabilities. Trace how data flows through the application, from where it enters to where it's used, following calls across files and services to find the paths that create real risk. 
4. Triage the findings. This is where false positives get removed. It double-checks each finding against your code. For example, spotting a sanitizer that already neutralizes the issue, so only real problems reach a developer.
5. Classify findings. Confirmed findings get a CWE label, severity, and the evidence behind them, the exact data flow and call paths, so they're ready to act on. Severity is scored against your application's context: the same flaw can rate higher in an internet-facing web service than in a local CLI tool, depending on how the code runs and what's exposed.

Coverage feeds recall, triage protects precision, and the code graph is what lets the model reason about behavior instead of surface patterns. That is the architecture behind the numbers above. Every confirmed finding also comes with exploit validation, to show how an issue is reachable and exploitable, and remediation, with suggested code changes you can apply.

Beyond detection, the same architecture shapes how findings show up. They surface where code is written: a baseline scan, a pull-request check, or locally through the IDE, CLI, an MCP server, and hooks and skills for AI coding agents, so a flaw can appear as the code is written, including code an agent generates.

Every tool has its tradeoffs

If you want near-zero false positives and can accept missing most of what's there, a frontier model used directly gives you that. If you want fast, deterministic scans and dependable coverage of the well-understood vulnerability classes like injection, a pattern-based scanner delivers that, provided you have the resources to triage the noise. We optimized for the balance most teams actually need: enough recall to trust that the serious issues surface, with little enough noise that developers keep paying attention.

Getting Started

Endor Labs AI SAST is generally available. If you're figuring out how to secure the code your developers and AI agents are shipping, we'd like to show you what it surfaces in your own codebase, with the call paths and evidence behind every finding. Book a demo.