How to Benchmark AI Code Security Review Tools Before You Buy

TL;DR

• Benchmark any AI code security review tool before you buy: seed a throwaway repo on your own stack with a known flaw for each category, plus a benign decoy where one applies, open one PR per branch, and score the results at PR time.
• Seed the eight categories a review should flag on a pull request: logic and authorization flaws, cross-file injection, authentication flaws, hardcoded secrets, risky or hallucinated dependencies, unsafe deserialization and weak crypto, insecure design, and newly reachable vulnerable code.
• A good finding shows traceable evidence, a triage verdict with reasoning you can audit, a fix that applies and passes tests, and a standard ID your auditors recognize. Low comment noise and an honest "unknown" matter too.
• Endor Labs AI Security Review covers every category on each PR, posting triage verdicts with reasoning and proposed fixes as PR comments, and uses full-stack reachability to flag code a PR makes newly reachable.

To benchmark an AI code security review tool, give it known flaws to find in the eight categories it should flag on a pull request: logic and authorization flaws, injection that crosses files, authentication flaws, hardcoded secrets, risky or hallucinated dependencies, unsafe deserialization and weak crypto, insecure design, and newly reachable vulnerable code. This guide gives you a seeded test case for each one, a benign decoy where one applies, and what a passing finding includes. Every finding also needs to show its work: evidence you can trace, a triage verdict with reasoning you can audit, a fix in the PR, and a standard ID your auditors recognize.

The phrase "AI code security review" is used two ways. It can mean AI doing the review, or someone reviewing code an AI wrote. This guide covers both. A PR from a coding agent carries the same flaw classes as one from a person, and the reviewer has to catch them no matter who opened it.

How do you test an AI code security review tool?

You test an AI code security review tool by seeding a disposable branch with known flaws and benign decoys, opening one PR per category, and scoring each finding against the rubric in the next section. The method works for any tool that handles code security review, and it gives you results on your own languages and frameworks instead of a vendor's demo repo.

In Endor Labs' Agent Security League benchmark, over 80% of the functionally correct solutions that AI coding agents produced still contained security vulnerabilities. If your reviewer misses a category, that category ships.

To set up the test:

1. Copy a small service into a throwaway repo. Pick one that uses your real stack: the same ORM, auth middleware, and package manager.
2. Create one branch per category. Each branch gets a seeded true positive and at least one benign decoy that looks similar but is safe.
3. Open one PR per branch. Mixing categories makes it hard to tell which seeded flaw a comment refers to.
4. Run the tool at PR time, the way your developers will see it. A nightly full scan tells you something different.
5. Record results in a scoring sheet.

A simple scoring sheet looks like this:

Use fake or revoked credentials only. Never publish the made-up package names you test with, and confirm they don't exist on the public registry before you use them.

What does a good finding look like?

A good finding names the vulnerable path, gives a verdict you can audit, proposes a fix that fits your code, and maps to a standard ID.

Logic and authorization flaws

Broken access control happens when code lets a user act on data they shouldn't reach. IDOR (CWE-639) and missing tenant checks are the common forms, and pattern matching rarely catches them because nothing on any single line looks wrong.

Test case

Add a multi-tenant export endpoint. The route reads a project ID from the URL and calls a new repository helper that filters by project but never checks that the project belongs to the caller's org.

// routes/exports.ts
router.get("/projects/:projectId/export", async (req, res) => {
  const rows = await exportService.buildExport(req.params.projectId);
  res.json(rows);
});

// repositories/exportRepo.ts
export function findExportRows(projectId: string) {
  return db("export_rows").where({ project_id: projectId });
}

Any authenticated user from any org can export another tenant's project by changing the ID.

Decoy

On a second branch, keep the same repository helper but mount a requireProjectAccess middleware on the route. That middleware loads the project and compares project.orgId to req.session.orgId. A good tool reads the middleware and stays quiet.

Good finding

The finding names req.params.projectId as the request-supplied ID and shows the path from route to service to repository with no tenant comparison along it. It proposes a query scoped by tenant, such as .where({ project_id: projectId, org_id: session.orgId }), and maps to CWE-639 under Broken Access Control.

Injection that crosses files

Cross-file injection happens when untrusted input enters in one module and reaches a query, command, or template in another with no safe handling along the way (CWE-89 for SQL).

Test case

A sort query parameter travels from a controller through a helper in a separate module into a raw ORDER BY clause. One caller sanitizes it, and the other doesn't.

# api/admin_reports.py
def admin_report(request):
    sort = sanitize_sort(request.args.get("sort"))  # allowlist check
    return run_report(build_report_query(sort))

# api/public_reports.py
def public_report(request):
    sort = request.args.get("sort", "created_at")
    return run_report(build_report_query(sort))

# db/queries.py
def build_report_query(sort):
    return f"SELECT id, name, total FROM reports ORDER BY {sort}"

Good finding

The finding shows the full trace from request.args in public_reports.py to the f-string in db/queries.py, and explains why the admin path is safe and the public path isn't. A column name can't be bound as a query parameter, because placeholders only work for values, so "use parameterized queries" is a generic fix that won't work. A passing fix maps the input to an allowlist of known column names and rejects everything else.

Authentication flaws

Authentication flaws let a request prove less than the code assumes it proved, such as a token that isn't verified (CWE-347) or a route that skips authentication entirely (CWE-306).

Test case

Add a JWT helper that decodes claims without checking the signature, and register a new route before the auth middleware runs.

// auth/claims.ts
export function getClaims(token: string) {
  return jwt.decode(token); // no signature check
}

// app.ts
app.get("/internal/usage", usageHandler); // registered before auth
app.use(authMiddleware);

For a variant, call jwt.verify with no algorithms option on a library version that accepts whatever the token header says. As an optional second case, add a password reset token stored with no expiry and no single-use flag (CWE-640).

Good finding

The finding lists every route that trusts the output of getClaims, not only the helper itself. It proposes the library's verify call with a pinned algorithm, for example jwt.verify(token, publicKey, { algorithms: ["RS256"] }). For the unprotected route, it points out the registration order and suggests moving the route below the middleware or attaching auth to it directly.

Hardcoded and leaked secrets

A secrets finding flags credentials committed to code, logs, or Git history (CWE-798), where anyone with repo access can read them.

Test cases

• Put a revoked key, or a documented example key in a valid format such as AWS's AKIAIOSFODNN7EXAMPLE, inside a test fixture.
• Write a token to a debug log line: logger.debug(f"refresh token={token}").
• Commit a fake secret in one commit, then delete it in a later commit of the same PR. The final diff looks clean, but the history doesn't.

Good finding

The finding names the file and the commit, including the commit where the secret was added even if a later one removed it. It says whether the credential is live or revoked, which tells the developer how urgent rotation is. The guidance says rotate first, then remove, because deleting the line alone leaves a working credential in Git history.

Endor Labs Secrets Detection scans full Git history, validates findings as live or revoked, and stops leaks at pre-commit. Use the deleted-in-a-later-commit case to see whether your tool reads history or only the final diff.

Risky or hallucinated dependencies

This category covers packages a PR adds that don't exist on the registry, typosquats of popular names, and known-vulnerable versions. Code models sometimes suggest plausible package names that were never published, and an attacker who registers that name gets code execution on every machine that installs it.

Test cases

For the hallucinated package, pick a made-up name, confirm it returns nothing on the registry, and add it to your manifest. Never publish it.

# requirements.txt
requests==2.32.3
fastjson-schema-guardx==1.4.0   # confirmed absent from PyPI, never publish

For the vulnerable version, pin a dependency below a known fix where the advisory ties the risk to one function. In npm, lodash versions before 4.17.21 have a command injection issue in _.template. Open two PRs on that pinned version:

• PR A calls _.template(userSuppliedString) from a request handler.
• PR B uses only _.chunk and _.uniq.

Good finding

For the hallucinated package, the finding reports that the name doesn't resolve on the registry and recommends removing it or replacing it with the intended package. For the vulnerable version, a strong tool flags PR A as reachable, with the call path from the handler to _.template, and treats PR B as lower priority because the vulnerable function is never called. A tool that rates both PRs the same is reporting version noise.

‍

Endor Labs Upgrade Impact Analysis ranks upgrade paths by how likely they are to break your application code, and Endor Patches backports fixes when an upgrade isn't practical. 

Unsafe deserialization and weak crypto

Unsafe deserialization lets attacker-controlled bytes become live objects, which can lead to code execution (CWE-502). Weak crypto means using a broken algorithm or a predictable random source for a security purpose (CWE-327, CWE-338). AI-generated code inherits these patterns from its training data, so expect to see them in agent-written PRs.

Test cases

# Deserializing a request body
config = yaml.load(request.data, Loader=yaml.Loader)
session = pickle.loads(request.get_data())

# Password storage
hashed = hashlib.md5(password.encode()).hexdigest()

# Reset tokens
token = "".join(random.choice(string.ascii_letters) for _ in range(32))

Decoy

cache_key = hashlib.md5(request.url.encode(), usedforsecurity=False).hexdigest()

MD5 as a cache key isn't a security control. A tool that flags every md5 string is keyword matching. A good tool stays quiet, or labels it informational and explains why.

Good finding

For deserialization, the finding shows untrusted input reaching the deserializer and proposes yaml.safe_load or a schema-validated format like JSON. For crypto, it explains why the algorithm is wrong for this specific use. MD5 fails for passwords because it's fast and unsalted, so offline guessing is cheap. NIST SP 800-63B-4 calls for salted, purpose-built password hashing, and NIST SP 800-131A Rev. 2 covers which algorithms are disallowed. Use a password hashing function such as Argon2 or bcrypt, plus secrets.token_urlsafe() for reset tokens.

Insecure design and business logic

Insecure design is a flaw where every line looks correct but the workflow breaks a rule the business depends on, like "a coupon is used once" or "you can't refund an unpaid order."

Test cases

• A coupon redemption that reads coupon.used, then writes used = True in a separate query. Two concurrent requests both pass the check (CWE-362).
• A refund endpoint that issues money back without checking whether the order was paid, shipped, or already refunded.
• A cart endpoint that accepts negative quantities, so a large negative line item drops the total.

Good finding

The finding states the broken invariant in plain words and lists the abuse steps in order: send two redemption requests within milliseconds, both read used = False, and both apply the discount. The fix enforces the invariant atomically or on the server, for example with a conditional update (UPDATE coupons SET used = true WHERE id = ? AND used = false) and a check on the affected row count.

For more on how multi-agent review handles this class of flaw, read the whitepaper on detecting security design flaws at scale.

Changes that make old vulnerable code reachable

Some PRs add no vulnerable lines at all. They wire a route or job to existing code that was vulnerable but unused, so the diff looks clean.

Test case

On your main branch, commit a dormant helper with a path traversal flaw (CWE-22) that nothing calls yet:

# utils/templates.py  (already on main, no callers)
def read_template(name):
    with open(os.path.join(TEMPLATE_DIR, name)) as f:
        return f.read()

Then open a PR that adds only this:

# api/preview.py
@app.get("/preview")
def preview(request):
    return read_template(request.args["template"])

The new request for ?template=../../etc/passwd reads arbitrary files.

Good finding

The tool flags the PR even though the vulnerable lines are unchanged. It shows the new call path from /preview through request.args["template"] into os.path.join and open. It proposes a fix in the helper, such as resolving the path and confirming it stays inside TEMPLATE_DIR, since the next caller would hit the same flaw.

Diff-only review can't catch this. It takes a model of the whole application, which is what Endor Labs' code context graph and full-stack reachability provide. They model application behavior at line level and produce the data flow, call path, and reachability proof a finding like this needs.

What should the review do beyond detection?

Beyond detection, the review should triage with reasoning you can audit, put fixes in the PR, keep comment noise low, and say "unknown" when it can't decide. Those behaviors decide whether developers read the comments or mute the bot.

Triage verdicts you can audit

Endor Labs' triage agent classifies each finding as true positive, false positive, or unknown, with reasoning your team can audit. Read the reasoning on your seeded cases. If it doesn't mention the middleware on the authorization decoy, the verdict was a guess.

"Unknown" as a quality signal

An honest "unknown" beats a confident guess, because it routes the right findings to a person. Test for it with a deliberately ambiguous case. For example, add a query that skips the tenant filter on a table where tenant isolation depends on a database row-level security policy defined outside the repo. The tool can't see that policy. A good reviewer says so, names what it couldn't verify, and asks a person to confirm.

Noise you can count

Count comments per PR, and count flags raised on your decoys. Those numbers matter as much as true positives. A reviewer that posts 30 comments on a 40-line PR teaches developers to scroll past all of them, including the one that mattered.

Fixes that apply

Score each suggested fix on two things: whether it applies cleanly, and whether the tests pass afterward. A correct fix that doesn't compile still costs a developer time.

Some decisions stay with people, such as accepting a design tradeoff or weighing a risk the tool can't see. Our guide on reviewing code an agent wrote covers where human reviewers still own the call.

Checklist summary

How Endor Labs AI Security Review covers the checklist

If you're evaluating tools, run this checklist against Endor Labs AI Security Review and score it the same way you'd score anything else. It maps to each category like this:

• Logic, authorization, and injection: detection agents trace how data moves across files and functions to catch broken access control, IDOR, injection, and insecure design.
• Authentication: AI Security Review catches fail-open authentication alongside the other logic flaws.
• Secrets: a single PR scan covers any secrets the PR introduced, and Secrets Detection validates findings as live or revoked.
• Dependencies: the same scan runs SCA on the dependencies the PR changed, and Upgrade Impact Analysis and Endor Patches shape the solution.
• Deserialization and weak crypto: AI SAST works across the full OWASP 2025 Top 10 and reads context. It classified SHA-1 used to implement ASN.1 DER encoding as a false positive, the same distinction your MD5 cache key decoy tests.
• Newly reachable code: full-stack reachability and the code context graph provide the call path and reachability proof this category needs.

Every confirmed finding gets a triage verdict with reasoning and a proposed fix matched to your codebase. Results post as PR comments, connect to your CI/CD pipeline and Endor Labs' policy engine, built on Open Policy Agent (OPA), and run on GitHub, GitLab, and Bitbucket. Teams can block a PR only on high-confidence findings and let everything else keep moving.

The numbers behind it:

• AI SAST, the engine AI Security Review runs on every PR, delivers up to 92% false positive reduction.
• Across 8 real-world projects in Java, Python, JavaScript, TypeScript, C#, and Go, with ground truth set by Endor Labs' security researchers, AI SAST found 192 real vulnerabilities: 2.6x what Claude (Opus 4.7) caught and 3.5x what Codex (GPT-5.5) caught, plus 63 vulnerabilities no other tool in the test reported.
• After rollout, fewer than 10% of PRs need human security review.

Your seeded branches will show how it performs on your own code.

Frequently asked questions

Does AI code security review only apply to AI-generated code?

No, the categories in this checklist apply to every PR, whether a developer or a coding agent wrote it. Agent-written PRs carry the same flaw classes, and higher PR volume makes automated review more useful, but the test cases work the same way for human-written code.

Can AI code security review replace human security reviewers?

AI code security review can take most PRs off a person's plate, but it shouldn't replace the people who make judgment calls. A good reviewer handles routine triage and fixes, then sends design tradeoffs and "unknown" verdicts to someone with the context to decide.

How often should you rerun these test cases against your tool?

Rerun the test cases whenever the tool, its model, or its configuration changes, and whenever you adopt a new language or framework. Keep the seeded branches in a repo so a rerun takes minutes. A quarterly check is a reasonable default if nothing else changes.