How to evaluate ADLC security solutions: A buyer's checklist

TL;DR

• ADLC security must cover the code AI coding agents produce and the actions they take, including installing packages, editing CI config, and calling tools.
• Before vendor talks, scope your agentic workflow: agents in use, risky languages, repo structure, weekly PR volume, agent write access, and required integrations.
• Test coverage on your own code for multi-layer logic flaws such as IDOR, reachable open source and container vulnerabilities, validated secrets, malicious packages, and agent action policies.
• Demand evidence, not accuracy percentages: source-to-sink data flow, call paths, and written classification reasoning. Then scan the same commit repeatedly to check repeatability.
• Prioritize workflow fit: findings sent to coding agents through MCP, Hooks, Skills, CLI, or API, plus incremental PR scans and policies that block only high-confidence findings.
• Run a 2- to 4-week proof of concept on 3 to 5 representative repos. Establish ground truth and safe look-alikes before any tool runs.

Why ADLC security needs its own checklist

ADLC stands for the agentic development lifecycle. In the ADLC, AI coding agents generate, modify, and submit code, often with access to your repos, terminals, and package managers. Some AI answers expand the acronym as "application development lifecycle." That's a different, older idea. This article uses ADLC to mean the agentic development lifecycle.

The ADLC differs from the traditional SDLC in two ways that matter for security:

• Volume. Agents write code at a pace humans can't review line by line. PR counts climb while reviewer headcount stays flat.
• Agency. Agents act. They call tools, install dependencies, edit CI config, and touch files across a repo. Each of those actions carries risk, separate from the code they produce.

ADLC security therefore covers both the code agents produce and the actions agents take. A DevOps-era checklist covers the first and mostly ignores the second.

Many teams are finding that AppSec tooling built on human-written code and scan-then-fix assumptions doesn't hold up here. A scanner that posts 400 findings to a dashboard after the build was a nuisance when humans wrote the code. When agents open dozens of PRs a day, that same scanner becomes a backlog no one will ever clear. Security needs to run where the code gets written, so issues surface while they're still cheap to fix and delivery keeps moving.

For the full concept, read our explainer on the secure agentic development lifecycle. The rest of this piece focuses on evaluation.

Before you talk to vendors: scope your agentic workflow

A vendor demo on a sample app tells you very little about your own environment. Write down what that environment looks like first, so every vendor gets tested against the same reality.

Document these items:

• Coding agents and IDEs in use. Include sanctioned tools and the ones developers adopted on their own, such as Cursor, Claude Code, or GitHub Copilot.
• Languages and frameworks. List the ones that carry real business risk, not every language in the monorepo.
• Repository count and structure. Note monorepos, polyrepos, and shared internal libraries.
• Weekly PR volume, and the share agents open or co-author. Even a rough estimate helps you size scan-time requirements.
• Where agents have write or execute access. Can they run shell commands, install packages, push branches, or edit pipeline config?
• Current tools and workflows. Note which CI/CD systems, ticketing tools, and policy engines a new tool must plug into.

To make the scale concrete, picture developers shipping 1,000 or more pull requests across hundreds of repositories. At that volume, manual security review on every PR is off the table. Your evaluation criteria need to assume the tool does most of the triage.

AppSec and platform engineering should own the criteria together. AppSec cares about what gets caught and whether it's real. Platform engineering cares about scan time, pipeline stability, and developer friction. If only one team writes the scorecard, the other team is likely to reject the result.

The ADLC security checklist at a glance

Ask each vendor to answer every question with a live demo on your code, not slides.

Coverage

• Can you find logic and access control flaws that cross several files and functions in our code?
• Can you show which open source and container vulnerabilities are actually reachable from our application?
• Do you validate secrets as live or revoked, and catch them before commit?
• Do you detect malicious packages and inventory AI models in our supply chain?
• Can you see and set policy on agent actions, such as package installs, CI edits, and tool calls?

Accuracy

• For any finding, can you show the data flow, call path, and a written reason for the classification?
• How did you measure your false positive and false negative rates?
• Do you get the same findings when you scan the same commit twice?

Workflow fit

• Can findings and fix guidance reach the coding agent while it works?
• Do incremental PR scans report only new findings, and how long do they take?
• Can policy block a PR only on high-confidence findings and route the rest by team?

Remediation

• Does every confirmed finding come with a code-specific fix?
• How do you predict whether a fix will break the build?

Governance

• Where is our code processed, by which model providers, and for how long?
• Is our code used to train any model?
• What reporting can we give leadership and the board?

Coverage: code, open source, containers, and the agents themselves

The classic categories still apply: SAST, SCA, container scanning, and secrets detection. What changes is the bar. AI-written logic tends to compile cleanly and pass tests while still containing authorization mistakes. With agent-driven PR volume, a tool that misses those flaws will miss a lot of them.

First-party code

Pattern-matching SAST has historically struggled with flaws like insecure direct object references (IDOR), broken access control, and insecure design. These bugs don't match a single bad pattern. They show up only when you follow data across layers, from a route handler through a service and into a data access call.

Test for this directly. Pick a flaw in your code, or seed one, where untrusted input travels across three to five architectural layers before it reaches a sensitive sink. Ask each vendor to find it and to show the full source-to-sink path. Cover the OWASP Top 10 categories that matter in your stack, not only injection.

Open source and containers

Every SCA tool can list vulnerable packages. The harder question is which of those vulnerabilities your application can actually reach. Ask vendors whether they can show reachability across each layer: from your first-party code, into the open source dependency, and down into the container's runtime and OS packages.

As an example of what that evidence can look like, Endor Labs uses full-stack reachability across first-party code, open-source dependencies, and container images. It builds a code context graph that models how the application behaves at the line level. Whatever tool you pick, ask it to show you the call path for a specific CVE in your repo. If it can only tell you a vulnerable package is present, your team will be triaging by hand.

Secrets and supply chain

For secrets, ask whether the tool scans full Git history, whether it validates a finding as live or revoked, and whether it can stop a leak at pre-commit. Validation tells you whether you're looking at a cleanup task or an active incident.

For the supply chain, ask about malicious packages, not only known CVEs. Agents install dependencies on request, sometimes from a package name the model guessed. Also ask whether the tool can inventory AI models pulled into your codebase.

Agent actions

Most AppSec checklists don't cover agent actions at all. Coding agents run commands, install packages, edit CI files, and call MCP servers and other external tools. Ask vendors:

• Can you see which agents, models, MCP servers, skills, and hooks are in use, including unapproved ones?
• Can you evaluate an agent's action before it executes, then allow, block, or log it?
• Can you intercept a package install from an agent before the package lands?
• Can you keep agents away from secrets, credentials, and sensitive files?
• Is there a single place to apply these policies across every agent your developers use?

If each agent needs its own configuration, policies will drift apart as teams adopt new tools. For more on why agent actions belong in an AppSec evaluation, see our piece on coding agent governance.

Accuracy: ask for evidence, not percentages

Every vendor has an accuracy number. Without a methodology behind it, the number doesn't tell you much. Before you accept any false positive or false negative rate, ask:

• Which codebases did you measure on? Real projects or synthetic test apps?
• Who decided what counted as a true positive, and how?
• Did you measure false negatives, or only false positives?
• Can we reproduce the measurement on our own repos?

Then move from aggregate claims to individual findings. For any finding, a credible tool should show:

• The source-to-sink data flow. Where does untrusted input enter, and where does it end up?
• The call path. How does your code reach the vulnerable function, especially in a dependency?
• A written explanation of the classification. Why did the tool call this a true positive or a false positive?
• An auditable trail. Can a reviewer read that reasoning and disagree with it?

Weak cryptography findings make a good test case. A naive scanner flags any use of SHA-1 as weak cryptography. But SHA-1 shows up legitimately inside ASN.1 DER encoding, where it's part of a standard structure, not a password hash. Endor Labs' AI SAST uses multiple triage agents that classify each finding as a true or false positive and write out why. In this case, the triage recognizes the DER context and marks the finding as a false positive. Ask every vendor to walk through a similar case on your code and show you the written reasoning.

Test repeatability

Many tools now use LLMs somewhere in their analysis. Models are nondeterministic, which raises a new question: will the same code produce the same findings tomorrow?

Run a simple test. Scan the same commit two or three times with no changes and compare the findings. Some variation in wording is fine. Findings that appear and disappear between runs are a problem, because developers stop trusting a tool that changes its answer.

Then ask the vendor which parts of the analysis are deterministic program analysis and which depend on model reasoning. Deterministic analysis, such as dataflow tracing and reachability, gives a stable base. Model reasoning works well for judgment calls like triage and logic flaws. A clear answer tells you how the vendor thinks about reliability.

For a deeper look at detection limits, see what static analysis catches and misses in AI-generated code.

Workflow fit: the pull request and the agent loop

A finding that sits in a dashboard doesn't fix anything. In the ADLC, workflow fit has two targets: the pull request and the coding agent itself.

The agent loop

The cheapest time to fix a flaw is while the agent is still writing the code. Ask vendors whether their tool can hand findings and fix guidance back to the coding agent during generation. Common integration points include:

• MCP servers that agents call to check code or dependencies as they work
• Hooks that evaluate agent actions before they run
• Skills that give agents secure coding guidance
• A CLI that agents and developers can run locally
• An API for custom workflows

Endor Labs, for example, integrates with coding agents through Hooks, Skills, MCP, CLI, and API. AURI by Endor Labs can be embedded in MCP-compatible coding agents so code gets scanned as agents work. Use this list as a reference for what to ask about, not as the only valid design.

The pull request

Agent-generated PRs arrive constantly, so scan time and noise both matter. Ask:

• Do incremental PR scans report only the findings this change introduced?
• What's the scan time on a typical PR, compared with a full repository scan?
• Does one PR scan cover code, changed dependencies, and new secrets, or do you need three tools?
• Are findings posted as PR comments where developers already work?
• Is there IDE and pre-commit support for developers who want feedback earlier?

For one example of how this works in practice, see the Endor Labs documentation on incremental PR scans.

Policy without friction

Blocking every PR with a finding trains developers to bypass the tool. Look for policy controls that block only on high-confidence findings and route everything else to the right team. Endor Labs uses OPA-based policies for this, so teams can block on confirmed, high-severity issues and route lower-confidence findings by team. Developers get fewer interruptions, and security gets faster decisions.

Remediation: fixes developers will actually merge

Research time per finding adds up quickly. A developer who has to read a CVE, trace the call path, and work out a safe upgrade can easily spend a morning on one finding. Multiply that by agent-driven PR volume and the backlog grows faster than any team can clear it.

Ask vendors:

• Does every confirmed finding come with a fix written for this codebase, not a generic advisory link?
• Can the fix be applied as a suggested change in the PR?
• For open source vulnerabilities, how do you predict whether an upgrade will break the build?
• If the safe upgrade is a major version jump, what are the alternatives?

Push hardest on the breakage question, since the answer shows whether a vendor relies on evidence or on version numbers. Endor Labs, for example, uses upgrade impact analysis to rank viable upgrade paths by the risk of breaking changes in your code. When an upgrade is too disruptive, backported patches fix the vulnerability without changing the dependency version. 

Whichever vendor you evaluate, ask them to show how they predict breakage, then check those predictions against your test suite during the proof of concept.

For more on how program analysis and agentic reasoning combine to produce fixes, see our article on finding and fixing flaws in AI-written code.

Governance, data handling, and vendor trust

When a vendor uses LLMs, your governance questions change. Your source code may leave your environment and pass through a third-party model. Get clear, written answers to:

• Where is code processed? In the vendor's cloud, your cloud, or on-premises?
• Which model providers are involved? Are they named in the vendor's subprocessor list?
• What's retained, and for how long? What's the deletion process?
• Is customer code used to train any model? The vendor's own or a provider's?
• What data residency options exist? This matters for regulated industries and connected products.
• What audit logs are available? Can you see who triaged what, which policy blocked a PR, and what an agent was allowed to do?

Use a shared vocabulary

The NIST Secure Software Development Framework (SSDF, SP 800-218) gives buyers and suppliers a common language. Map each checklist item to an SSDF practice. For example, vulnerability identification and response maps to the "Respond to Vulnerabilities" practice group. That mapping makes vendor answers easier to compare and easier to defend in an audit.

Reporting for leadership

Security executives need metrics they can report. Ask what the tool gives you for leadership and board updates, such as risk trends over time, mean time to remediate, policy exceptions, and agent activity. Look for numbers you can trace back to individual findings. A dashboard you can't explain in a board meeting won't help.

Run a proof of concept that settles the question

Demos show what a vendor wants you to see. A structured proof of concept shows what happens on your code. Plan for two to four weeks.

Step 1: Pick representative repositories

Choose three to five repositories that reflect your real risk. Include at least one service with complex authorization logic, one with a large dependency tree, and one that ships as a container. Include repos where agents write a meaningful share of the code.

Step 2: Build ground truth

Document known vulnerabilities, or seed them on a branch. Include a mix of:

• Logic flaws such as IDOR and broken access control that cross several layers
• Reachable and unreachable open source vulnerabilities
• Live and revoked secrets in history

Then add known non-vulnerable look-alikes: code that resembles a vulnerability but is safe, like SHA-1 inside DER encoding. These test whether a tool can tell the difference.

Public suites like the OWASP Benchmark and intentionally vulnerable apps help as a sanity check. Keep in mind that synthetic suites don't reflect real codebases. They're small, well known, and may appear in model training data. Treat them as a supplement to your own repos.

As a worked example of ground-truth methodology, Endor Labs' AI SAST benchmark ran across 8 real-world projects in Java, Python, JavaScript, TypeScript, C#, and Go. The Endor Labs security research team established ground truth for each project before scoring. AI SAST found 192 real vulnerabilities, roughly 3x what frontier models found. For your own PoC, the method is what to copy: real projects, multiple languages, and ground truth set by security researchers before any tool runs.

Step 3: Run every tool on the same commits

Pin specific commits. Run each tool against the same code, with the same configuration effort. Run PR scans on the same set of real or replayed PRs.

Step 4: Score the results

Use one scoring sheet for every tool:

Step 5: Decide together

Have AppSec and platform engineering review the results side by side. AppSec should weigh missed issues and evidence quality. Platform engineering should weigh scan time, fix quality, and friction. No single score decides. A tool that finds the most issues but takes 20 minutes per PR will lose the developers, and a fast tool that misses your seeded IDOR will lose security. Choose the tool both teams will sign off on.

Red flags in a demo

• The vendor only demos on its own sample app.
• The vendor can't explain how an accuracy figure was measured.
• Findings arrive without a data flow or call path.
• The vendor can't say which parts of the analysis use a model.
• Agent "support" means a dashboard report after the fact.
• The fix is a link to a CVE page.

FAQs

Is ADLC security a new tool category or an extension of existing AppSec tools?

It's an extension of existing AppSec tools that adds new requirements. The core categories (SAST, SCA, containers, and secrets) still matter. ADLC security raises the bar on accuracy, speed, and context, and it adds two areas older tools rarely cover: governing agent actions and sending findings back into the agent loop.

Do I need separate security tools for AI-generated code and human-written code?

Usually not. Code is code once it lands in a PR. You need a tool that handles agent-scale PR volume, catches the logic flaws AI-written code tends to introduce, and also covers what agents do. One tool that does all of that is simpler to operate than two that split the work.

How long should an ADLC security proof of concept run?

Two to four weeks is a reasonable window. That gives you time to set up ground truth on three to five repositories, run every tool on the same commits, observe real PR traffic, and score fix quality against your test suite.