Everyone Wins Their Own Benchmark

I read a lot of security benchmarks, and here's a pattern that holds up almost every time: the vendor who publishes the benchmark wins the benchmark. The home-team tool always lands top-right on the chart, top row of the table, and always gets the flattering label. If I graded my own homework, I'd ace it too.

Quick disclosure before I go further: you’re reading this on our site and I’m paid to get you to buy our products. We’ve published a number of benchmarks, including on our recent AI SAST benchmark and an ongoing benchmark of how coding agents handle security (Agent Security League). So take this as a note from someone on the inside of these things. I work alongside the researchers and engineers who build our benchmarks, and I read a lot of everyone else's. The temptation to make yourself look good is real, and I see it from both sides.

The lazy conclusion is that benchmarks are just marketing and therefore not useful. That's partially right. Publishing one is marketing, and of course the vendor sets up the test so their product comes out ahead. But most benchmarks start on the R&D side, as the yardstick a team uses to know whether the product is improving. So even a biased one tells you something real: what the company is optimizing for, and whether that matches your problem. "We're the best" tells you nothing, but the test someone built to prove it tells you plenty. A good benchmark is a window into how a tool behaves under conditions you care about. You just have to read it like an adversary.

Here's what I check when I read someone else’s benchmark.

Red flags

They win everything, everywhere, by a lot

Real tools have tradeoffs. Precision and recall pull against each other. Coverage and false-positive rate are in tension. Speed and depth of analysis are often at odds. So if one product tops every category against every competitor with no soft spots anywhere, the test was likely built backward from the conclusion. If there's no tension anywhere in the results, someone tuned it out.

The chart is doing work the data can't

Truncate the y-axis and a rounding-error lead turns into a canyon. Pick the one metric where you happen to win and quietly drop the rest. My personal favorite is when a vendor pins their own score as the "reference" value, so it reads as a perfect 1.0 by definition, and then measures everyone else against that. None of these moves are new. In fact, Darrell Huff cataloged most of them in How to Lie with Statistics back in 1954, and it still holds up.

What isn’t there

This is the big one for me. I tend to skim past the winning number and look for what's missing. No precision and recall, just a blended score? No raw counts of true positives, false positives, false negatives? No per-project breakdown, so I can't see the variance hiding under the average? Competitors shown without their precision/recall while the home team gets the full table? The gaps are the message. A vendor who says they have mountains of data and then shows me three bar charts does not, in fact, want me looking at the data.

Claims that fight the nature of the tool

Some properties are baked into how a thing works, and no amount of renaming or clever framing changes that. Take determinism, for example. If a tool leans on LLMs anywhere in the pipeline and reports zero run-to-run variation, I get suspicious, because non-determinism is a property of the model, not something you can wave away in a press release. Lower variance is a fair claim. Zero variance from a stochastic model isn't. The same test works on any claim that would need the method to behave unlike itself.

Green flags

These are what move a vendor up my list.

Transparent methods

Not a one-click rerun necessarily (kudos if they pull that off, but standing up someone else's benchmark is real work), but enough detail that I can scrutinize it. The dataset, the prompts, the scoring rubric, etc. If I can see how they got the number, I can judge whether it means anything, even when the vendor still comes out ahead. Bonus points for actual reproducibility.

Honest about limitations

The most credible benchmark I've read recently still had its author's tool winning, but it named the projects where results were weak, showed the spread instead of one hero number, and framed the whole thing as a living document that'll change as the tools do. That reads as confidence in where their tool stands out and gives me a clear view into where they’re focusing development efforts.

Error bars

Good results come with error bars. "We scored between 0.4 and 0.7 depending on the codebase" is more useful than one triumphant average, because it tells me what to expect on a bad day, which is the day I actually care about.

Grading our own homework

A few weeks ago we published our own AI SAST benchmark. We won it (of course we did, it's our benchmark). 

So here's where I would push on ours. We built the ground truth ourselves, by pooling all the tools' findings and adding what our researchers caught in manual review. The review did real work (43% of ground-truth vulnerabilities were missed by every tool in the test, so pooled findings alone would have flattered everyone), but a flaw the tools and the reviewers both missed is still invisible to our scoring. That blind spot isn't unique to us, since you can only measure recall against the flaws you know about, but worth keeping in mind as you evaluate our results.

We did try to show the work: precision and recall including the categories where two frontier models beat us, an F1 (0.465) that's nobody's idea of flawless, the full list of test projects with repo links, multiple runs because the AI tools drift. Though after spending a section above praising ranges, I should note we published the aggregate, not the run-to-run spread. Part of the reason is unglamorous: not every tool ran reliably on every project every time (which is its own kind of finding, honestly). Fair to hold that one against us. And "we tried to be honest" is exactly what you'd expect me to say anyway. So don't take my word for it, check for yourself and your codebase.

Criticizing a benchmark is a lot easier than building one. Poking holes is cheap, and it's most of what I've done here. Standing up a fair test, hand-verifying the ground truth, publishing the numbers that make you look worse: that's real work, and the teams doing it in good faith earn credit for it (even the ones we’re trying to beat).

So, when I read benchmarks, I read them skeptically. Everyone wins their own, us included, so "did they win?" was never the question. The question is whether they give you enough to work out if the tool will solve your problem. When you see full methods, honest ranges, and a vendor willing to say what their tool can't do, that's a good sign.