Better models are making agent patch review more expensive, not less

Over the past few week, we ran two new frontier models through the Endor Labs Agent Security League: Anthropic's Claude Fable 5.1 on Claude Code, and OpenAI's GPT-6 Astra on Codex. Fable 5.1 reached 87.2% FuncPass and 37.4% SecPass, taking the top of the board. Astra landed just behind at 82.1% and 34.1%, a 14-point generational jump on both axes. If you're interested in the details, Luca has posted the detailed write-ups for both Fable 5.1 and Astra.

One reminder about the method, because it matters for both takeaways. Every task is real work in a real project, where the code involved was historically part of a security fix. The agent is never told a vulnerability exists, only to follow security best practices. FuncPass means the patch passes the project's functional tests. SecPass means it also passes the hidden tests that shipped with the original upstream fix.

There were a couple of things that stood out to me as worth writing into a separate blog, as I think they can be applied to both these, and previous results. If (when?) a model and harness combo comes along that upends these conclusions, I'll be sure to post about that too :-)

Takeaway 1: A green test suite tells you nothing about security

The headline gap, 87.2% functional vs. 37.4% secure, reads as "models are better at code than at security." True, but it undersells the problem. The number worth computing is the conditional one. Of the patches that passed every functional test, how many were also secure?

For Fable 5.1, 67 out of 156. Fifty-seven percent of patches that passed the full functional suite left the flaw exploitable.

Two things stand out. The generational gain is real: both new models moved the conditional rate from roughly 30% to roughly 42%, which is a step change rather than a rounding error. And they landed within 1.4 points of each other. Two labs, two harnesses, two training regimes, converging on the same figure. That looks less like a weakness in one model and more like a property of the problem.

Why the problem is hard is easiest to see in a specific case. One task Fable 5.1 solved addresses CVE-2023-31146 in the Vyper compiler, where a self-referencing array assignment stored the array's length word before copying the data, causing bounds checks to pass when they should have failed. The fix is an ordering invariant:

# note: wait until after the data is copied before clobbering the length

word.ret.append(copy_bytes(dst_, src_, len_, max_bytes))
ret.append(STORE(dst, len_))

Swap those two lines back, and every functional test in the project still passes. The compiler compiles, the output runs, and arrays behave correctly for every input anyone thought to write a test for. The out-of-bounds write is just sitting there for whoever goes looking.

That is the failure mode in miniature: plausible code rather than broken code. Test suites encode what the code is supposed to do, not what an attacker is supposed to be unable to do, so they are structurally poor at catching this class of defect.

If your review process for agent-generated patches is "CI is green, merge it," this data says you are flipping a coin. A slightly better coin than a year ago, which is progress, but not a foundation for policy.

Takeaway 2:  You can't ignore the costs 

There is a price attached to that coin flip, and it keeps falling. Fable 5.1's full run cost $672 against Opus 5's $1,116, roughly 40% less for a higher score. The more interesting part is where the money went.

Fable 5.1 read 334M cache tokens and wrote 6.24M output tokens, a ratio of about 53 to 1. Opus 5 read 1,348M against 9.37M written, about 144 to 1. Almost none of what these agents cost goes on producing code. They spend it re-reading their own accumulated context, once per tool call.

That makes turn count the dominant cost variable. Opus 5 used 75 tool calls per task against Fable 5.1's 42, roughly 1.6 times the turns, and burned four times the cache reads. Each additional turn replays a context that has itself grown, so spend rises faster than the turn count. Fable 5.0 makes the same point from the other direction: it read more cache tokens than 5.1 (437M against 334M) for half the SecPass. Wandering costs more than solving, and 5.0's figure is a lower bound, because 16 aborted runs went unrecorded.

Timeouts are the same effect at full price. Opus 5 timed out on 15 tasks and Astra on 29, each one billed in full and returning nothing. A wall clock cap is an economic control as much as a scheduling one.

Counted by outcome rather than by attempt, $672 bought 67 verified secure fixes, about $10 each, against roughly $19 for Opus 5. A 47% drop in the cost of a confirmed fix, in one model generation, a few months apart.

Now hold that next to takeaway 1. You cannot tell which 67 of those 156 functional passes are the secure ones without checking, which means the review queue is all 156. Assume half an hour a patch and a loaded engineer cost of $100 an hour, and that run generated something like $7,800 of human attention against $672 of compute. Better than ten to one, and compute is the side of the ratio getting cheaper. Human reading speed is fixed.

Any remediation program that plans to review what the agent produces therefore has a cost structure that gets worse as the models get better. Cheaper generation means more patches per sprint, and the review queue absorbs every one of them at the same price per item as last year.