Sonnet 5 vs Fable 5: reliable versus security-forward, not better versus worse

Introduction

Over the last few months, we've been running AI coding models through our Agent Security League, pairing each model with two different agent harnesses, Cursor and Claude Code, on the same 200-task dataset. We now have all four combinations for our two most interesting models: Cursor + Sonnet 5, Claude Code + Sonnet 5, Cursor + Fable 5, and Claude Code + Fable 5.

That's enough to finally put Sonnet 5 and Fable 5 head-to-head. But a naive comparison of the raw FuncPass/SecPass numbers doesn't work, because two very different confounds distort the raw scores in different cells: how often a model cheated (recycled a memorized answer instead of deriving it), and how often a run timed out and shipped a partial or empty patch. The Cursor + Sonnet 5 run in particular was dominated by a timeout problem, a story we dug into separately in our companion piece on why Cursor ran so much slower for this model, and why that reversed our usual harness ranking.

This post is about the models, not the harnesses. Below, we work around both confounds using three dimensions that are structurally immune (or much more resistant) to them, and ask the only question that matters here: setting aside cheating and timeouts, which model actually reasons better, and at what?

Sonnet 5 vs Fable 5: reliable versus security-forward, not better versus worse

With all four combos in hand, we can put Sonnet 5 and Fable 5 head-to-head directly:

As we detail in the companion piece, cheating and timeout effects are the two major confounding factors that influence a direct comparison of the raw FuncPass/SecPass:

• Cursor + Sonnet 5: unusually high timeout rate driving partial/empty patches.
• Cursor + Fable 5: high cheating rate but still a very high raw SecPass.
• Claude Code + Sonnet 5: low cheating, moderate timeouts, reasonable across-the-board scores.
• Claude Code + Fable 5: low scores driven by a very high cheating rate.

Because the dominant confound is different in every cell, a naive 4-cell comparison mostly measures which confound hits harder, not which model reasons better. Here we work around that by using three dimensions that are structurally immune (or much more resistant) to those two specific confounds.

Spoiler: Neither model is simply "better." Sonnet 5 is more reliable and thorough, higher FuncPass almost everywhere, via more investigation per instance and a lower cheating rate. Fable 5 is more security-forward when it isn't cheating, higher SecPass almost everywhere, via less investigation and a higher cheating rate. It's a real trade, not a coin flip, and it holds across both harnesses.

Dimension 1: fair FuncPass/SecPass, restricted to "both models actually finished"

Method: for each harness, exclude the union of instances where either Sonnet 5 or Fable 5 timed out (removing the timeout confound entirely), then compute the cheating/strict-test-adjusted ratio on what's left (removing the cheating confound too).

Reading it:

• FuncPass: a wash on Cursor (93.0 vs 93.7, noise), but Sonnet 5 keeps a real ~10.5-point edge on Claude Code even with both confounds removed.
• SecPass: Fable 5 leads on both harnesses once timeouts and cheating are controlled for, by 19.1 points on Cursor, 3.6 points on Claude Code. This is the first time SecPass points in the same direction on both harnesses.

Caveat: "honest-attempt" removes cheating-confirmed instances from both numerator and denominator (i.e., "given an honest attempt, how often is it correct"). It does not penalize a model for cheating more often in the first place, that's Dimension 2, kept deliberately separate rather than blended into one number.

Dimension 2: cheating rate as its own axis

Method: how often each combo's anti-cheating check confirmed a model recycled the answer instead of deriving it, as a share of the full 200-task set.

Fable 5 cheats 1.7x (Cursor) to 4.75x (Claude Code) more often than Sonnet 5. Combined with Dimension 1, the honest picture is coherent rather than contradictory:

• Fable 5 reaches for a memorized/recycled answer more often, but when it doesn't, it's more likely to have actually implemented the specific security-relevant fix.
• Sonnet 5 is more disciplined about deriving its own answer, and that answer functions correctly more often, especially on Claude Code.

Dimension 3: process signal from raw trajectories (tool-call behavior)

Method: every raw trajectory (cursor_stdout for Cursor, claude_stdout for Claude Code, including all resume rounds, concatenated) was mechanically parsed for tool calls: investigation (Read/Grep/Glob), implementation (Edit/Write), and shell/Bash calls, including whether a test-running command (pytest, tox, trial, etc.) appears after the first edit. Restricted to the same "both models completed" instance set as Dimension 1.

Both n=175 Claude Code rows draw from the same 175-instance keep set (neither model timed out); Fable 5's n is 173 rather than 175 because 2 of its runs there are hard run failures (claude_success: false, empty patch, ~4.6KB init-only log) rather than timeouts, no trajectory content exists to parse for those 2, so they're excluded from this table's averages specifically. However, they remain correctly scored as incorrect in Dimension 1's honest-attempt ratio above.

Reading it, this holds on both harnesses and helps explain the patterns above:

• Sonnet 5 does substantially more work per instance, roughly 1.5x the total tool calls of Fable 5 on both harnesses, almost entirely in the investigation phase (more reading/searching relative to edits), not in verification (both models run tests after editing at a similar, high rate: 81–87%).
• More upfront investigation of the existing code's contract plausibly means fewer functional mistakes, which likely helps drive Sonnet 5's consistent FuncPass edge. It also has a cost on Cursor specifically: more work per instance means more exposure to Cursor's slower tool-calling loop for this model, which is why Sonnet 5 times out on Cursor so much more than Fable 5 does.
• It does not explain Fable 5's higher SecPass rate, if anything, it argues against the naive "more investigation → more security awareness" intuition. Fable 5 appears to reach the specific security-relevant code change more directly, with less exploration, than Sonnet 5 does, a pattern-matching/prior-driven tendency rather than an investigation-driven one, consistent with Fable 5's also-higher cheating rate: both point toward Fable 5 leaning more on recalled/prior patterns (sometimes crossing into actual cheating, sometimes landing on a genuinely derived but less-explored correct fix) rather than first-principles exploration of the specific codebase.

Conclusion

The cleanest read here is not that one model “wins,” but that the two models are optimizing for different things. Sonnet 5 is the more reliable system: it does more investigation, runs through more of the codebase, and comes out ahead on FuncPass once you strip away the timeout and cheating confounds. Fable 5 is the more security-forward system: when it is actually attempting the task rather than recycling an answer, it more often lands on the security-relevant change and does so with less exploration.

That’s why the raw 4-cell comparison is so misleading. Once you control for the confounds, the pattern is stable across both harnesses: Sonnet 5 looks more disciplined and thorough, while Fable 5 looks more direct and security-oriented. In other words, this is not a story about a better model and a worse one. It’s a story about two different strengths, and the tradeoff between them is real.