The Sabotage Yardstick: Open Models Tie Sonnet on Vigilance, and Serving Weighs More Than Bits

A benchmark you have probably never heard of ran almost seventy complete coding campaigns on almost sixty models, cost its author past six thousand dollars, and answers a question the usual leaderboards dodge: not who writes the best code, but who notices that someone is poisoning theirs. It is akitaonrails' llm-coding-benchmark, and its v4 wrap-up, "the open source verdict", landed on 22 October.

The protocol is a Rails chat app built over seven sprints. Between sprints, a "malicious teammate" injects fourteen defects into the model's real code — some loud (a gem with a known CVE, an API key committed in plaintext), some silent (a cross-tenant data leak, an aggregate that quietly counts wrong, a unit test neutered to assert true). The model is never told anything. Fix a defect unprompted and it counts full weight; fix it after an explicit reveal and it counts forty percent; never fix it and you get zero. Tier A starts at 83. The evaluator grades through isolated subagents with cited evidence, and the artifacts — logs, ledgers, injection recipes — are all public in the repo.

The verdict

Open is in the pack. GLM 5.3 Flash and Qwen3.8 Flash-Next tie at 92.5 on a single rented B300; DeepSeek V4.1 Flash scores 92.5 in the cloud. Claude Sonnet 5, the closed model they are chasing, sits at 91.0 — below the two open ones. And a home PC takes a piece of this: Qwen3.8 Flash-Next quantized to IQ3_S, served by Strata, an expert-streaming MoE engine, on an RTX 5090 at a published 159.2 tok/s, scores 91.0 — tied with Sonnet — at zero marginal cost. I checked these figures at the evaluator's own files rather than trusting the blog table: the combined v4 report confirms 92.5 and 91.0, and the Strata field notes confirm the local run's Tier A.

What still separates the columns is consistency. 42 of 67 runs reached Tier A — but the open column carries D-tier scores (a Kimi K2 Thinking at 46, a GLM 4.7 Flash at 33.5) with no counterpart among the top closed models. A good open model is in the same band as a good closed one; the tail is where the risk lives.

The finding that travels further

Six models ran both through a cloud API and self-served from the official weights, and the scores moved more than any quantization step did. GLM 5.3 Flash scored 84.25 in the cloud and rose to 92.5 self-served on a B300. Nex N2.5 Pro fell from 94.0 to 27.0. Same checkpoint, different result. Tool parser, chat template, reasoning mode and engine version weighed as much as the number of bits — and the direction flipped per model. The author's own advice follows: use Q8 when it fits, drop to 4-bit when you must, and measure on your own workload before trusting anyone's leaderboard, including his.

One number deserves a footnote found at the repo, not the blog: the wrap-up publishes 159.2 tok/s for the RTX 5090 run, while the repo's field notes quote ~176 tok/s at full cache and ~147 with a 2.5 GB desktop VRAM reserve. Same order, different conditions — a small reminder that even a self-measured single number shifts with the room it was measured in.

Math is not coding

The post spends a section on why the West's formal-proof showcase — Erdős conjectures, Fermat in Lean — does not buy you vigilance in a Rails controller. The Fermat formalization Anthropic published ran on an internal model it describes as roughly comparable to Fable 5.1 — and that Fable 5.1 scored 95.5 here, below its predecessor's 100. Capability is trained by objective; the reward that lands a Lean proof is not the reward that spots a cross-tenant leak.

What to watch

The yardstick saturates at the top (seven models tie at 100) and its rerun noise reaches ten points — a repeat GPT 5.5 run scored 91 against its official 100. There is no independent replication, and the repo carries no license file. But as a decision aid for the one question it asks — will this model catch the sabotage it is never told about — it is currently the most honest table in the open-versus-closed fight, and it says: for vigilance, pick by consistency and by how you serve, not by the badge or the bit count.