AI Code Review Benchmark Results: How We Measure Ours (And Miss)
Our AI code review benchmark, published in full: 80% recall and 66.7% precision on a 25-fixture seeded-bug run, with methodology, the five misses listed by name, and how to run it yourself.
Our latest AI code review benchmark run found 80.0% of seeded bugs with 66.7% precision: 25 realistic fixture pull requests, one hand-authored bug planted in each, 20 bugs caught, 5 missed, 10 false positives flagged along the way. The full run, per-fixture tables, and the misses by name are public on the SuperDiffs benchmark page, rendered straight from the committed run JSON rather than retyped. We make SuperDiffs, an AI reviewer that verifies before it posts, and this piece explains how the benchmark works, why the numbers are middling on purpose, and why almost nobody else in the category publishes anything comparable. Context if you are shopping: our AI code review tools comparison and the CodeRabbit alternatives guide sit alongside this one.
The short version of why we bother: an AI reviewer you cannot calibrate is a reviewer you cannot trust, and the only honest calibration is a measured error rate on a corpus you can inspect.
How the seeded-bug benchmark works
The harness is public in the SuperDiffs repository. It contains 25 realistic PR diffs across Go, Java, JavaScript, Python, and TypeScript. Each diff carries exactly one hand-authored bug, drawn from classes that actually show up in production: SQL injection, command injection, race conditions, authorization bypasses, off-by-one errors, error swallowing, resource leaks, floating promises, loose type coercion, negative-amount handling, and others.
A run sends every fixture through the same review pipeline production uses, minus the GitHub posting. The run behind the current numbers used triage on one model and finding plus verification on another, cost $0.57 of total model time, and produced 30 findings across 25 fixtures. Scoring is mechanical: a finding counts as a catch when it matches the seeded bug's class and location, a miss when the fixture's bug produces no matching finding, and a false positive when a finding matches nothing planted. Matcher code and run JSON are both committed, and a test fails if the published page and the run ever disagree.
The results, in full
| Metric | Live run (matcher v2) |
|---|---|
| Recall | 80.0% |
| Precision | 66.7% |
| F1 | 0.727 |
| Anchor accuracy | 85.0% |
| Verified share | 100.0% |
| True / missed / false positives | 20 / 5 / 10 |
Per bug class, the spread matters more than the average. Race conditions, command injection, error swallowing, off-by-one, and SQL injection were all caught at 100%. Authorization bypasses sat at 50%, negative-amount handling at 50%, and two classes scored zero: a floating promise in an async notification path and a loose-equality type coercion. The benchmark page carries the complete per-class and per-fixture tables.
The five misses, by name
A benchmark that only reports averages is marketing. These are the fixtures the pipeline failed on the current run: async-floating-promise-notify, authz-missing-owner-check, input-unvalidated-integer-cast, money-negative-quantity-discount, and types-coercion-loose-equality. The list is derived from the scored JSON, not curated, and it is the honest answer to "what does this thing miss."
Why 66.7% precision is the headline, not the footnote
Most AI review vendors publish no accuracy numbers at all. The ones that gesture at quality talk about what their reviewer catches, never what it invents. But on a real team, a false positive costs a engineer's attention and a little trust, every single time. Ten false positives across 25 reviews is one invented problem for every two real ones found. That is the number we have to beat, and publishing it is how we know we are beating it.
There is a subtler honesty point on the verified-share row. Every matched finding in this run survived the second verification pass, which sounds great until you notice the verifier was the same model that found the bugs. A verifier that never disagrees with itself is partially a rubber stamp, and the benchmark page says exactly that. Independent verification, running the harness on a clean machine and publishing disagreement, is invited by email and would improve on everything here.
What this benchmark is not
It is not a cross-tool comparison. The page reserves an empty cross-tool table whose dataset does not exist yet, and it stays empty rather than borrowing this run's numbers, because comparing your own fixture corpus to competitors on different corpora is how benchmark pages lie. It is also not a fixed score: a published run is a snapshot of a pinned model and prompt. When either changes, the numbers change, and if a re-run is worse, the worse number is the one that gets published.
Run it yourself
The harness is public. Clone the repository, point it at your own model keys, and score your own reviewer, ours, or a competitor's, on the same 25 fixtures. If your results disagree with ours, publish them, and if you would rather we hosted yours alongside, write to support@superdiffs.com. Reproduction beats trust.
Frequently asked questions
What is a good precision for an AI code reviewer?
Higher than ours, honestly. At 66.7% precision, one in three posted findings is wrong. The defensible position is publishing the number at all: no major per-seat vendor does, so buyers currently shop on feature lists instead of error rates. Our goal is to push both numbers up run over run, in public.
Why seed bugs instead of reviewing real PRs?
Real PRs have no ground truth: you cannot score a catch without knowing the bug existed. Seeded bugs give every fixture a known answer, which makes recall and precision mechanically computable instead of vibes. The tradeoff is realism, which is why fixtures are written as realistic diffs in five languages rather than toy snippets.
Does the benchmark include competitors?
Not yet. The cross-tool table on the benchmark page is deliberately empty until a shared dataset and a fair gating method exist. Borrowing our single-tool numbers to fill it would be dishonest, so it stays blank.
How often is the benchmark re-run?
Whenever the model or prompt pin changes, and the old run stays up beside the new one. The benchmark page's rule is that a result is only replaced by a newer measured result, never by a better-looking one. The current run's methodology and every table live on superdiffs.com/benchmark, and the pricing page carries the review-depth settings the benchmark measures.