Methodology in one paragraph
Real-world code is the ground truth. Clean merged PRs should pass (measures precision). Known, already-fixed bugs are re-introduced with git revert and ReviewGate reviews the "re-introducing the bug" diff — the original fix and its advisory are the ground truth (measures recall). Dimensions that rarely survive to a merged PR in the wild (performance, style) are validated with planted defects and clearly labeled as such. All runs use a real LLM endpoint (deepseek-v4-pro) and are reproducible from the committed datasets.
Real CVE recall — 5/5 blocked across 5 languages
Five real, published security fixes were reverted and the vulnerable diff reviewed. All five were blocked with a correct mechanistic explanation:
| Vulnerability (advisory) | Type | Language | Result |
|---|---|---|---|
| axios #10750 (GHSA-6chq-wfr3-2hj9) | Prototype pollution → FormData spoof → header injection | JavaScript | BLOCK (security, high, 0.86) |
| gradio #13437 (GHSA-qqr5-x4m8-g4gq) | Path traversal | Python | BLOCK (logic, high, 0.89) |
| echo #1718 | Directory traversal (Windows \ escaping path.Clean) | Go | BLOCK (logic, high, 0.86) |
| smallvec (RUSTSEC-2021-0003) | insert_many buffer overflow in unsafe code | Rust | BLOCK (logic, high, 0.81) |
| cJSON #800 | Heap out-of-bounds read (deletion-only diff: a bounds check removed) | C | BLOCK (security, high, 0.82) |
The cJSON case is notable: the diff contains no added lines at all — only a removed bounds check — and the review still reconstructed the out-of-bounds path. Raw logs: recall-cve-reverts.
Gold-standard real-PR reverts — 4/4 caught
Four real bug fixes (axios prototype-pollution SSRF guard, requests Content-Type parsing, gin ClientIP multi-header handling, ripgrep cross-root gitignore matching) were reverted with all default rules on. All four were flagged: 2 BLOCK, 2 WARN, each with the correct root cause identified. Raw logs: real-pr-revert-goldstandard.
AACR-Bench (official scorer) — precise but conservative
Scored with Alibaba's aacr-bench official evaluator (LLM semantic matching against the official reference set), 12 PRs across C / C# / C++ / Go:
| Configuration | Precision | Recall | F1 |
|---|---|---|---|
| Default dimensions (style excluded) | ~57% | 8.5% | ~15% |
| With style dimension (before it was made opt-in) | 33.3% | 8.5% | 13.6% |
Honest caveats, before you quote these numbers:
• This is not a controlled tool-vs-tool comparison with other published AACR scores — model backends differ, so cross-tool conclusions don't hold.
• n=12 with visible run-to-run variance; read trends, not decimals.
• The reference set is non-exhaustive (72% AI-assisted labeling); several unmatched ReviewGate findings were verified real, so precision is systematically underestimated.
• Recall of 8.5% reflects a deliberate design choice: ReviewGate is a conservative gate that folds low-confidence findings rather than flooding the PR. It reports little, but what it reports is high-confidence. Raising recall is an open research direction, documented in Limitations.
A useful side-result: the style dimension produced 10 findings and 0 semantic hits on real defects, so it was removed from the default dimension set (still available via --dimensions style). A quality gate should not block merges over style — that's a linter's job. Raw logs: aacr-bench-official.
Big-PR stress — up to 132 files, never a fake PASS
Real open-source PRs and multi-commit ranges, reviewed with deliberately tight token budgets to force unit splitting:
| Target | Language | Size | Units | Findings | Verdict |
|---|---|---|---|---|---|
axios HEAD~30..HEAD | JS | 55 files / ~5,000 lines | 8 | 12 | WARN + incomplete |
gin HEAD~80..HEAD | Go | 46 files / ~2,500 lines | 6 | 13 (incl. a real Slowloris DoS) | BLOCK + incomplete |
ripgrep HEAD~40..HEAD | Rust | 28 files | 4 | 6 (incl. a real hallucinated API) | WARN + incomplete |
curl HEAD~30..HEAD | C | 132 files / ~2,400 lines | 11 | ~0 (budget too tight — honest failure) | WARN + incomplete |
The curl row is kept on purpose: with a budget too tight for 132 dense C files, most units ran out of context — and the result was an explicit WARN + incomplete, not a silent pass. The invariant "any skipped, timed-out, or truncated unit degrades the verdict" held in every run. Details and mechanism: Big PR handling.
Coverage matrix
- ≥30 real PRs/commits across security, performance, logic, AI-smell, and intent review — every dimension validated on real code (planted defects used only where real merged PRs can't exercise the dimension, and labeled as such).
- 16 real business-logic issues: 13 BLOCK + 2 WARN.
- Languages exercised: JavaScript, TypeScript, Python, Go, Rust, C, C++, C#, and more via grep fallback.
Full matrix: production-readiness-30pr. Everything else lives in docs/evals/, including the runs that went badly.
Reproduce it
Datasets (dataset.tsv, dataset-recall.tsv), the AACR scoring script (scripts/eval-aacr-official.py), and per-run raw outputs are all in the repository. The revert method needs nothing but git and an LLM key.