On October 10, 2026, researchers from the National University of Singapore and Alibaba Group submitted a paper to arXiv with a title that inverted the usual question about AI coding. The benchmark was called the Last Secure Code Benchmark (LSCBench), and its central finding was blunt: six frontier coding agents passed between 73.6% and 91.3% of functional tests, but even the strongest agent, Claude Fable 5, produced code that was both functional and secure on only 25.1% of tasks.
The 91/25 gap drew the most attention. It described a problem that every engineering organization deploying AI coding tools already suspected: the code works, but it does not survive contact with an adversary.
What received less attention was the second-order question the benchmark raised. If the AI can write code that passes functional tests but fails security tests, what about the AI that reviews it? The reviewer is also a model. It is trained on the same data, optimized for the same objectives, and evaluated by the same kinds of benchmarks. If the generator has a 91/25 problem, what does the reviewer have?
The answer, according to a growing body of independent benchmarks, is that the reviewer has a different problem — and in some ways, a more consequential one.
What LSCBench Actually Measured
The design of LSCBench matters for understanding why the 91/25 gap exists.
The benchmark built 450 tasks from 150 real vulnerabilities. Each task gave the agent a specification that asked only for functionality and never mentioned security. The prompt for a Java configuration importer, for example, asked the agent to implement the file import feature. It did not ask for a feature that resists YAML deserialization attacks.
The agent that failed — Kimi-K3 — used SnakeYAML's defaultYaml() constructor, which honors every !!class tag in untrusted YAML. All 14 functional tests passed. All 4 security tests failed. The bug was a single constructor choice. Claude Fable 5 and GPT-5.6-Sol used SafeConstructor and passed both suites.
The benchmark graded on two hidden suites: functional tests and security tests that replayed the original attack. No partial credit. No LLM judge. One attempt per task. The agents were run at their highest reasoning effort.
The result was consistent across models. Functional pass rates ranged from 73.6% to 91.3%. Functional-and-secure pass rates ranged from 16.4% to 25.1%. The gap between "it runs" and "it is safe" was roughly three to one for every agent tested.
That gap is the generator's problem. The reviewer's problem is whether it can detect it.
The Reviewer's Benchmark
The AI code review market has produced its own benchmarks, and they tell a story that is different from LSCBench but equally uncomfortable.
GitHub's ReviewBench, released on October 5, 2026, tests review agents on 219 pull requests from 187 public repositories across 19 programming languages. It measures how many known issues each agent finds, how many findings are valid, and how performance varies by severity and category, including security issues. The benchmark reports precision, recall, and F1, with an AI judge assessing findings outside the reference set.
Cecuro's OpenSSF CVE Benchmark tests review tools against 82 real vulnerabilities. The results are scattered. Cursor Bugbot caught 66 of 82. Ozone caught 63. DeepSource caught 48. Claude Code caught 35. CodeRabbit caught 16.
A separate independent benchmark run across 309 pull requests put GitHub Copilot's review bot at 54% bug detection, CodeRabbit at 44%, and Greptile at 82%. Another benchmark of four tools against 18 security and logic defects found that Codzee detected all ten additional security defects, Greptile detected seven, CodeRabbit five, and CodeAnt four.
The numbers are not consistent across benchmarks. They are not supposed to be. Each benchmark uses a different dataset, a different definition of "detection," and a different scoring methodology. What they have in common is that no reviewer catches everything, and most reviewers catch less than half of what the benchmark considers a security-relevant issue.
The reviewer's job is harder than the generator's. The generator sees a specification and writes code. The reviewer sees code and has to infer the specification, identify the weakness, and articulate the fix. The generator can fail by omission; the reviewer can fail by omission, misdiagnosis, or false positive.
A benchmark that measures whether the reviewer blocked a pull request is not the same as a benchmark that measures whether the reviewer identified the vulnerability that made the pull request unsafe.
The Verdict–Diagnosis Gap
That distinction is the subject of a separate 2026 paper from researchers at the University of Bristol and other institutions. The paper introduces MalPR-Bench, a benchmark of 89 malicious pull requests and 50 benign controls across 44 repositories and eight language families. Its central finding is what the authors call the Verdict–Diagnosis (VD) gap: a reviewer can correctly block a malicious pull request for the wrong reason.
"A block may be triggered by an unrelated issue rather than the vulnerability that makes the PR unsafe," the authors write. "Such a review appears successful under verdict-based evaluation, yet can misdirect remediation and leave the actual vulnerability unaddressed."
The paper's solution is PRGuard, a reviewer that separates vulnerability identification from evidence validation. On 31 held-out malicious pull requests, PRGuard and CodeRabbit achieved nearly identical blocking performance. But PRGuard identified 1.38 times as many target vulnerabilities. On absence-type pull requests — where a required guard or ownership check is missing — PRGuard identified three times more vulnerabilities than CodeRabbit.
The VD gap is not a minor technicality. It means that a review that looks successful — the pull request was blocked — may have failed at the task that matters: telling the developer what was wrong and how to fix it.
The reviewer benchmark that measures only "did it block?" is measuring the verdict. The benchmark that measures "did it diagnose?" is measuring the thing that actually prevents the next vulnerability.
The 91/25 Problem in the Reviewer
The LSCBench findings and the reviewer benchmarks describe the same structural problem from two directions.
The generator produces code that passes functional tests and fails security tests. The reviewer is supposed to catch the failure. But the reviewer is evaluated, in most commercial benchmarks, by whether it flags issues — not by whether it catches the specific security defect that the generator missed. The reviewer's benchmark and the generator's benchmark are not aligned. They measure different things.
A reviewer that catches style violations and typos can score well on a benchmark that counts all findings equally. A reviewer that catches only the critical security flaw but misses the style issues can score poorly on the same benchmark. Neither benchmark tells you whether the reviewer would have caught the Kimi-K3 YAML deserialization bug.
The LSCBench authors made a design choice that the reviewer benchmarks have not yet replicated. They asked the generator to write code from a specification that never mentioned security. Then they tested the code against the specific weakness that the original vulnerability represented. The benchmark was constructed so that the generator could not game it, and so that the security test was specific to the task.
A reviewer benchmark built on the same principle would give the reviewer a pull request with a specific, known vulnerability and ask: did you identify this vulnerability, and did you explain why it is dangerous? Not "did you find some issue." Not "did you block the merge." Did you catch the thing that matters?
That benchmark does not yet exist as a standard industry tool. MalPR-Bench and PRGuard are steps toward it. LSCBench is a step toward understanding why it is necessary.
What the Gap Means for Buyers
The practical implication for enterprises evaluating AI code review tools is that the benchmark landscape is not yet mature enough to support procurement decisions on its own.
A tool that scores well on GitHub's ReviewBench may score poorly on Cecuro's CVE benchmark. A tool that catches 82% of bugs in one independent test may catch 44% in another. A tool that blocks a high percentage of malicious pull requests may diagnose only a fraction of the vulnerabilities those requests contain.
The benchmarks are useful for establishing a floor. They are not useful for establishing a ceiling. And they are not yet designed to answer the question that matters most: if an AI coding agent produces code that is 91% functional and 25% secure, will this reviewer catch the 66% that is functional but insecure?
The answer depends on the reviewer, the vulnerability, and the benchmark. The data does not yet support a general answer.
What Comes Next
The next generation of reviewer benchmarks will need to do what LSCBench did for generators: construct tasks from real vulnerabilities, specify the weakness, and grade on whether the reviewer identified it. The diagnosis, not the verdict, is what prevents the next vulnerability.
Until those benchmarks exist, enterprises are left with a patchwork of vendor claims, independent tests with different methodologies, and the uncomfortable knowledge that the 91/25 gap in the generator is not being systematically measured in the reviewer.
The reviewer is a model trained on the same data and subject to the same optimization pressures, with the same tendency to satisfy the specification it is given. If the specification is "find issues," it will find issues. If the specification is "find this vulnerability," it might find it. The difference is the benchmark.
Sources: Last Secure Code Benchmark (LSCBench), arXiv:2610.14766, submitted October 10, 2026, by researchers at the National University of Singapore and Alibaba Group, via explainx.ai (October 14, 2026) and DEV Community (October 16, 2026); GitHub ReviewBench via Help Net Security (October 6, 2026); Cecuro OpenSSF CVE Benchmark via cecuro.ai (2026); independent 309-PR benchmark via DEV Community (August 26, 2026); 18-defect security and logic benchmark via DEV Community (October 12, 2026); MalPR-Bench and PRGuard, arXiv:2608.25730 (August 26, 2026).
Disclaimer
The information provided in this article is for general informational and educational purposes only. It does not constitute legal, financial, or professional advice. The author and publisher are not responsible for any actions taken based on the content of this article. Readers should consult qualified professionals for advice specific to their situation. All trademarks and references to third-party products, services, or organizations are the property of their respective owners. The performance data and benchmarks discussed are based on specific research studies and may not generalize to all use cases or environments. As of the publication date, the AI landscape continues to evolve rapidly, and readers should verify current information independently.
Limitations
This analysis is based on preprints, benchmark publications, and vendor documentation available as of the article date.
LSCBench and MalPR-Bench are arXiv preprints and have not completed peer review; findings should be read as directional.
Benchmark results vary with dataset, prompt design, and scoring methodology; cross-benchmark comparisons are indicative only.
Vendor detection rates are reported by third parties and were not independently reproduced.
Precision and recall figures depend on the reference answer key, including any self-inclusion issues noted in the text.
Reporting standards differ across benchmarks; figures may not be directly comparable.
This sector is changing quickly; claims can become outdated soon.
Sources
- Last Secure Code Benchmark (LSCBench), arXiv:2610.14766, submitted October 10, 2026, by researchers at the National University of Singapore and Alibaba Group, via explainx.ai (October 14, 2026) and DEV Community (October 16, 2026)
- GitHub ReviewBench via Help Net Security (October 6, 2026)
- Cecuro OpenSSF CVE Benchmark via cecuro.ai (2026)
- independent 309-PR benchmark via DEV Community (August 26, 2026)
- 18-defect security and logic benchmark via DEV Community (October 12, 2026)
- MalPR-Bench and PRGuard, arXiv:2608.25730 (August 26, 2026).
The information provided in this article is for general informational and educational purposes only. It does not constitute legal, financial, or professional advice. The author and publisher are not responsible for any actions taken based on the content of this article. Readers should consult qualified professionals for advice specific to their situation. All trademarks and references to third-party products, services, or organizations are the property of their respective owners. The performance data and benchmarks discussed are based on specific research studies and may not generalize to all use cases or environments. As of the publication date, the AI landscape continues to evolve rapidly, and readers should verify current information independently.
Limitations: This analysis is based on preprints, benchmark publications, and vendor documentation available as of the article date.; LSCBench and MalPR-Bench are arXiv preprints and have not completed peer review; findings should be read as directional.; Benchmark results vary with dataset, prompt design, and scoring methodology; cross-benchmark comparisons are indicative only.; Vendor detection rates are reported by third parties and were not independently reproduced.; Precision and recall figures depend on the reference answer key, including any self-inclusion issues noted in the text.; Reporting standards differ across benchmarks; figures may not be directly comparable.; This sector is changing quickly; claims can become outdated soon.