Benchmark
14 pull requests, 3 runs each.
Last updated 30 October 2026
Small repositories with a known base and head. Some hide a silent break; the rest change only what their pull request says. Each went through the real pipeline, Nemotron on Token Factory and every check in Nebius Sandboxes, several times over, because each run pins its own checks.
The numbers
- Hit rate on silent breaks
- 79 %
- False alarms on announced changes
- 0 %
- Inconclusive
- 0
- Model cost, all runs
- $0.86
19 of 24 runs ended BROKEN
0 of 18 runs ended BROKEN
of 42 runs, counted against both rates
14 min 6 s on the clock, 3 runs at a time
A hit is a run on a silent-break case that ends BROKEN. A false alarm is a BROKEN verdict on a case where every change is announced. Cost is the estimate from model tokens; sandbox time is free while Sandboxes is in beta.
Case by case
| Case | Expected | Verdicts | Result |
|---|---|---|---|
| backoff-cap-announcedThe default cap drops from 30 s to 10 s and the pull request says so; delays under the cap do not change. | conserved | conserved · conserved · conserved | 3/3 correct |
| backoff-jitter-at-capOriginal code, patterned on a common refactor slip: folding two steps into one expression moves the cap in front of jitter, so late retries no longer get jitter. The repo suite only uses jitter on early attempts. | broken | broken · broken · broken | 3/3 caught |
| camel-long-numbers-announcedThe same feature written exactly as announced (two or more digits only), so short numbers like e2e do not change. | conserved | conserved · conserved · conserved | 3/3 correct |
| camel-short-numbersModeled on sindresorhus/camelcase#54: a fix for letters after numbers turned e2e into e2E. Original code, not camelcase. The text limits the rule to numbers of two or more digits, the regex does not. | broken | broken · broken · broken | 3/3 caught |
| cart-threshold-onlyThe tiny-cart change without its silent discount rounding slip: only the announced threshold moves. | conserved | conserved · conserved · conserved | 3/3 correct |
| duration-cut-to-roundModeled on sindresorhus/pretty-ms#76, where a value just under a unit boundary rounded up across it. Original code, not pretty-ms. The refactor swaps cutting for rounding; the repo suite only uses exact tenths. | broken | broken · broken · broken | 3/3 caught |
| duration-rounding-announcedSame code change as duration-cut-to-round, but the pull request says exactly what changes (cut to rounded), so every flip is announced. | conserved | conserved · conserved · conserved | 3/3 correct |
| hopperThe flagship fixture: a refactor drops the grounded check around friction, so friction acts in mid-air. The repo suite and the pull request text both miss it. | broken | broken · conserved · conserved | 1/3 caught, 2 missed |
| paginate-past-the-endOriginal code, patterned on a common refactor slip: a shared validation helper arrives together with a dropped clamp, so a page past the end gets an inverted range. The repo suite never asks for one. | broken | broken · broken · broken | 3/3 caught |
| slug-acronym-pluralsModeled on sindresorhus/slugify#79: the fix for plural acronyms narrowed a character class and stopped splitting APIUser. Original code, not slugify. | broken | broken · broken · conserved | 2/3 caught, 1 missed |
| slug-underscore-announcedEvery slug changes from hyphens to underscores and the pull request says so; the judge must call each flip intended. | conserved | conserved · conserved · conserved | 3/3 correct |
| tiny-cartA free-shipping threshold change is announced; the same diff silently changes discount rounding from round to floor. | broken | broken · conserved · conserved | 1/3 caught, 2 missed |
| uniq-filter-refactorA pure refactor that really keeps behaviour (first duplicate wins). The input is spread before filter, so holes in sparse arrays and non-array iterables come out as they did from the loop; any alarm here is a false alarm. | conserved | conserved · conserved · conserved | 3/3 correct |
| uniq-keeps-lastOriginal code, patterned on a common refactor slip: a Map built in one pass quietly keeps the last duplicate instead of the first. The repo suite only uses duplicates that are identical. | broken | broken · broken · broken | 3/3 caught |
Where it falls short
In every miss, each pinned check either still held on head or flipped in a way the judge read as announced: the proof ran, but the behaviour that slipped was never pinned, or it hid behind an announced one.
hopperrun 2, conserved: All 4 pinned behaviours were checked: 4 still hold on head.hopperrun 3, conserved: All 3 pinned behaviours were checked: 3 still hold on head.slug-acronym-pluralsrun 3, conserved: All 18 pinned behaviours were checked: 10 still hold on head, and 8 changed exactly as the pull request announced.tiny-cartrun 2, conserved: All 8 pinned behaviours were checked: 7 still hold on head, and 1 changed exactly as the pull request announced.tiny-cartrun 3, conserved: All 12 pinned behaviours were checked: 10 still hold on head, and 2 changed exactly as the pull request announced.
Run it yourself
The cases live in evals/cases with their base, head and pull request text. One command runs them all and writes a markdown and a JSON report; this page is built from the newest JSON. The runner stops before it spends more than $1.50 a round.
python -m evals.run --runs 3 --runner contreeThis round ran at commit 6271398. Full report: evals/results/20261030-134637.md.