What a reviewer would check
Self-audit
Benchmarks get audited now, and the ones that survive publish their own gaps first. This page rates the release the way an independent review would, using the criteria Epoch AI's Benchmark Reviews programme applies: reviewability, four classes of score-affecting defect, and a set of evaluation-quality expectations. The statuses are ours. Anyone can check them against the repository.
The finding a reviewer would make first. The organisation that publishes this benchmark builds the leading system. The primary score comes from a frozen scorer, with equivalence rules, written by the same team after the runs. It raised both systems, but raised Proto by 60 attempts and Codex by 43, so a raw lead of 16 became a frozen lead of 34. An earlier snapshot in this repository's history (commit 87f624e) recorded four systems under the original grader with Codex ahead of three Proto cells on flash-tier models; two of those cells are not in this release and the Proto cohort here is a later runtime revision. None of this is hidden: raw and frozen verdicts sit side by side in the ledger, the earlier snapshot is in git, and the exclusions are listed in provenance. But it is the reason the result should be read as "what two configured systems delivered on this workload under this scorer" and nothing stronger.
1. Reviewability
| Criterion | Status | Evidence |
|---|
| All tasks inspectable | Full | 187 desk and 20 build tasks with generators, workspaces, checks, references, and traps in tasks/. |
| Scoring logic inspectable | Full | The frozen package (scoring/frozen-v7, 847 hashed files) with its base grader, task copies, and 7 equivalence graders; the runner's grader; per-check verdicts for every attempt. |
| Harness settings disclosed | Full | Model, provider route, reasoning effort, temperature or its absence, token cap, iteration cap, and time budget per cell in provenance and the adapter scripts. |
| Historical environment reproducible | Partial | Runtime revision and CLI version are recorded; the Proto runtime binary, container digests, and authenticated skill inventories are not distributed. Raw output workspaces are private; the ledger holds their hashes. |
2. Task correctness
| Criterion | Status | Evidence |
|---|
| Answerable as written | Mechanically checked | Every task's reference solution passes and the untouched workspace fails. This proves solvability, not that the ask is unambiguous to a reader who has not seen the checks. |
| Independent practitioner review | Not done | Most tasks were authored with model assistance and reviewed by the authors. No sample has been audited by anyone outside the authoring loop. A stratified 50-task audit is the first roadmap item. |
| Hidden requirements | Partly addressed | The equivalence graders exist because the original checks rejected valid alternative layouts on seven tasks. That is evidence the problem is real, and no systematic search for it has been run across the other 180. |
3. Grading validity
| Criterion | Status | Evidence |
|---|
| False negatives (overly strict) | Reduced, unquantified | The frozen scorer converted 60 (Proto) and 43 (Codex) raw failures into passes, 8 and 0 of them on the equivalence tasks and the rest through the frozen base grader. No human has re-read a sample of the remaining failures to estimate what is left. |
| False positives (lax or exploitable) | Partly guarded | The validator rejects value pins that appear verbatim in inputs; workbooks are recalculated natively so cached values cannot pass; no-error and live-formula checks guard pinned cells. The equivalence graders are more permissive by design and were not adversarially tested; no plausible-wrong-answer negative controls exist. |
| Scorer authored independently of the systems | No | The scorer, including the equivalence rules, was written by the team that builds Proto, after the outputs existed. Raw verdicts are published so the effect is inspectable, and the frozen package is hashed so it cannot drift further. |
| Grader determinism | Yes | Deterministic checks; recalculation pinned to a container image; a recorded replay of 1,341 attempts reported zero verdict differences. |
4. Benchmark consistency
| Criterion | Status | Evidence |
|---|
| Versioned tasks and scorer | Yes | Release manifest hashes every task and runner file; the frozen scorer manifest digest is b4720db00f2a55461ca70767a4baf8dd2d25aa9d301e8fe22ea13d21ccca358f. |
| Verdicts bound to a scorer identity | Yes | Every attempt carries a receipt hash, the scorer manifest hash, the original result hash, and output artifact hashes. |
| Comparable with the earlier snapshot | No | The earlier four-system snapshot used the original grader and a different Proto runtime. Its numbers and this release's are not on one leaderboard, and the site does not put them on one. |
| Cohorts contemporaneous | No | Proto: broad-desk-v43-c8f62dd60 r1; repeat-desk-v43-c8f62dd60 local r1/r2 map to logical r2/r3. Codex: full-1-codex-sol r1/r2/r3. Sampling controls differ: CLI-managed; not asserted matched to Proto. |
5. Elicitation adequacy
| System | Abnormal / 561 | Rate | Reading |
|---|
| Proto / DeepSeek V4.1 Flash | 3 | 0.5% | Clean |
| Codex / GPT-5.6 sol | 0 | 0.0% | Clean |
Involuntary termination (timeout or nonzero exit) is negligible in this release, and every frozen pass also completed normally. The earlier snapshot's flash-tier Proto cells terminated abnormally in 8 to 20% of attempts inside the same 1,200 second budget; those cells are not part of this comparison, and if they are rerun the budget question comes first.
6. Evaluation quality
| Expectation | Status | Evidence |
|---|
| Runs per system for statistics | 3 | Reviewers ask for five or more. A task-clustered bootstrap interval is reported as descriptive uncertainty; it is not a claim about unseen tasks. |
| Contamination assessment | Exposed set, low prior exposure | Inputs are generated fixtures, not scraped documents, and the repository was private until this release. It is now public, so the set is exposed. The task set was used during development of the leading system, which is disclosed rather than relabeled as generalization. Each generator takes a seed for sealed variants. |
| Human completability baseline | Not established | No human performance or time baseline exists for any sample. |
| Fair scaffold across models | Different by design | The unit is the configured system. A same-model, different-harness control pair is planned and not published. |
| Construct validity | Argued, not demonstrated | The paper argues why artifact contracts are the right target. No study yet links scores to owner acceptance in practice. |
| Conflict of interest | Present, disclosed | ERP.AI develops Proto and wrote the tasks, the runner, and the scorer. Independent authorship of the audit sample and scorer review is the mitigation on the roadmap. |
What changes before a review request
- Independent stratified audit of 50 desk tasks (seven per category) by reviewers outside ERP.AI, with per-task verdicts committed to the repository.
- Independent review of the frozen scorer's equivalence graders against adversarial wrong answers, and plausible-wrong negative controls for every check type.
- Human re-read of a sample of remaining failures from both systems to estimate the residual false-negative rate.
- A contemporaneous rerun: same task version, same scorer, five repetitions per system, plus a same-model harness control pair.
- Container digests and the runtime revision recorded per attempt by the runner.
- A human completability baseline on the audited sample.
Task disputes and grading errata are tracked as GitHub issues on erphq/business-bench. Accepted corrections change the task set version; they are never applied silently to a published snapshot.