Business Bench · a verifiable benchmark for agents doing business work
Business work is the domain where agent correctness can actually be measured.
Code benchmarks have exact verifiers, a narrow domain, and documented contamination and reward hacking. Work benchmarks have economic breadth and grade by rubric, checkpoint, or expert preference, so they measure plausibility. Business work has long-horizon, many-document inputs and a deliverable with a ground truth. Business Bench uses that to measure conjunctive correctness, repeat reliability, and self-verification with executable checks only: no rubric, no LLM judge, no partial credit.
What it measures
A task t declares required checks Ct over the files the agent leaves behind: exact identifier sets, keyed values, workbook figures after native recalculation, sentence-level text rules, task-specific graders. An attempt passes iff every check in Ct passes. Each task runs k = 3 times in a fresh workspace. Four numbers follow:
- pass@1, the mean attempt pass rate, and pass^k, the fraction of tasks passed in all k repetitions. The second is what an owner experiences.
- Check rate c̄, passed required checks over all required checks, and the conjunctive gap c̄ − pass@1: the credit a partial-credit grader awards for work the owner would have to redo.
- Near-miss share, the fraction of failures one check away from passing: how often the agent had what it needed and did not verify.
- Cost per pass, from captured usage at list price, beside wall time and token work.
Every task is a generator with a seed and a planted truth, so the same task exists as sealed variants and at several difficulty bands, and the whole set can become an environment with dense exact rewards. Verdicts are bound by receipts to a hashed scorer; raw and frozen verdicts are both published.
Findings from the current release
Seven findings with their computation and caveats are on the findings page. 116 of 187 tasks are passed by both systems every time; the current set separates them on about a fifth of its tasks, which is why v2 adds difficulty bands and three new capability axes.
Where it sits
| Benchmark | Verifier | Domain | What it cannot tell you |
|---|---|---|---|
| SWE-bench Verified | Unit tests | Python repositories | Rated Flawed by Epoch AI (Sept 2026): contamination, and an OpenAI audit finding flawed tests in a majority of the audited sample. |
| Terminal-Bench 4.0 | Test scripts | Shell tasks | Epoch counts 30 of 66 tasks with scoring defects, including a verifier pipe an agent can write "pass" to. |
| GDPval | Expert pairwise preference | 44 occupations | Which deliverable a professional prefers, not whether it is correct. |
| TheAgentCompany | Checkpoints with partial credit | Simulated software company | Partial credit hides the conjunctive gap; some checkpoints are LLM-assessed. |
| τ-bench | Database state | Customer-service tool calls | Introduced pass^k, which this benchmark adopts; single-domain, dialogue-shaped. |
| SpreadsheetBench | Cell-level exact match | Single spreadsheets | One file, one question; no cross-document state, no process. |
| Business Bench | Executable checks on artifacts, conjunctive | Business deliverables and applications | Whether a result generalises past this workload; see the self-audit for every other limit. |
Complete desk comparison
Pass rate over all 561 attempts per system under the shared frozen scorer. Failures and timeouts stay in the denominator. The systems use different models and were not run at the same time, so this compares two configured systems, not a harness or a model in isolation.
| System | Frozen pass rate | Passed / 561 | Raw / 561 | All 3 of 3 / 187 | Mean cost |
|---|---|---|---|---|---|
| Proto / DeepSeek V4.1 Flash | 507 | 447 | 147 | $0.04 | |
| Codex / GPT-5.6 sol | 473 | 431 | 141 | $0.39 |
Proto leads by 34 attempts under the frozen scorer and by 16 under the original grader. One task passed nowhere; 116 passed everywhere. Per-task matrix, category splits, resource use, and the scorer's effect on every verdict are on the results page.
Read before comparing. ERP.AI publishes this benchmark and builds Proto, the leading system. The frozen scorer, including its equivalence rules, was written by the same team, and it raised both systems' scores relative to the original grader. Raw verdicts are published beside frozen ones so the effect is inspectable. An earlier four-system snapshot under the original grader remains in the repository history. All of this is set out on the self-audit page.
Two handoffs, one principle
Desk track: a folder and a short ask
The agent receives business-shaped exports, spreadsheets, PDFs, and an email or two, then must leave the requested deliverables. Inputs are messy on purpose: merged headers, currency strings, mixed dates, a requirement buried in a thread. Where the files cannot support an answer, the correct deliverable says so.
| Spreadsheet | 41 | Reconciled, calculated, or reshaped workbook |
| Bookkeeping | 36 | Reconciliation, schedule, transaction classifications |
| Reports | 30 | Data summary and source-grounded memo |
| Reformatting | 26 | Destination-compatible import file |
| Extraction | 26 | Structured records extracted from documents |
| Drafting | 19 | Business text preserving required facts and rules |
| Tooling | 9 | Small file-based tool or static page |
Build track: an application, then three changes
The agent must hand over a working application a tester can reach and log into, import the seed data, enforce roles and row-level scope on the server, show a live dashboard, and survive a restart. Then it accommodates three change requests without losing earlier requirements.
Twenty applications: CRM, inventory, field service, leave, purchase approvals, invoicing, memberships, events, assets, helpdesk, vendors, expenses, quotes, donors, appointments, property maintenance, recruiting, work orders, point of sale.
No build campaign has completed its human acceptance pass. The task packs are released and validated; build scores are not published. Browse the build tasks.
Mechanics
Executable checks, conjunctive
A task passes only when every required check passes: exact identifier sets, keyed values, live spreadsheet formulas recalculated in LibreOffice, sentence-level text rules, and task-specific graders. A percentage of checks is not a pass.
One frozen scorer, hashed
Every attempt in the release was scored by the same immutable package. Each verdict is bound by a receipt to the original result hash, the output-file hashes, and the scorer manifest, so a rescoring can never be confused with a rerun.
Reliability beside accuracy
Three repetitions per task. The site reports attempt-level pass rate and the share of tasks passed all three times, alongside wall time, token usage, and estimated model cost.
The full paper is on the paper page; the protocol and grader are summarised under methods; the setup and run commands are under reproduce.