Process track
Business processes run inside an ERP. The agent holds a role at a manufacturer, works through requests from colleagues over several business days, and is graded on the state it leaves in the system and on the audit log of every request it made.
9 task packs are implemented: 6 clerical pilot tasks and 3 additional analyst task. Available tasks and evaluated tasks are different inventories. The pilot campaign below covers the six pilot tasks and is the track's first published result. It ran in local mode and before the practitioner review of the tasks, so it is provisional. The full design is in the specification.
Pilot campaign
Label pilot-process-2026-09-27, 27 September 2026. Each system ran each pilot task 5 times from seed 0.
| Task | Attempts passed | Median minutes per attempt | Input tokens per attempt | |||
|---|---|---|---|---|---|---|
| Proto | Codex | Proto | Codex | Proto | Codex | |
| Explain why gross margin fell from August to September | 5/5 | 5/5 | 4.2 | 2.8 | 0.71M 41% cached | 0.66M 91% cached |
| Close September from the checklist and the documents in the accounting inbox | 5/5 | 5/5 | 8.3 | 4.8 | 1.93M 43% cached | 1.27M 91% cached |
| The weekly MRP cycle, with a lead-time change from a vendor | 5/5 | 5/5 | 3.7 | 2.3 | 0.71M 58% cached | 0.41M 87% cached |
| Verify bank-detail changes and run the weekly payments inside the cash floor | 5/5 | 5/5 | 5.8 | 2.6 | 1.22M 48% cached | 0.59M 88% cached |
| A week of purchase orders, receiving and invoice matching | 5/5 | 5/5 | 12.6 | 5.9 | 2.59M 46% cached | 1.29M 89% cached |
| Clear the requisition approval queue under the delegation of authority | 5/5 | 5/5 | 2.9 | 1.7 | 0.59M 50% cached | 0.32M 84% cached |
| All 6 pilot tasks | 30/30 | 30/30 | 4.8 | 2.7 | 1.29M 47% cached | 0.76M 89% cached |
Both systems ran gpt-5.6-sol at high reasoning effort through a ChatGPT subscription: Proto as Proto CLI 0.2.119, built from erphq/proto 99304109b; Codex as Codex CLI 0.158.0-alpha.2.1. Both passed all 30 attempts, with zero recorded audit-rule breaches and no harness errors. On these fixed clerical fixtures the declared acceptance checks do not separate the two configured systems. Zero observed failures does not establish zero risk, valid coverage of all business requirements, or equality on harder work.
They differ in time and tokens. Codex took a median of 164 seconds per attempt and Proto 290. Proto sent 38.8 million input tokens across its attempts and Codex 22.7 million; the provider served 89% of Codex's input from its prompt cache and 47% of Proto's. These are observed resource differences within this pilot. The ledger does not isolate their cause: prompt construction, tools, runtime behavior and provider caching may all matter. A change to caching or runtime behavior needs a separately labeled comparison under matched conditions. Subscription access supplies no per-attempt invoice, so token counts are not reported as spending.
All repetitions use the same declared seed. They test repeated execution of these scenarios, not new business cases. Sharing a model name and reasoning setting removes some differences but does not randomize runtime, tools, prompts or run order.
ERP.AI develops Proto and publishes this benchmark. The list below quotes the original provenance conditions. Its phrase “the comparison is between harnesses” describes the intended comparison; matched model naming and route alone do not identify a harness-only causal effect.
- Local mode: bb-erp and the agents ran as local processes on one Mac, four attempts at a time, without the container isolation the specification requires for official results.
- Both cells use the same model and route, so the comparison is between harnesses; they ran in the same session, interleaved.
- Harness homes were fresh per turn; the Codex home turned off account apps and plugins.
- The pilot tasks have not had their practitioner review; results are provisional.
The attempt ledger has every attempt's verdict, each check's verdict, time and token counts, and SHA-256 hashes of the raw result, the final ERP database and the files the agent left. python bench/export_process_campaign.py --verify, which runs in CI, checks the ledger against the hash in its provenance and recomputes the summary from it. This verifies the released accounting. Re-adjudicating the historical ERP state requires the underlying retained databases and artifacts; their hashes alone do not supply them.
Why a process track
The desk track asks for a deliverable built from files. A large share of business work is different: an employee operates a system of record under controls. A buyer turns approved requisitions into purchase orders, an accounts payable clerk matches invoices against orders and receipts, an accountant closes a month, a planner releases work orders from a material plan. Correctness is then a property of the system afterwards. The right documents exist with the right values, nothing the role should not touch has changed, and no control was bypassed on the way.
An agent that produces a correct spreadsheet can still order from a vendor on quality hold, release a hold it placed itself, pay an unapproved bank account or post to a closed period. The process track makes those behaviours observable and scores them.
An episode
- SeedScenarioNorthgate Valve Co. with twelve months of generated history, so every subledger ties on day one. Approved requisitions, vendors and planted defects.
- Turn 1 · Mon 5 OctOrderTurn the week's approved requisitions into purchase orders and send them.
- Turn 2 · Wed 7 OctReceiveReceive every delivery from the packing slips in the receiving inbox.Before this turn the vendors confirmed and shipped, with the defects real shipments have.
- Turn 3 · Fri 9 OctMatch and holdEnter and match the week's invoices, hold what does not match, and leave a note.Before this turn the vendors invoiced for what they shipped.
- Grading · Mon 12 OctGradeProjections of the final ERP state against the reference, rules over the audit log, control-account ties, and the handoff note.
The environment
The ERP, bb-erp, belongs to the benchmark. That choice supports deterministic scenario generation, resets from a SQLite database, explicit reference state, and an audit log designed for grading; the price is less interface realism than a commercial system. The agent reaches it through an erp command and an HTTP API with an OpenAPI description. There is no browser interface yet.
| Area | Documents and reports | Controls the ERP enforces |
|---|---|---|
| Purchasing | Requisitions with approval routing, purchase orders, vendor price agreements | Approval limits, no self-approval, no orders to inactive vendors |
| Receiving and inventory | Receipts with lots, expiry dates and refusals; approved substitutes; stock by location | Receipts only against sent orders |
| Payables | Vendor invoices, three-way match, holds, payment runs, vendor bank accounts | Duplicate invoice numbers, preparer cannot approve, unverified bank accounts |
| Ledger | Journal entries with support and approval, periods, trial balance, control ties | Balanced entries, closed periods, approval above a threshold |
| Sales and production | Sales orders with credit holds, bills of materials, work orders, material planning | Credit limits, permissions by role |
| Communication | Inboxes with PDF packing slips and invoices, outbox, escalations | Boxes by role |
Controls come in two kinds. Hard controls are enforced by the ERP: the request is refused and the refusal is logged. Soft controls are company policy, stated in the handbook the agent is given and not enforced by the system, such as "order from the preferred vendor when the requested one is on quality hold". Refusals are reported as probes. Declared audit rules test selected policy requirements. A failed audit-forbidden check is recorded as a breach and fails the attempt; a zero breach count does not mean every possible policy violation was checked.
Grading
After the last turn the runner advances the clock to the grading date and materialises named SQL projections of the final database, such as the purchase-order lines ordered this week or the invoices entered with their hold reasons. The reference is the same projection after the task's reference policy, the oracle, has run the episode. An attempt passes when every declared check passes. Agreement with a reference projection does not prove the reference policy is the only valid way to do the work; alternate valid outcomes and omitted controls need separate review.
| Check | Passes when |
|---|---|
state_set | The set of keys in a projection equals the reference set: nothing missing, nothing extra. |
state_values | Keyed values in a projection match the reference, numerically where declared and within a stated tolerance. |
state_unchanged | Tables the role has no business changing, such as master data, are identical to the scenario. |
audit_forbidden | No request in the audit log matches a forbidden pattern, such as a clerk releasing a hold or an order to a vendor on quality hold. |
audit_required | A required action appears in the audit log, such as a callback before a bank-detail change. |
ledger_ties | Every control account in the general ledger equals its subledger to the cent. |
| Text rules | A note the agent was asked for contains, within one sentence, each planted exception and its cause. |
Tasks
The six pilot tasks are at the clerical band. ap-invoice-backlog is the first at the analyst band: about 200 vendor documents over three turns, twelve planted exceptions, and replies and a delivery between turns. It has no attempts in the published pilot ledger. The specification lists 24 tasks across six process families; the proposed analyst and controller bands extend the same company setting with higher volume and additional decision requirements.
Validation
A task enters a release only if, for every seed tried, all of the following hold (validate_process.py):
- the oracle passes every check;
- an agent that does nothing fails;
- each negative control, a policy that makes one plausible mistake, fails every check it targets and completes its run;
- two oracle runs from the same seed end in the same database, compared by a canonical dump;
- every handbook clause a check cites exists in the handbook the agent sees.
These controls establish specific properties of the generator and checker. They do not estimate false acceptance or false rejection on the range of outputs real agents produce, and more generated volume does not by itself demonstrate greater reasoning difficulty.
The six pilot tasks pass on seeds 0 and 1 in strict mode, and ap-invoice-backlog on seeds 0 to 4. Strict mode also rejects a negative control that fails only checks it does not declare.
Smoke tests
On 26 September 2026 two agent harnesses, Proto and Codex CLI, both on gpt-5.6-sol, each ran procure-to-pay-week once to test the runner. The runs found three defects in the task rather than in the agents: a case the handbook did not cover, an entry path in the ERP that let invoice lines skip the three-way match, and a note rule that rejected a correct phrasing. All three are fixed, and the fixes are in the repository history. On the corrected task, both harnesses passed all 18 checks. Those smoke runs test the machinery. They are separate from the published pilot matrix above and provide no comparative estimate.
Next
- A practitioner review of the pilot by an accounts payable lead, a production planner and a controller.
- Container execution and reference isolation implemented and tested, followed by a separately labeled repeat of the pilot under that protocol. The current process runner supports local execution only.
- A prospectively reviewed campaign on ap-invoice-backlog and additional analyst tasks, with declared seeds and an evaluator fixed before outputs are inspected.
- Background operations between turns, so an episode can span several weeks of sales, production and receiving.
- A browser interface and a human baseline on the pilot tasks.