Tasks · desk · Bookkeeping
Tie out the Keystone supply statement to our books
vendor-statement-reconcile · timeout 1200 s · source
The ask
Sent verbatim as the prompt. The agent works in a copy of the workspace below and sees nothing else.
Keystone's August statement is in the folder with our ledger for them. Reconcile the two before we pay, Luis's note has how he likes it done. Save it as vendor_recon.csv.
Workspace
- ap_vendor_ledger_keystone.csv
- keystone_statement_2026-08.pdf
- note_statement_recon.txt
Expected deliverables
- vendor_recon.csv
This task also has a task-specific grader (check.py).
Checks
The task passes only when every required check passes. Reference answers, tolerances, and the planted traps are in the repository, not here.
| Check | Type | Required | Failed (frozen) / 6 |
|---|---|---|---|
| note's columns | csv_columns | yes | 0 |
| one row per open item | csv_set_equal | yes | 0 |
| row count | csv_row_count | yes | 0 |
| status per document | csv_values_match | yes | 0 |
| difference per document | csv_values_match | yes | 0 |
| variance ties to the balances | custom | yes | 0 |
Recorded results, complete desk comparison
pass pass, frozen scorer only fail timed out
| System | Repetitions | Frozen / 3 | Raw / 3 | Median minutes | Mean cost (USD) |
|---|---|---|---|---|---|
| Proto / DeepSeek V4.1 Flash | 3 | 3 | 1.8 | 0.0244 | |
| Codex / GPT-5.6 sol | 3 | 3 | 1.3 | 0.1895 |
Per-attempt check verdicts, raw and frozen verdicts, usage, receipt and artifact hashes for this task are in results/latest/attempts.jsonl under "task": "vendor-statement-reconcile".