Memo to the crews on the 2026 policy changes
policy-update-memo · timeout 1200 s · source
The ask
Sent verbatim as the prompt. The agent works in a copy of the workspace below and sees nothing else.
We've rewritten the crew policies for 2026. Please write the memo that goes to all the crews telling them what changed - both versions are in the folder and Carol's email has the notes. Save it as memo.md.
Workspace
- crew_policies_2025.md
- crew_policies_2026_FINAL.pdf
- crew_policies_2026_draft_v1.pdf
- email_from_owner.txt
Expected deliverables
- memo.md
This task also has a task-specific grader (check.py).
Checks
The task passes only when every required check passes. Reference answers, tolerances, and the planted traps are in the repository, not here.
| Check | Type | Required | Failed (frozen) / 6 |
|---|---|---|---|
| memo.md exists | file_exists | yes | 0 |
| overtime approval | text_sentence_matches | yes | 0 |
| per diem | text_sentence_matches | yes | 0 |
| tip split | text_sentence_matches | yes | 1 |
| mileage | text_sentence_matches | yes | 0 |
| damage reporting | text_sentence_matches | yes | 0 |
| moved clause and effective date | custom | yes | 2 |
Recorded results, complete desk comparison
pass pass, frozen scorer only fail timed out
| System | Repetitions | Frozen / 3 | Raw / 3 | Median minutes | Mean cost (USD) |
|---|---|---|---|---|---|
| Proto / DeepSeek V4.1 Flash | 3 | 0 | 0.6 | 0.0084 | |
| Codex / GPT-5.6 sol | 1 | 0 | 1.0 | 0.2397 |
Per-attempt check verdicts, raw and frozen verdicts, usage, receipt and artifact hashes for this task are in results/latest/attempts.jsonl under "task": "policy-update-memo".