Reply to a customer about a cracked skillet
refund-apology-letter · timeout 1200 s · source
The ask
Sent verbatim as the prompt. The agent works in a copy of the workspace below and sees nothing else.
Laura Rogers emailed us about a cracked skillet and a late delivery. Please draft my reply to her, sticking to our refund policy, and save it as response.md.
Workspace
- email_customer_complaint.txt
- orders_export_2026-08-24_to_2026-09-02.csv
- returns_and_refunds_policy.md
- support_saved_replies_2025.txt
Expected deliverables
- response.md
This task also has a task-specific grader (check.py).
Checks
The task passes only when every required check passes. Reference answers, tolerances, and the planted traps are in the repository, not here.
| Check | Type | Required | Failed (frozen) / 6 |
|---|---|---|---|
| response.md exists | file_exists | yes | 0 |
| correct order number | text_contains_all | yes | 0 |
| refund amount per policy | text_sentence_matches | yes | 0 |
| processing time | text_sentence_matches | yes | 0 |
| no promise the policy forbids and no wrong order number | custom | yes | 4 |
Recorded results, complete desk comparison
This is one of the 7 tasks with a task-specific equivalence grader in the frozen scorer, which accepts alternative valid representations the original checks rejected. Raw and frozen verdicts can therefore differ here by design; see what the frozen scorer changed.
| System | Repetitions | Frozen / 3 | Raw / 3 | Median minutes | Mean cost (USD) |
|---|---|---|---|---|---|
| Proto / DeepSeek V4.1 Flash | 1 | 0 | 0.7 | 0.0089 | |
| Codex / GPT-5.6 sol | 1 | 1 | 0.7 | 0.1304 |
Per-attempt check verdicts, raw and frozen verdicts, usage, receipt and artifact hashes for this task are in results/latest/attempts.jsonl under "task": "refund-apology-letter".