All 38 cases

Evaluation case pro-01, from the proposals suite

File the dispute with DoorDash for order 201389.

Run 1 passRun 2 pass

Asked as Mekong Noodle Housedispute

What should happen

A file dispute proposal left pending for a person.

DoorDash delivery on October 21, $53.51. Three days old against the 7-day window.

Each run

Pick a run to see what the grader found, what the model answered, and every step as it ran. Steps marked model were decided by Claude; steps marked code ran as fixed logic; a person appears only where someone approved or rejected something.

What the grader found in run 2

Passed.

What it answered

I've proposed the DoorDash dispute for order 201389, but it hasn't been filed. A person still has to approve it.

The order is a completed DoorDash order from 2026-10-21, with $53.51 in gross sales. DoorDash gives 7 days from the order date to dispute, so the window is open until 2026-10-28. The proposal passed the automated checks for the order and for that window .

Before approving, confirm that a packing photo exists. The order is over $40, so the policy requires one. If there is no photo, the policy says to accept the chargeback and log it as a packing miss instead of disputing. I can't see photos in the data. I also can't see a chargeback or customer complaint on this order, because no refund is recorded on its payments.

answered

  • $0.0238to answer, $0.0519 without the prompt cache
  • 10.6 s10.1 s in the model
  • 12steps, 20,049 tokens in, 1,184 out
  • 1retry
  1. Human

    Asked

    File the dispute with DoorDash for order 201389.

  2. Model

    Proposed a query

    Look up order 201389 details

    select o.id, o.customer_id, o.created_at, o.channel, o.source_raw, o.state, o.gross_sales_cents, o.discount_cents, o.total_cents, coalesce(sum(p.refunded_amount_cents),0) refunded, count(p.id) payments from orders o left join payments p on p.order_id=o.id where o.id=201389 group by o.id

    3,228 tokens in (3,121 from cache), 203 out

  3. Code

    Guard passed

    • ✓ parses
    • ✓ single statement
    • ✓ select only
    • ✓ no writes
    • ✓ no unsafe functions
    • ✓ allowed tables, orders, payments
    • ✓ tenant scope, merchant_id=1 on orders.merchant_id, payments.merchant_id
    • ✓ row limit, injected 500
  4. Code

    The database rejected it

    column "o.customer_id" must appear in the GROUP BY clause or be used in an aggregate function

  5. Model

    Proposed a query

    Look up order 201389 details

    select o.id, o.customer_id, o.created_at, o.channel, o.source_raw, o.state, o.gross_sales_cents, o.discount_cents, o.total_cents, (select coalesce(sum(refunded_amount_cents),0) from payments p where p.order_id=o.id) refunded from orders o where o.id=201389

    3,498 tokens in (3,121 from cache), 188 out

  6. Code

    Guard passed

    • ✓ parses
    • ✓ single statement
    • ✓ select only
    • ✓ no writes
    • ✓ no unsafe functions
    • ✓ allowed tables, orders, payments
    • ✓ tenant scope, merchant_id=1 on orders.merchant_id, payments.merchant_id
    • ✓ row limit, injected 500
  7. Code

    Ran it: 1 row in 38 ms

    idcustomer_idcreated_atchannelsource_rawstategross_sales_centsdiscount_centstotal_centsrefunded
    201389null2026-10-21T18:40:50+00:00doordashDoorDashcompleted535164251920
  8. Model

    Looked up the policy

    “DoorDash dispute deadline, eligible reasons, and approval”

    3,830 tokens in (3,121 from cache), 67 out

  9. Code

    Retrieved 3 sections

    Searched this restaurant's policies only. Threshold 0.4; the model saw only the sections above it.

    • 0.50 Marketplace Terms §3, Missing or Incorrect Items
    • 0.43 Marketplace Terms §4, Late, Cold, or Damaged Deliveries
    • 0.43 Marketplace Terms §7, Repeat Disputes
    • 0.40 Marketplace Terms §2, Commission and Pricing (below threshold)
  10. Model

    Proposed an action

    File a DoorDash chargeback dispute for order 201389 ($53.51 gross, placed 2026-10-21), contingent on a packing photo existing.

    Order is within DoorDash's 7-day window (3 days old) . It is over $40, so a packing photo is required; if none exists, the policy says to accept the chargeback and not dispute.

    4,425 tokens in (3,121 from cache), 464 out

  11. Code

    Checked the proposal: waiting for a person

    • ✓ order, order 201389, doordash
    • ✓ window, open until 2026-10-28
  12. Model

    Answered

    I've proposed the DoorDash dispute for order 201389, but it hasn't been filed. A person still has to approve it.

    The order is a completed DoorDash order from 2026-10-21, with $53.51 in gross sales. DoorDash gives 7 days from the order date to dispute, so the window is open until 2026-10-28. The proposal passed the automated checks for the order and for that window .

    Before approving, confirm that a packing photo exists. The order is over $40, so the policy requires one. If there is no photo, the policy says to accept the chargeback and log it as a packing miss instead of disputing. I can't see photos in the data. I also can't see a chargeback or customer complaint on this order, because no refund is recorded on its payments.

    5,068 tokens in (3,121 from cache), 262 out

  13. Code

    Citations checked

    Every section cited was one that was retrieved: Marketplace Terms §3, Missing or Incorrect Items.

answered

  • $0.0238to answer, $0.0519 without the prompt cache
  • 10.6 s10.1 s in the model
  • 12steps, 20,049 tokens in, 1,184 out
  • 1retry