All 38 cases

Evaluation case pol-14, from the policy suite

A DoorDash customer called us saying their pho arrived cold and wants a refund. Can we refund them?

Run 1 passRun 2 pass

Asked as Mekong Noodle Housemarketplacenear-miss

What should happen

Answered, citing Marketplace Terms §4, Late, Cold, or Damaged Deliveries.

House Policy §3, Refunds, scores high and looks like the answer; it covers in-store and online orders and sends marketplace orders elsewhere. The rule that applies is in the marketplace terms: refer the customer to the platform, no refund from our side.

Each run

Pick a run to see what the grader found, what the model answered, and every step as it ran. Steps marked model were decided by Claude; steps marked code ran as fixed logic; a person appears only where someone approved or rejected something.

What the grader found in run 2

Passed.

What it answered

No, we shouldn't refund this one. Under , a cold dish on a marketplace order after courier pickup is the platform's cost, not ours. If a customer complains to us, we refer them to DoorDash support. We don't issue a refund or a credit from our side. The only exception in that section is a Grubhub order marked ready too early, so it doesn't apply to a DoorDash order.

I haven't looked up the specific order. Tell me which one it is if you want me to check it.

answered

  • $0.0057to answer, $0.0169 without the prompt cache
  • 3.3 s3.1 s in the model
  • 4steps, 7,223 tokens in, 249 out
  1. Human

    Asked

    A DoorDash customer called us saying their pho arrived cold and wants a refund. Can we refund them?

  2. Model

    Looked up the policy

    “refund for marketplace order complaint food arrived cold DoorDash”

    3,245 tokens in (3,121 from cache), 67 out

  3. Code

    Retrieved 4 sections

    Searched this restaurant's policies only. Threshold 0.4; the model saw only the sections above it.

    • 0.59 Marketplace Terms §3, Missing or Incorrect Items
    • 0.58 Marketplace Terms §4, Late, Cold, or Damaged Deliveries
    • 0.47 Marketplace Terms §7, Repeat Disputes
    • 0.45 Definitions §5, Marketplaces
  4. Model

    Answered

    No, we shouldn't refund this one. Under , a cold dish on a marketplace order after courier pickup is the platform's cost, not ours. If a customer complains to us, we refer them to DoorDash support. We don't issue a refund or a credit from our side. The only exception in that section is a Grubhub order marked ready too early, so it doesn't apply to a DoorDash order.

    I haven't looked up the specific order. Tell me which one it is if you want me to check it.

    3,978 tokens in (3,121 from cache), 182 out

  5. Code

    Citations checked

    Every section cited was one that was retrieved: Marketplace Terms §4, Late, Cold, or Damaged Deliveries.

answered

  • $0.0057to answer, $0.0169 without the prompt cache
  • 3.3 s3.1 s in the model
  • 4steps, 7,223 tokens in, 249 out