All 38 cases

Evaluation case pol-02, from the policy suite

How long do we have to dispute a missing-item chargeback on a DoorDash order?

Run 1 failRun 2 fail

Asked as Ridgeline Coffeecross-tenantnot-on-marketplace

What should happen

Answered, citing Online Ordering and Loyalty Policy §2, Delivery Marketplaces.

What the trace shows

A retrieval miss. The section that says Ridgeline is on no marketplace scored 0.28 against the model's searches, far under the 0.50 threshold, because a question about DoorDash dispute deadlines does not read like a sentence saying there are no marketplaces. Shown nothing twice, the model said the policies do not cover it and suggested the DoorDash portal, which is the wrong advice for a restaurant that is not on DoorDash.

Each run

Pick a run to see what the grader found, what the model answered, and every step as it ran. Steps marked model were decided by Claude; steps marked code ran as fixed logic; a person appears only where someone approved or rejected something.

What the grader found in run 2

  • cited House Policy §5, Online Orders but did not cite Online Ordering and Loyalty Policy §2, Delivery Marketplaces

What it answered

The policies I can search don't give a deadline for disputing a missing-item chargeback on a DoorDash order. The only section that came back was . It covers Square Online pickup orders and doesn't apply to DoorDash. A second search for marketplace dispute terms returned nothing relevant.

I won't guess at a number of days. Check DoorDash's merchant terms or your DoorDash merchant portal for the exact window. Because the deadline may be short, file as soon as you see the chargeback. If you give me the order, I can pull its details and propose a dispute for you to approve.

answered

  • $0.0067to answer, $0.0235 without the prompt cache
  • 4.9 s4.2 s in the model
  • 6steps, 10,147 tokens in, 322 out
  1. Human

    Asked

    How long do we have to dispute a missing-item chargeback on a DoorDash order?

  2. Model

    Looked up the policy

    “DoorDash dispute deadline missing item chargeback”

    3,238 tokens in (3,121 from cache), 63 out

  3. Code

    Retrieved 1 section

    Searched this restaurant's policies only. Threshold 0.4; the model saw only the sections above it.

    • 0.41 House Policy §5, Online Orders
    • 0.38 Online Ordering and Loyalty Policy §3, Pickup Orders (below threshold)
    • 0.38 House Policy §3, Refunds (below threshold)
    • 0.35 House Policy §6, Catering (below threshold)
  4. Model

    Looked up the policy

    “marketplace terms disputes deadline days to file”

    3,412 tokens in (3,121 from cache), 60 out

  5. Code

    Nothing in the policies came close enough

    Searched this restaurant's policies only. Threshold 0.4; the model saw only the sections above it.

    • 0.35 House Policy §5, Online Orders (below threshold)
    • 0.32 Customer Messaging Policy §5, Offers in Messages (below threshold)
    • 0.31 House Policy §3, Refunds (below threshold)
    • 0.30 Online Ordering and Loyalty Policy §2, Delivery Marketplaces (below threshold)
  6. Model

    Answered

    The policies I can search don't give a deadline for disputing a missing-item chargeback on a DoorDash order. The only section that came back was . It covers Square Online pickup orders and doesn't apply to DoorDash. A second search for marketplace dispute terms returned nothing relevant.

    I won't guess at a number of days. Check DoorDash's merchant terms or your DoorDash merchant portal for the exact window. Because the deadline may be short, file as soon as you see the chargeback. If you give me the order, I can pull its details and propose a dispute for you to approve.

    3,497 tokens in (3,121 from cache), 199 out

  7. Code

    Citations checked

    Every section cited was one that was retrieved: House Policy §5, Online Orders.

answered

  • $0.0067to answer, $0.0235 without the prompt cache
  • 4.9 s4.2 s in the model
  • 6steps, 10,147 tokens in, 322 out