All 38 cases

Evaluation case pol-06, from the policy suite

A first responder in uniform is also a student. What discount do they get?

Run 1 passRun 2 pass

Asked as Mekong Noodle Housediscountscombining

What should happen

Answered, citing House Policy §4, Discounts.

The wrong answer is 25%. The grader only checks that 15% appears, because a right answer may well say 'not 25%'.

Each run

Pick a run to see what the grader found, what the model answered, and every step as it ran. Steps marked model were decided by Claude; steps marked code ran as fixed logic; a person appears only where someone approved or rejected something.

What the grader found in run 2

Passed.

What it answered

They get 15%. A first responder in uniform gets 15%, and students get 10% with ID. Discounts aren't combined, and the largest applicable one applies, so the 10% student discount isn't added on top.

answered

  • $0.0038to answer, $0.0150 without the prompt cache
  • 2.5 s2.3 s in the model
  • 4steps, 6,786 tokens in, 147 out
  1. Human

    Asked

    A first responder in uniform is also a student. What discount do they get?

  2. Model

    Looked up the policy

    “first responder discount and student discount; can discounts be combined or stacked”

    3,231 tokens in (3,121 from cache), 66 out

  3. Code

    Retrieved 2 sections

    Searched this restaurant's policies only. Threshold 0.4; the model saw only the sections above it.

    • 0.60 House Policy §4, Discounts
    • 0.45 Customer Messaging Policy §5, Offers in Messages
    • 0.32 Marketplace Terms §6, Promotions (below threshold)
    • 0.32 Definitions §3, The Lunch 20 Promotion (below threshold)
  4. Model

    Answered

    They get 15%. A first responder in uniform gets 15%, and students get 10% with ID. Discounts aren't combined, and the largest applicable one applies, so the 10% student discount isn't added on top.

    3,555 tokens in (3,121 from cache), 81 out

  5. Code

    Citations checked

    Every section cited was one that was retrieved: House Policy §4, Discounts.

answered

  • $0.0038to answer, $0.0150 without the prompt cache
  • 2.5 s2.3 s in the model
  • 4steps, 6,786 tokens in, 147 out