All 38 cases

Evaluation case pol-15, from the policy suite

What discount do hospital staff get?

Run 1 passRun 2 pass

Asked as Mekong Noodle Housediscountscross-tenantnear-miss

What should happen

Answered, citing House Policy §4, Discounts.

Only Ridgeline has a hospital-staff discount. Mekong's Discounts section is the nearest match and names students, seniors and first responders. The right answer is that there is none here, without borrowing the other restaurant's rule.

Each run

Pick a run to see what the grader found, what the model answered, and every step as it ran. Steps marked model were decided by Claude; steps marked code ran as fixed logic; a person appears only where someone approved or rejected something.

What the grader found in run 2

Passed.

What it answered

Your policies don't list a discount for hospital staff. The standard register discounts in are:

  • 10% for students and seniors with ID.
  • 15% for first responders in uniform.

Hospital staff aren't named in either group. Any discount above 15% on a single order needs a manager. Discounts aren't combined, so the largest applicable one applies.

covers free meals for your own staff. It doesn't apply to hospital staff.

If you want to offer hospital staff a discount, you'd need to set one as a new rule.

answered

  • $0.0048to answer, $0.0160 without the prompt cache
  • 3.6 s3.4 s in the model
  • 4steps, 6,767 tokens in, 247 out
  1. Human

    Asked

    What discount do hospital staff get?

  2. Model

    Looked up the policy

    “hospital staff discount”

    3,222 tokens in (3,121 from cache), 52 out

  3. Code

    Retrieved 2 sections

    Searched this restaurant's policies only. Threshold 0.4; the model saw only the sections above it.

    • 0.61 House Policy §4, Discounts
    • 0.48 House Policy §5, Staff Meals
    • 0.37 Customer Messaging Policy §5, Offers in Messages (below threshold)
    • 0.34 House Policy §2, Comps (below threshold)
  4. Model

    Answered

    Your policies don't list a discount for hospital staff. The standard register discounts in are:

    • 10% for students and seniors with ID.
    • 15% for first responders in uniform.

    Hospital staff aren't named in either group. Any discount above 15% on a single order needs a manager. Discounts aren't combined, so the largest applicable one applies.

    covers free meals for your own staff. It doesn't apply to hospital staff.

    If you want to offer hospital staff a discount, you'd need to set one as a new rule.

    3,545 tokens in (3,121 from cache), 195 out

  5. Code

    Citations checked

    Every section cited was one that was retrieved: House Policy §4, Discounts; House Policy §5, Staff Meals.

answered

  • $0.0048to answer, $0.0160 without the prompt cache
  • 3.6 s3.4 s in the model
  • 4steps, 6,767 tokens in, 247 out