All 38 cases

Evaluation case pol-12, from the policy suite

Can a lead give a regular 25% off a catering order?

Run 1 failRun 2 pass

Asked as Ridgeline Coffeediscountscatering

What should happen

Answered, citing House Policy §4, Discounts.

Each run

Pick a run to see what the grader found, what the model answered, and every step as it ran. Steps marked model were decided by Claude; steps marked code ran as fixed logic; a person appears only where someone approved or rejected something.

What the grader found in run 2

Passed.

What it answered

No. A lead can't approve 25% off. Under , a discount above 10% that isn't one of the standard ones needs a lead, and a discount above 20% needs the owner. So 25% needs the owner's approval.

Two catering rules also apply under . A catering order over $300 needs the owner's confirmation. Catering orders also need 72 hours' notice and a 50% deposit. Seasonal Specials can never be discounted, so they would have to be left out of the 25%.

The policies don't say that being a regular changes any of this. I didn't find a rule giving regulars a different discount limit.

answered

  • $0.0058to answer, $0.0170 without the prompt cache
  • 4.0 s3.7 s in the model
  • 4steps, 7,151 tokens in, 272 out
  1. Human

    Asked

    Can a lead give a regular 25% off a catering order?

  2. Model

    Looked up the policy

    “lead approval discount limit catering order”

    3,229 tokens in (3,121 from cache), 56 out

  3. Code

    Retrieved 4 sections

    Searched this restaurant's policies only. Threshold 0.4; the model saw only the sections above it.

    • 0.50 House Policy §6, Catering
    • 0.43 House Policy §4, Discounts
    • 0.43 Customer Messaging Policy §5, Offers in Messages
    • 0.41 Online Ordering and Loyalty Policy §3, Pickup Orders
  4. Model

    Answered

    No. A lead can't approve 25% off. Under , a discount above 10% that isn't one of the standard ones needs a lead, and a discount above 20% needs the owner. So 25% needs the owner's approval.

    Two catering rules also apply under . A catering order over $300 needs the owner's confirmation. Catering orders also need 72 hours' notice and a 50% deposit. Seasonal Specials can never be discounted, so they would have to be left out of the 25%.

    The policies don't say that being a regular changes any of this. I didn't find a rule giving regulars a different discount limit.

    3,922 tokens in (3,121 from cache), 216 out

  5. Code

    Citations checked

    Every section cited was one that was retrieved: House Policy §4, Discounts; House Policy §6, Catering.

answered

  • $0.0058to answer, $0.0170 without the prompt cache
  • 4.0 s3.7 s in the model
  • 4steps, 7,151 tokens in, 272 out