Evaluation case pol-07, from the policy suite
A student is redeeming a free loyalty drink and wants their 10% off the rest. Can they have both?
Run 1 passRun 2 pass
Asked as Ridgeline Coffeediscountscombiningcross-tenant
What should happen
Answered, citing House Policy §4, Discounts.
The opposite rule to Mekong's. The retrieval must not leak Mekong's 'discounts are not combined'.
Each run
Pick a run to see what the grader found, what the model answered, and every step as it ran. Steps marked model were decided by Claude; steps marked code ran as fixed logic; a person appears only where someone approved or rejected something.
What the grader found in run 2
Passed.
What it answered
Yes, the student can have both. The policy says a free loyalty drink is not a discount, so it can be combined with a standard discount on the rest of the order. The student's 10% off with ID is a standard discount .
Two limits apply:
- Seasonal Specials: these are never discounted . They are also excluded from the free drink .
- Eligible drinks: the free drink has to come from Signature Drinks, House Favorites, or Matcha. Flights and Bubble Tea are excluded .
The free drink is rung up at its menu price as the discount amount . The 10% then applies to the rest of the order.
answered
- $0.0067to answer, $0.0179 without the prompt cache
- 4.0 s3.6 s in the model
- 4steps, 7,152 tokens in, 361 out
- Human
Asked
A student is redeeming a free loyalty drink and wants their 10% off the rest. Can they have both?
- Model
Looked up the policy
“stacking discounts: free loyalty reward drink combined with student 10% discount”
3,244 tokens in (3,121 from cache), 70 out
- Code
Retrieved 4 sections
Searched this restaurant's policies only. Threshold 0.4; the model saw only the sections above it.
- 0.66 House Policy §4, Discounts
- 0.58 Definitions §5, Free Drinks
- 0.58 Online Ordering and Loyalty Policy §4, Loyalty Program
- 0.41 Customer Messaging Policy §5, Offers in Messages
- Model
Answered
Yes, the student can have both. The policy says a free loyalty drink is not a discount, so it can be combined with a standard discount on the rest of the order. The student's 10% off with ID is a standard discount .
Two limits apply:
- Seasonal Specials: these are never discounted . They are also excluded from the free drink .
- Eligible drinks: the free drink has to come from Signature Drinks, House Favorites, or Matcha. Flights and Bubble Tea are excluded .
The free drink is rung up at its menu price as the discount amount . The 10% then applies to the rest of the order.
3,908 tokens in (3,121 from cache), 291 out
- Code
Citations checked
Every section cited was one that was retrieved: Definitions §5, Free Drinks; House Policy §4, Discounts; Online Ordering and Loyalty Policy §4, Loyalty Program.
answered
- $0.0067to answer, $0.0179 without the prompt cache
- 4.0 s3.6 s in the model
- 4steps, 7,152 tokens in, 361 out