All 38 cases

Evaluation case pol-13, from the policy suite

How many stamps earn a free drink, and can it be used on a Seasonal Special?

Run 1 passRun 2 pass

Asked as Ridgeline Coffeeloyaltymulti-part

What should happen

Answered, citing Online Ordering and Loyalty Policy §4, Loyalty Program.

Each run

Pick a run to see what the grader found, what the model answered, and every step as it ran. Steps marked model were decided by Claude; steps marked code ran as fixed logic; a person appears only where someone approved or rejected something.

What the grader found in run 2

Passed.

What it answered

Nine stamps earn a free drink. Customers earn one stamp per drink when they give their phone number at checkout.

The free drink can't be used on a Seasonal Special. It can be any size from Signature Drinks, House Favorites, or Matcha. Seasonal Specials, Flights, and Bubble Tea are excluded. Seasonal Specials are also never discounted.

Two time limits apply. Stamps expire 12 months after they are earned, and an issued reward expires 60 days after it is issued.

answered

  • $0.0064to answer, $0.0176 without the prompt cache
  • 3.4 s3.2 s in the model
  • 4steps, 7,176 tokens in, 326 out
  1. Human

    Asked

    How many stamps earn a free drink, and can it be used on a Seasonal Special?

  2. Model

    Looked up the policy

    “loyalty stamps free drink reward Seasonal Special”

    3,238 tokens in (3,121 from cache), 64 out

  3. Code

    Retrieved 4 sections

    Searched this restaurant's policies only. Threshold 0.4; the model saw only the sections above it.

    • 0.69 Online Ordering and Loyalty Policy §4, Loyalty Program
    • 0.61 House Policy §4, Discounts
    • 0.61 Definitions §5, Free Drinks
    • 0.48 Definitions §3, Seasonal Specials
  4. Model

    Answered

    Nine stamps earn a free drink. Customers earn one stamp per drink when they give their phone number at checkout.

    The free drink can't be used on a Seasonal Special. It can be any size from Signature Drinks, House Favorites, or Matcha. Seasonal Specials, Flights, and Bubble Tea are excluded. Seasonal Specials are also never discounted.

    Two time limits apply. Stamps expire 12 months after they are earned, and an issued reward expires 60 days after it is issued.

    3,938 tokens in (3,121 from cache), 262 out

  5. Code

    Citations checked

    Every section cited was one that was retrieved: House Policy §4, Discounts; Online Ordering and Loyalty Policy §4, Loyalty Program.

answered

  • $0.0064to answer, $0.0176 without the prompt cache
  • 3.4 s3.2 s in the model
  • 4steps, 7,176 tokens in, 326 out