All 38 cases

Evaluation case pol-03, from the policy suite

Can we send a promotional text at 10pm?

Run 1 passRun 2 fail

Asked as Mekong Noodle Housemessaging

What should happen

Answered, citing Customer Messaging Policy §3, Quiet Hours.

What the trace shows

The same slip as pol-11, on a different question. The model cited Quiet Hours with its full label, then shortened the next two to [§4, Frequency] and [§5, Offers in Messages]. The citations step accepts labels exactly as retrieved and failed them. The answer itself is right: no texts between 8pm and 9am.

Each run

Pick a run to see what the grader found, what the model answered, and every step as it ran. Steps marked model were decided by Claude; steps marked code ran as fixed logic; a person appears only where someone approved or rejected something.

What the grader found in run 2

  • the citations step failed: [§4, Frequency], [§5, Offers in Messages] are not labels of sections that were shown

What it answered

No. Under , no text of any kind can go out between 8pm and 9am in the customer's local time. That covers promotional texts, and 10pm falls inside the window. If a customer's local time isn't known, the policy says to use Central time.

If you want to send a promotion, schedule it for between 9am and 8pm in the customer's local time. The other rules still apply. Recipients need verified SMS consent . Each customer can get at most four marketing texts a month [§4, Frequency]. Any offer is capped at 15% or $10, whichever is lower. It must also state an expiry date within 14 days and say "one per customer" [§5, Offers in Messages].

answered

  • $0.0061to answer, $0.0173 without the prompt cache
  • 3.7 s3.6 s in the model
  • 4steps, 7,038 tokens in, 324 out
  1. Human

    Asked

    Can we send a promotional text at 10pm?

  2. Model

    Looked up the policy

    “quiet hours for promotional text messages, time of day sending”

    3,226 tokens in (3,121 from cache), 63 out

  3. Code

    Retrieved 4 sections

    Searched this restaurant's policies only. Threshold 0.4; the model saw only the sections above it.

    • 0.68 Customer Messaging Policy §3, Quiet Hours
    • 0.54 Customer Messaging Policy §5, Offers in Messages
    • 0.53 Customer Messaging Policy §4, Frequency
    • 0.52 Customer Messaging Policy §2, Who May Receive Marketing Texts
  4. Model

    Answered

    No. Under , no text of any kind can go out between 8pm and 9am in the customer's local time. That covers promotional texts, and 10pm falls inside the window. If a customer's local time isn't known, the policy says to use Central time.

    If you want to send a promotion, schedule it for between 9am and 8pm in the customer's local time. The other rules still apply. Recipients need verified SMS consent . Each customer can get at most four marketing texts a month [§4, Frequency]. Any offer is capped at 15% or $10, whichever is lower. It must also state an expiry date within 14 days and say "one per customer" [§5, Offers in Messages].

    3,812 tokens in (3,121 from cache), 261 out

  5. Code

    Citations failed

    cited a section that was not retrieved.

    Not retrieved: §4, Frequency; §5, Offers in Messages

answered

  • $0.0061to answer, $0.0173 without the prompt cache
  • 3.7 s3.6 s in the model
  • 4steps, 7,038 tokens in, 324 out