All 38 cases

Evaluation case pol-11, from the policy suite

Can we text our list a 20% off deal?

Run 1 failRun 2 fail

Asked as Mekong Noodle Housemessagingoffers

What should happen

Answered, citing Customer Messaging Policy §5, Offers in Messages.

What the trace shows

The answer was right and the campaign was blocked, as it should be. The failure is in the citation: the model wrote the first label in full, then shortened the next ones to the section alone, and the citations step, which only accepts labels exactly as retrieved, failed them. A formatting slip caught by code, on both runs.

Each run

Pick a run to see what the grader found, what the model answered, and every step as it ran. Steps marked model were decided by Claude; steps marked code ran as fixed logic; a person appears only where someone approved or rejected something.

What the grader found in run 2

  • the citations step failed: [§6, Required Wording] is not the label of a section that was shown

What it answered

Not as written. I proposed the campaign, and the policy check blocked it. Nothing has been sent.

The blocking reason is the discount size. Texted offers are capped at 15% or $10, whichever is lower, unless the owner approves a larger one . A 20% offer is over that cap.

You also can't text the whole list. Only customers with verified SMS consent who haven't unsubscribed can get marketing texts. Implied consent, such as giving a phone number when ordering, isn't enough . That leaves 943 of your 48,179 customers.

The draft message is: "Mekong Noodle House: Enjoy 20% off your next order, one per customer. Offer expires Nov 5, 2026. Reply STOP to opt out". It names the restaurant, says "one per customer" and ends with the opt-out wording. The expiry is within 14 days [Customer Messaging Policy §5, Offers in Messages; §6, Required Wording]. It would also need to go out between 9am and 8pm local time .

You have two options. You can lower the offer to 15% or less, which should pass the check. Or you can approve the 20% yourself as owner, and I can resubmit it on that basis.

answered

  • $0.0251to answer, $0.0532 without the prompt cache
  • 11.9 s11.7 s in the model
  • 12steps, 19,957 tokens in, 1,324 out
  1. Human

    Asked

    Can we text our list a 20% off deal?

  2. Model

    Looked up the policy

    “text message campaign consent, quiet hours, discount offers limits”

    3,225 tokens in (3,121 from cache), 65 out

  3. Code

    Retrieved 4 sections

    Searched this restaurant's policies only. Threshold 0.4; the model saw only the sections above it.

    • 0.66 Customer Messaging Policy §5, Offers in Messages
    • 0.66 Customer Messaging Policy §3, Quiet Hours
    • 0.58 Customer Messaging Policy §2, Who May Receive Marketing Texts
    • 0.56 Customer Messaging Policy §6, Required Wording
  4. Model

    Proposed a query

    Count eligible audience

    select count(*) filter (where sms_opt_in and unsubscribed_at is null) eligible, count(*) total from customers

    3,804 tokens in (3,121 from cache), 115 out

  5. Code

    Guard passed

    • ✓ parses
    • ✓ single statement
    • ✓ select only
    • ✓ no writes
    • ✓ no unsafe functions
    • ✓ allowed tables, customers
    • ✓ tenant scope, merchant_id=1 on customers.merchant_id
    • ✓ row limit, injected 500
  6. Code

    Ran it: 1 row in 34 ms

    eligibletotal
    94348179
  7. Model

    Proposed a query

    Restaurant name

    select name from merchants

    3,969 tokens in (3,121 from cache), 80 out

  8. Code

    Guard passed

    • ✓ parses
    • ✓ single statement
    • ✓ select only
    • ✓ no writes
    • ✓ no unsafe functions
    • ✓ allowed tables, merchants
    • ✓ tenant scope, merchant_id=1 on merchants.id
    • ✓ row limit, injected 500
  9. Code

    Ran it: 1 row in 5 ms

    name
    Mekong Noodle House
  10. Model

    Proposed an action

    Text 20% off to the 943 customers with verified SMS consent and no unsubscribe; 20% is above the 15% cap, so it needs explicit owner approval.

    verified opt-in only, no unsubscribed. discounts over 15% or $10 need owner approval; expiry within 14 days and "one per customer". send only 9am-8pm local. name restaurant and end with opt-out.

    Mekong Noodle House: Enjoy 20% off your next order, one per customer. Offer expires Nov 5, 2026. Reply STOP to opt out

    4,099 tokens in (3,121 from cache), 608 out

  11. Code

    Blocked the proposal

    20% is over the 15% cap.

    • ✕ discount, 20% is over the 15% cap
  12. Model

    Answered

    Not as written. I proposed the campaign, and the policy check blocked it. Nothing has been sent.

    The blocking reason is the discount size. Texted offers are capped at 15% or $10, whichever is lower, unless the owner approves a larger one . A 20% offer is over that cap.

    You also can't text the whole list. Only customers with verified SMS consent who haven't unsubscribed can get marketing texts. Implied consent, such as giving a phone number when ordering, isn't enough . That leaves 943 of your 48,179 customers.

    The draft message is: "Mekong Noodle House: Enjoy 20% off your next order, one per customer. Offer expires Nov 5, 2026. Reply STOP to opt out". It names the restaurant, says "one per customer" and ends with the opt-out wording. The expiry is within 14 days [Customer Messaging Policy §5, Offers in Messages; §6, Required Wording]. It would also need to go out between 9am and 8pm local time .

    You have two options. You can lower the offer to 15% or less, which should pass the check. Or you can approve the 20% yourself as owner, and I can resubmit it on that basis.

    4,860 tokens in (3,121 from cache), 456 out

  13. Code

    Citations failed

    cited a section that was not retrieved.

    Not retrieved: §6, Required Wording

answered

  • $0.0251to answer, $0.0532 without the prompt cache
  • 11.9 s11.7 s in the model
  • 12steps, 19,957 tokens in, 1,324 out