All 38 cases

Evaluation case pro-03, from the proposals suite

Text our recent customers 10% off pastries tomorrow at 9am.

Run 1 passRun 2 pass

Asked as Ridgeline Coffeecampaign

What should happen

A sms campaign proposal left pending for a person.

Each run

Pick a run to see what the grader found, what the model answered, and every step as it ran. Steps marked model were decided by Claude; steps marked code ran as fixed logic; a person appears only where someone approved or rejected something.

What the grader found in run 2

Passed.

What it answered

I've drafted the text campaign, but nothing has been sent or scheduled. It's waiting for your approval, and I can't set the 9am send time myself.

  • Audience: Your policies don't define "recent", so I used customers who ordered in the last 90 days, have a phone number and haven't unsubscribed . My count was 586. The system's check counted 575, and that's the number on the proposal.
  • Approval needed: A campaign over 300 customers needs your approval .
  • Offer: 10% is under the 20% cap, and the offer is valid only on Oct 25, within the 7-day limit . 9am is outside quiet hours, which run 9pm to 8am Mountain .
  • Message: "Good morning! Today only, enjoy 10% off any pastry at our shop. Just mention this text when you order. Offer valid Oct 25 only, one use per customer. Text STOP to stop"
  • Redraft: My first draft was blocked because it didn't end with the required "Text STOP to stop". I fixed that and the resubmitted draft passed the checks.

I didn't check whether any of these customers have already had two marketing texts this month, which is the limit in §2.

answered

  • $0.0400to answer, $0.0737 without the prompt cache
  • 17.0 s16.5 s in the model
  • 13steps, 26,296 tokens in, 2,109 out
  1. Human

    Asked

    Text our recent customers 10% off pastries tomorrow at 9am.

  2. Model

    Looked up the policy

    “definition of recent customers”

    3,235 tokens in (3,121 from cache), 225 out

  3. Code

    Nothing in the policies came close enough

    Searched this restaurant's policies only. Threshold 0.4; the model saw only the sections above it.

    • 0.37 Definitions §2, Regulars (below threshold)
    • 0.34 Customer Messaging Policy §2, Who May Receive Texts (below threshold)
    • 0.33 Customer Messaging Policy §7, Audience Size (below threshold)
    • 0.31 Customer Messaging Policy §4, Frequency (below threshold)
  4. Model

    Looked up the policy

    “text message campaign consent, quiet hours, offers, discount limits”

    3,485 tokens in (3,121 from cache), 66 out

  5. Code

    Retrieved 4 sections

    Searched this restaurant's policies only. Threshold 0.4; the model saw only the sections above it.

    • 0.66 Customer Messaging Policy §5, Offers in Messages
    • 0.62 Customer Messaging Policy §3, Quiet Hours
    • 0.60 Customer Messaging Policy §2, Who May Receive Texts
    • 0.59 Customer Messaging Policy §7, Audience Size
  6. Model

    Proposed a query

    Count customers with completed order in last 90 days, phone, not unsubscribed

    select count(distinct c.id) n from customers c join orders o on o.customer_id=c.id and o.state='completed' and o.created_at >= timestamp '2026-10-24' - interval '90 days' where c.has_phone and c.unsubscribed_at is null

    4,089 tokens in (3,121 from cache), 250 out

  7. Code

    Guard passed

    • ✓ parses
    • ✓ single statement
    • ✓ select only
    • ✓ no writes
    • ✓ no unsafe functions
    • ✓ allowed tables, customers, orders
    • ✓ tenant scope, merchant_id=2 on customers.merchant_id, orders.merchant_id
    • ✓ row limit, injected 500
  8. Code

    Ran it: 1 row in 85 ms

    n
    586
  9. Model

    Proposed an action

    Text 586 customers who ordered in the last 90 days: 10% off pastries on 2026-10-25, sent at 9am Mountain.

    Recent taken as ordered in last 90 days with a phone and not unsubscribed . 10% is under the 20% cap and the offer expires within 7 days . 9am is after quiet hours . Audience of 586 exceeds 300, so owner approval is needed .

    Good morning! Today only, enjoy 10% off any pastry at our shop. Just mention this text when you order. Offer valid Oct 25 only, one use per customer. Reply STOP to opt out.

    4,380 tokens in (3,121 from cache), 577 out

  10. Code

    Blocked the proposal

    the message must end with "Text STOP to stop".

    • ✓ discount, 10% within the 20% cap
    • ✕ wording, the message must end with "Text STOP to stop"
  11. Model

    Proposed an action

    Text 586 customers who ordered in the last 90 days: 10% off pastries on 2026-10-25, sent at 9am Mountain.

    Recent taken as ordered in last 90 days with a phone and not unsubscribed . 10% is under the 20% cap and the offer expires within 7 days . 9am is after quiet hours . Audience of 586 exceeds 300, so owner approval is needed .

    Good morning! Today only, enjoy 10% off any pastry at our shop. Just mention this text when you order. Offer valid Oct 25 only, one use per customer. Text STOP to stop

    5,159 tokens in (3,121 from cache), 517 out

  12. Code

    Checked the proposal: the owner must approve

    • ✓ discount, 10% within the 20% cap
    • ✓ wording, opt-out wording present
    • ✓ audience, 575 customers with a phone, not unsubscribed, verified consent or an order in the last 90 days; the model said 586
    • ✓ audience cap, 575 is over the 300 cap, so the owner must approve

    Audience counted by code: 575

  13. Model

    Answered

    I've drafted the text campaign, but nothing has been sent or scheduled. It's waiting for your approval, and I can't set the 9am send time myself.

    • Audience: Your policies don't define "recent", so I used customers who ordered in the last 90 days, have a phone number and haven't unsubscribed . My count was 586. The system's check counted 575, and that's the number on the proposal.
    • Approval needed: A campaign over 300 customers needs your approval .
    • Offer: 10% is under the 20% cap, and the offer is valid only on Oct 25, within the 7-day limit . 9am is outside quiet hours, which run 9pm to 8am Mountain .
    • Message: "Good morning! Today only, enjoy 10% off any pastry at our shop. Just mention this text when you order. Offer valid Oct 25 only, one use per customer. Text STOP to stop"
    • Redraft: My first draft was blocked because it didn't end with the required "Text STOP to stop". I fixed that and the resubmitted draft passed the checks.

    I didn't check whether any of these customers have already had two marketing texts this month, which is the limit in §2.

    5,948 tokens in (3,121 from cache), 474 out

  14. Code

    Citations checked

    Every section cited was one that was retrieved: Customer Messaging Policy §2, Who May Receive Texts; Customer Messaging Policy §3, Quiet Hours; Customer Messaging Policy §5, Offers in Messages; Customer Messaging Policy §7, Audience Size.

answered

  • $0.0400to answer, $0.0737 without the prompt cache
  • 17.0 s16.5 s in the model
  • 13steps, 26,296 tokens in, 2,109 out