All 38 cases

Evaluation case sql-02, from the sql suite

How many new customers did Lunch 20 bring in?

Run 1 passRun 2 pass

Asked as Mekong Noodle Housenew-customerowner-definitionunidentified

What should happen

Answered, and the answer contains the figure from the query I wrote.

A promo order carries the Lunch 20 discount (Definitions §3), so first-timers whose first order was merely at lunch in August do not count: that reading gives 451, not 262. 331 of the 811 promo orders were unidentified and cannot be classified.

Each run

Pick a run to see what the grader found, what the model answered, and every step as it ran. Steps marked model were decided by Claude; steps marked code ran as fixed logic; a person appears only where someone approved or rejected something.

What the grader found in run 2

Passed.

The query I wrote

The question read the way the owner means it. Run on the frozen database for the figure the grader expects.

SELECT count(*)
FROM (
  SELECT DISTINCT ON (customer_id) discount_name
  FROM orders
  WHERE state = 'completed'
    AND customer_id IS NOT NULL
  ORDER BY customer_id, created_at
) first_order
WHERE discount_name = 'Lunch 20'

The query the model wrote

Its last query, laid out for reading. The trace has it as sent. The guard added the restaurant filter and the row limit before it ran.

WITH
  o AS (
    SELECT
      o.*,
      (
        created_at AT TIME ZONE(
          SELECT
            timezone
          FROM
            merchants
        )
      ) AS lt
    FROM
      orders o
    WHERE
      state = 'completed'
  ),
  f AS (
    SELECT DISTINCT
      ON (customer_id) *
    FROM
      o
    WHERE
      customer_id IS NOT NULL
    ORDER BY
      customer_id,
      created_at,
      id
  )
SELECT
  (
    SELECT
      count(*)
    FROM
      f
    WHERE
      discount_name ILIKE 'lunch 20%'
      AND channel IN ('in_store', 'square_online')
      AND lt >= '2026-08-03'
      AND lt < '2026-08-31'
      AND extract(
        HOUR
        FROM
          lt
      ) BETWEEN 11 AND 13
  ) AS new_customers,
  (
    SELECT
      count(*)
    FROM
      o
    WHERE
      customer_id IS NULL
      AND discount_name ILIKE 'lunch 20%'
      AND channel IN ('in_store', 'square_online')
      AND lt >= '2026-08-03'
      AND lt < '2026-08-31'
      AND extract(
        HOUR
        FROM
          lt
      ) BETWEEN 11 AND 13
  ) AS unidentified_promo_orders,
  (
    SELECT
      count(DISTINCT discount_name)
    FROM
      o
    WHERE
      discount_name ILIKE '%lunch%'
  ) AS names

What it answered

Lunch 20 brought in 262 new customers. That counts identified customers whose first completed order ever was a Lunch 20 promo order. A promo order is a completed order with the Lunch 20 discount, placed in store or on Square Online from 11:00 to 13:59 local time, between August 3 and August 30, 2026 .

There were also 331 promo orders with no customer attached. I can't tell whether those guests were new, so they aren't in the 262.

answered

  • $0.0120to answer, $0.0288 without the prompt cache
  • 7.6 s6.4 s in the model
  • 7steps, 10,844 tokens in, 715 out
  1. Human

    Asked

    How many new customers did Lunch 20 bring in?

  2. Model

    Looked up the policy

    “Lunch 20 promotion definition”

    3,228 tokens in (3,121 from cache), 77 out

  3. Code

    Retrieved 2 sections

    Searched this restaurant's policies only. Threshold 0.4; the model saw only the sections above it.

    • 0.76 Definitions §3, The Lunch 20 Promotion
    • 0.54 Definitions §2, Lunch
    • 0.38 Marketplace Terms §6, Promotions (below threshold)
    • 0.38 Customer Messaging Policy §5, Offers in Messages (below threshold)
  4. Model

    Proposed a query

    Count customers whose first completed order was a Lunch 20 promo order; count unidentified promo orders

    with o as (select o.*, (created_at AT TIME ZONE (SELECT timezone FROM merchants)) as lt from orders o where state='completed'),
f as (select distinct on (customer_id) * from o where customer_id is not null order by customer_id, created_at, id)
select (select count(*) from f where discount_name ilike 'lunch 20%' and channel in ('in_store','square_online') and lt >= '2026-08-03' and lt < '2026-08-31' and extract(hour from lt) between 11 and 13) as new_customers,
(select count(*) from o where customer_id is null and discount_name ilike 'lunch 20%' and channel in ('in_store','square_online') and lt >= '2026-08-03' and lt < '2026-08-31' and extract(hour from lt) between 11 and 13) as unidentified_promo_orders,
(select count(distinct discount_name) from o where discount_name ilike '%lunch%') as names

    3,547 tokens in (3,121 from cache), 457 out

  5. Code

    Guard passed

    • ✓ parses
    • ✓ single statement
    • ✓ select only
    • ✓ no writes
    • ✓ no unsafe functions
    • ✓ allowed tables, merchants, orders
    • ✓ tenant scope, merchant_id=1 on merchants.id, orders.merchant_id
    • ✓ row limit, injected 500
  6. Code

    Ran it: 1 row in 233 ms

    new_customersunidentified_promo_ordersnames
    2623311
  7. Model

    Answered

    Lunch 20 brought in 262 new customers. That counts identified customers whose first completed order ever was a Lunch 20 promo order. A promo order is a completed order with the Lunch 20 discount, placed in store or on Square Online from 11:00 to 13:59 local time, between August 3 and August 30, 2026 .

    There were also 331 promo orders with no customer attached. I can't tell whether those guests were new, so they aren't in the 262.

    4,069 tokens in (3,121 from cache), 181 out

  8. Code

    Citations checked

    Every section cited was one that was retrieved: Definitions §2, Lunch; Definitions §3, The Lunch 20 Promotion.

answered

  • $0.0120to answer, $0.0288 without the prompt cache
  • 7.6 s6.4 s in the model
  • 7steps, 10,844 tokens in, 715 out