All 38 cases

Evaluation case sql-03, from the sql suite

How many regulars did we lose over the summer?

Run 1 passRun 2 pass

Asked as Mekong Noodle Houseregularsowner-definitionchurn

What should happen

Answered, and the answer contains the figure from the query I wrote.

Regular, summer and lost are all the owner's words, Definitions §4: three or more orders in March to May, nothing since the end of August. 177 of 599.

Each run

Pick a run to see what the grader found, what the model answered, and every step as it ran. Steps marked model were decided by Claude; steps marked code ran as fixed logic; a person appears only where someone approved or rejected something.

What the grader found in run 2

Passed.

The query I wrote

The question read the way the owner means it. Run on the frozen database for the figure the grader expects.

WITH visits AS (
  SELECT customer_id, (created_at AT TIME ZONE 'America/Chicago')::date AS local_day
  FROM orders
  WHERE state = 'completed'
    AND customer_id IS NOT NULL
)
SELECT count(*)
FROM (
  SELECT customer_id
  FROM visits
  GROUP BY customer_id
  HAVING count(*) FILTER (WHERE local_day BETWEEN '2026-03-01' AND '2026-05-31') >= 3
     AND count(*) FILTER (WHERE local_day > '2026-08-31') = 0
) lost_regulars

The query the model wrote

Its last query, laid out for reading. The trace has it as sent. The guard added the restaurant filter and the row limit before it ran.

WITH
  o AS (
    SELECT
      customer_id,
      (
        created_at AT TIME ZONE(
          SELECT
            timezone
          FROM
            merchants
        )
      ) AS lt
    FROM
      orders
    WHERE
      state = 'completed'
      AND customer_id IS NOT NULL
  )
SELECT
  count(*) AS lost_regulars
FROM
  (
    SELECT
      customer_id
    FROM
      o
    GROUP BY
      customer_id
    HAVING
      count(*) FILTER (
        WHERE
          lt >= '2026-03-01'
          AND lt < '2026-06-01'
      ) >= 3
      AND count(*) FILTER (
        WHERE
          lt >= '2026-09-01'
      ) = 0
  ) x

What it answered

We lost 177 regulars over the summer of 2026. The owner's definition is a customer with three or more completed orders in March, April and May 2026 and no completed order since the end of August . I counted none from September 1 through today, October 24, using local time. The count only covers identified customers, because guest orders with no customer ID can't be tied to a person.

answered

  • $0.0099to answer, $0.0267 without the prompt cache
  • 7.0 s6.2 s in the model
  • 7steps, 10,485 tokens in, 576 out
  1. Human

    Asked

    How many regulars did we lose over the summer?

  2. Model

    Looked up the policy

    “definition of regular customer”

    3,225 tokens in (3,121 from cache), 90 out

  3. Code

    Retrieved 1 section

    Searched this restaurant's policies only. Threshold 0.4; the model saw only the sections above it.

    • 0.56 Definitions §4, Regulars
    • 0.31 Marketplace Terms §7, Repeat Disputes (below threshold)
    • 0.30 Customer Messaging Policy §7, Audience Size (below threshold)
    • 0.29 House Policy §4, Discounts (below threshold)
  4. Model

    Proposed a query

    Count customers with 3+ completed orders Mar-May 2026 and none since Aug 31 2026

    with o as (select customer_id, (created_at AT TIME ZONE (SELECT timezone FROM merchants)) as lt from orders where state='completed' and customer_id is not null)
select count(*) as lost_regulars from (select customer_id from o group by customer_id having count(*) filter (where lt >= '2026-03-01' and lt < '2026-06-01') >= 3 and count(*) filter (where lt >= '2026-09-01') = 0) x

    3,425 tokens in (3,121 from cache), 364 out

  5. Code

    Guard passed

    • ✓ parses
    • ✓ single statement
    • ✓ select only
    • ✓ no writes
    • ✓ no unsafe functions
    • ✓ allowed tables, merchants, orders
    • ✓ tenant scope, merchant_id=1 on merchants.id, orders.merchant_id
    • ✓ row limit, injected 500
  6. Code

    Ran it: 1 row in 81 ms

    lost_regulars
    177
  7. Model

    Answered

    We lost 177 regulars over the summer of 2026. The owner's definition is a customer with three or more completed orders in March, April and May 2026 and no completed order since the end of August . I counted none from September 1 through today, October 24, using local time. The count only covers identified customers, because guest orders with no customer ID can't be tied to a person.

    3,835 tokens in (3,121 from cache), 122 out

  8. Code

    Citations checked

    Every section cited was one that was retrieved: Definitions §4, Regulars.

answered

  • $0.0099to answer, $0.0267 without the prompt cache
  • 7.0 s6.2 s in the model
  • 7steps, 10,485 tokens in, 576 out