All 38 cases

Evaluation case sql-10, from the sql suite

How many regulars did we lose over the summer?

Run 1 passRun 2 pass

Asked as Ridgeline Coffeeregularsowner-definitionchurn

What should happen

Answered, and the answer contains the figure from the query I wrote.

The same question as sql-03 under the other owner's definition: four or more orders in May, Definitions §2. 18 of 59. Mekong's three-in-three-months rule gives a different count here.

Each run

Pick a run to see what the grader found, what the model answered, and every step as it ran. Steps marked model were decided by Claude; steps marked code ran as fixed logic; a person appears only where someone approved or rejected something.

What the grader found in run 2

Passed.

The query I wrote

The question read the way the owner means it. Run on the frozen database for the figure the grader expects.

WITH visits AS (
  SELECT customer_id, (created_at AT TIME ZONE 'America/Denver')::date AS local_day
  FROM orders
  WHERE state = 'completed'
    AND customer_id IS NOT NULL
)
SELECT count(*)
FROM (
  SELECT customer_id
  FROM visits
  GROUP BY customer_id
  HAVING count(*) FILTER (WHERE local_day BETWEEN '2026-05-01' AND '2026-05-31') >= 4
     AND count(*) FILTER (WHERE local_day > '2026-08-31') = 0
) lost_regulars

The query the model wrote

Its last query, laid out for reading. The trace has it as sent. The guard added the restaurant filter and the row limit before it ran.

WITH
  o AS (
    SELECT
      customer_id,
      (
        created_at AT TIME ZONE(
          SELECT
            timezone
          FROM
            merchants
        )
      ) AS ts
    FROM
      orders
    WHERE
      state = 'completed'
      AND customer_id IS NOT NULL
  )
SELECT
  count(*)
FROM
  (
    SELECT
      customer_id
    FROM
      o
    GROUP BY
      customer_id
    HAVING
      count(*) FILTER (
        WHERE
          ts >= '2026-05-01'
          AND ts < '2026-06-01'
      ) >= 4
      AND count(*) FILTER (
        WHERE
          ts >= '2026-09-01'
      ) = 0
  ) x

What it answered

We lost 18 regulars over the summer of 2026. I used your definition: a regular is a customer with four or more completed orders in May 2026 who has had no completed order since the end of August, meaning none from September 1 through today (October 24). Only identified customers are counted, because guest orders with no customer record can't be tied to a person.

answered

  • $0.0098to answer, $0.0266 without the prompt cache
  • 6.5 s5.8 s in the model
  • 7steps, 10,687 tokens in, 524 out
  1. Human

    Asked

    How many regulars did we lose over the summer?

  2. Model

    Looked up the policy

    “definition of regular customer and summer”

    3,225 tokens in (3,121 from cache), 55 out

  3. Code

    Retrieved 2 sections

    Searched this restaurant's policies only. Threshold 0.4; the model saw only the sections above it.

    • 0.70 Definitions §2, Regulars
    • 0.44 Definitions §3, Seasonal Specials
    • 0.34 Customer Messaging Policy §4, Frequency (below threshold)
    • 0.34 Customer Messaging Policy §5, Offers in Messages (below threshold)
  4. Model

    Proposed a query

    Count customers with 4+ completed orders in May 2026 and none since Sep 1 2026

    with o as (select customer_id, (created_at AT TIME ZONE (SELECT timezone FROM merchants)) as ts from orders where state='completed' and customer_id is not null)
select count(*) from (select customer_id from o group by customer_id
having count(*) filter (where ts>='2026-05-01' and ts<'2026-06-01')>=4
and count(*) filter (where ts>='2026-09-01')=0) x

    3,535 tokens in (3,121 from cache), 351 out

  5. Code

    Guard passed

    • ✓ parses
    • ✓ single statement
    • ✓ select only
    • ✓ no writes
    • ✓ no unsafe functions
    • ✓ allowed tables, merchants, orders
    • ✓ tenant scope, merchant_id=2 on merchants.id, orders.merchant_id
    • ✓ row limit, injected 500
  6. Code

    Ran it: 1 row in 61 ms

    count
    18
  7. Model

    Answered

    We lost 18 regulars over the summer of 2026. I used your definition: a regular is a customer with four or more completed orders in May 2026 who has had no completed order since the end of August, meaning none from September 1 through today (October 24). Only identified customers are counted, because guest orders with no customer record can't be tied to a person.

    3,927 tokens in (3,121 from cache), 118 out

  8. Code

    Citations checked

    Every section cited was one that was retrieved: Definitions §2, Regulars.

answered

  • $0.0098to answer, $0.0266 without the prompt cache
  • 6.5 s5.8 s in the model
  • 7steps, 10,687 tokens in, 524 out