Evaluation case sql-01, from the sql suite

Compared with the same four weeks last year, how much more did lunch bring in during the Lunch 20 promo?

Run 1 passRun 2 pass

Asked as Mekong Noodle Houseliftowner-definition

What should happen

Answered, and the answer contains the figure from the query I wrote.

The window, the hours and the two channels are in the owner's Definitions §3; lift against the same window a year earlier is the vendor's definition in the prompt. The month before gives $4,442 instead of $5,272.

Each run

Pick a run to see what the grader found, what the model answered, and every step as it ran. Steps marked model were decided by Claude; steps marked code ran as fixed logic; a person appears only where someone approved or rejected something.

What the grader found in run 2

Passed.

The query I wrote

The question read the way the owner means it. Run on the frozen database for the figure the grader expects.

WITH lunch AS (
  SELECT gross_sales_cents, created_at AT TIME ZONE 'America/Chicago' AS local_time
  FROM orders
  WHERE state = 'completed'
    AND channel IN ('in_store', 'square_online')
)
SELECT sum(gross_sales_cents) FILTER (WHERE local_time::date BETWEEN '2026-08-03' AND '2026-08-30')
     - sum(gross_sales_cents) FILTER (WHERE local_time::date BETWEEN '2025-08-03' AND '2025-08-30')
FROM lunch
WHERE extract(hour FROM local_time) BETWEEN 11 AND 13

The query the model wrote

Its last query, laid out for reading. The trace has it as sent. The guard added the restaurant filter and the row limit before it ran.

SELECT
  extract(
    YEAR
    FROM
      lt
  ) yr,
  count(*) orders,
  sum(gross_sales_cents) gross,
  sum(discount_cents) disc,
  sum(coalesce(r.ref, 0)) refunds,
  sum(gross_sales_cents - discount_cents - coalesce(r.ref, 0)) net
FROM
  (
    SELECT
      id,
      gross_sales_cents,
      discount_cents,
      created_at at time zone(
        SELECT
          timezone
        FROM
          merchants
      ) lt
    FROM
      orders
    WHERE
      state = 'completed'
      AND channel IN ('in_store', 'square_online')
      AND created_at >= '2025-08-02'
      AND created_at < '2026-09-01'
  ) x
  LEFT JOIN (
    SELECT
      order_id,
      sum(refunded_amount_cents) ref
    FROM
      payments
    GROUP BY
      order_id
  ) r ON r.order_id = x.id
WHERE
  (
    (lt::date BETWEEN '2026-08-03' AND '2026-08-30')
    OR (lt::date BETWEEN '2025-08-03' AND '2025-08-30')
  )
  AND extract(
    HOUR
    FROM
      lt
  ) BETWEEN 11 AND 13
GROUP BY
  1
ORDER BY
  1

What it answered

Lunch net sales rose by $1,751.36 during Lunch 20 compared with the same dates last year. This covers in-store and Square Online orders only, from 11:00 to 13:59 local time. Net sales means gross sales minus discounts minus refunds.

Lunch 20 window (Aug 3–30, 2026): net sales were $34,610.93 from 1,414 completed orders. Gross sales were $40,261.46, less $5,373.26 in discounts and $277.27 in refunds.
Same dates in 2025 (Aug 3–30): net sales were $32,859.57 from 1,194 completed orders. Gross sales were $34,989.77, less $1,497.21 in discounts and $632.99 in refunds.

That is about 5.3% more net, or 18% more orders. Gross sales rose by $5,271.69, but most of that went back out as promo discounts.

I used the same calendar dates for last year. Aug 3, 2025 fell on a Sunday and Aug 3, 2026 on a Monday, so the weekday mix differs slightly.

Policy basis: and .

answered

$0.0301to answer, $0.0454 without the prompt cache
17.3 s11.0 s in the model
10steps, 15,860 tokens in, 1,364 out
1retry

Human
Asked
Compared with the same four weeks last year, how much more did lunch bring in during the Lunch 20 promo?
Model
Looked up the policy
“Lunch 20 promotion definition dates and lunch hours”
3,246 tokens in, 65 out
Code
Retrieved 3 sections
Searched this restaurant's policies only. Threshold 0.4; the model saw only the sections above it.
- 0.78 Definitions §3, The Lunch 20 Promotion
- 0.62 Definitions §2, Lunch
- 0.44 House Policy §5, Staff Meals
- 0.40 House Policy §4, Discounts (below threshold)

Model

Proposed a query

Lunch net sales on in_store and square_online, promo window vs same dates 2025

select extract(year from lt) yr, count(*) orders, sum(gross_sales_cents) gross, sum(discount_cents) disc, sum(coalesce(ref,0)) refunds, sum(gross_sales_cents-discount_cents-coalesce(ref,0)) net from (select o.*, o.created_at at time zone (select timezone from merchants) lt, (select sum(refunded_amount_cents) from payments p where p.order_id=o.id) ref from orders o where state='completed' and channel in ('in_store','square_online')) x where ((lt::date between '2026-08-03' and '2026-08-30') or (lt::date between '2025-08-03' and '2025-08-30')) and extract(hour from lt) between 11 and 13 group by 1 order by 1

3,672 tokens in (3,121 from cache), 485 out

Code
Guard passed
- ✓ parses
- ✓ single statement
- ✓ select only
- ✓ no writes
- ✓ no unsafe functions
- ✓ allowed tables, merchants, orders, payments
- ✓ tenant scope, merchant_id=1 on merchants.id, orders.merchant_id, payments.merchant_id
- ✓ row limit, injected 500
Code
The database rejected it
canceling statement due to statement timeout

Model

Proposed a query

Lunch net sales promo window vs 2025

select extract(year from lt) yr, count(*) orders, sum(gross_sales_cents) gross, sum(discount_cents) disc, sum(coalesce(r.ref,0)) refunds, sum(gross_sales_cents-discount_cents-coalesce(r.ref,0)) net from (select id, gross_sales_cents, discount_cents, created_at at time zone (select timezone from merchants) lt from orders where state='completed' and channel in ('in_store','square_online') and created_at >= '2025-08-02' and created_at < '2026-09-01') x left join (select order_id, sum(refunded_amount_cents) ref from payments group by order_id) r on r.order_id=x.id where ((lt::date between '2026-08-03' and '2026-08-30') or (lt::date between '2025-08-03' and '2025-08-30')) and extract(hour from lt) between 11 and 13 group by 1 order by 1

4,204 tokens in (3,121 from cache), 411 out

Code
Guard passed
- ✓ parses
- ✓ single statement
- ✓ select only
- ✓ no writes
- ✓ no unsafe functions
- ✓ allowed tables, merchants, orders, payments
- ✓ tenant scope, merchant_id=1 on merchants.id, orders.merchant_id, payments.merchant_id
- ✓ row limit, injected 500
Code
Ran it: 2 rows in 131 ms
yr orders gross disc refunds net
2025 1194 3498977 149721 63299 3285957
2026 1414 4026146 537326 27727 3461093
Model
Answered
Lunch net sales rose by $1,751.36 during Lunch 20 compared with the same dates last year. This covers in-store and Square Online orders only, from 11:00 to 13:59 local time. Net sales means gross sales minus discounts minus refunds.
- Lunch 20 window (Aug 3–30, 2026): net sales were $34,610.93 from 1,414 completed orders. Gross sales were $40,261.46, less $5,373.26 in discounts and $277.27 in refunds.
- Same dates in 2025 (Aug 3–30): net sales were $32,859.57 from 1,194 completed orders. Gross sales were $34,989.77, less $1,497.21 in discounts and $632.99 in refunds.
That is about 5.3% more net, or 18% more orders. Gross sales rose by $5,271.69, but most of that went back out as promo discounts.

I used the same calendar dates for last year. Aug 3, 2025 fell on a Sunday and Aug 3, 2026 on a Monday, so the weekday mix differs slightly.

Policy basis: and .
4,738 tokens in (3,121 from cache), 403 out
Code
Citations checked
Every section cited was one that was retrieved: Definitions §2, Lunch; Definitions §3, The Lunch 20 Promotion.

yr	orders	gross	disc	refunds	net
2025	1194	3498977	149721	63299	3285957
2026	1414	4026146	537326	27727	3461093

answered

$0.0301to answer, $0.0454 without the prompt cache
17.3 s11.0 s in the model
10steps, 15,860 tokens in, 1,364 out
1retry

Compared with the same four weeks last year, how much more did lunch bring in during the Lunch 20 promo?

What should happen

Each run

What the grader found in run 2

The query I wrote

The query the model wrote

What it answered

Asked

Looked up the policy

Retrieved 3 sections

Proposed a query

Guard passed

The database rejected it

Proposed a query

Guard passed

Ran it: 2 rows in 131 ms

Answered

Citations checked