Evaluation case sql-08, from the sql suite

Since the Seasonal Specials came back on September 1, which drink category has lost the most sales compared with the four weeks before?

Run 1 passRun 2 pass

Asked as Ridgeline Coffeebefore-afterowner-definitioncategory

What should happen

Answered, and the answer contains the figure from the query I wrote.

Four weeks either side of the launch and the list of drink categories are Definitions §3 and §4. House Favorites fell $720; Signature Drinks, the obvious guess, rose $150.

Each run

Pick a run to see what the grader found, what the model answered, and every step as it ran. Steps marked model were decided by Claude; steps marked code ran as fixed logic; a person appears only where someone approved or rejected something.

What the grader found in run 2

Passed.

The query I wrote

The question read the way the owner means it. Run on the frozen database for the figure the grader expects.

WITH drink_sales AS (
  SELECT c.name AS category, oi.total_cents, (o.created_at AT TIME ZONE 'America/Denver')::date AS local_day
  FROM order_items oi
  JOIN orders o ON o.id = oi.order_id
  JOIN items i ON i.id = oi.item_id
  JOIN categories c ON c.id = i.category_id
  WHERE o.state = 'completed'
    AND c.name IN ('Signature Drinks', 'House Favorites', 'Matcha', 'Bubble Tea', 'Flights')
)
SELECT category
FROM drink_sales
WHERE local_day BETWEEN '2026-08-04' AND '2026-09-28'
GROUP BY category
ORDER BY sum(total_cents) FILTER (WHERE local_day >= '2026-09-01')
       - sum(total_cents) FILTER (WHERE local_day < '2026-09-01')
LIMIT 1

The query the model wrote

Its last query, laid out for reading. The trace has it as sent. The guard added the restaurant filter and the row limit before it ran.

SELECT
  c.name,
  sum(
    CASE
      WHEN d >= date '2026-09-01' THEN oi.total_cents
      ELSE 0
    END
  ) after_c,
  sum(
    CASE
      WHEN d < date '2026-09-01' THEN oi.total_cents
      ELSE 0
    END
  ) before_c,
  sum(
    CASE
      WHEN d >= date '2026-09-01' THEN oi.total_cents
      ELSE - oi.total_cents
    END
  ) change_c
FROM
  (
    SELECT
      o.id,
      (
        o.created_at at time zone(
          SELECT
            timezone
          FROM
            merchants
        )
      )::date d
    FROM
      orders o
    WHERE
      o.state = 'completed'
  ) x
  JOIN order_items oi ON oi.order_id = x.id
  JOIN items i ON i.id = oi.item_id
  JOIN categories c ON c.id = i.category_id
WHERE
  x.d >= date '2026-08-04'
  AND x.d < date '2026-09-29'
  AND c.name IN (
    'Signature Drinks',
    'House Favorites',
    'Matcha',
    'Seasonal Specials',
    'Bubble Tea',
    'Flights'
  )
GROUP BY
  c.name
ORDER BY
  change_c

What it answered

House Favorites lost the most. Its sales were $8,377.25 in the four weeks after the September 1 launch (Sept 1–28, 2026), against $9,097.25 in the four weeks before (Aug 4–31). That is a drop of $720.00. The windows come from , and the drink categories from .

Matcha was the only other category that fell, by $215.00, from $9,232.50 to $9,017.50. Signature Drinks, Bubble Tea and Flights rose slightly. Seasonal Specials went from $0 to $9,712.00, because it isn't on the menu before September 1.

These figures are item sales on completed orders, using the restaurant's local time. They don't subtract discounts.

answered

$0.0142to answer, $0.0310 without the prompt cache
8.2 s7.2 s in the model
7steps, 11,516 tokens in, 798 out

Human
Asked
Since the Seasonal Specials came back on September 1, which drink category has lost the most sales compared with the four weeks before?
Model
Looked up the policy
“Seasonal Specials definition and drink categories”
3,250 tokens in (3,121 from cache), 60 out
Code
Retrieved 4 sections
Searched this restaurant's policies only. Threshold 0.4; the model saw only the sections above it.
- 0.69 Definitions §3, Seasonal Specials
- 0.64 Definitions §4, Drinks
- 0.45 Definitions §5, Free Drinks
- 0.44 House Policy §4, Discounts

Model

Proposed a query

Drink category sales, 4 weeks before vs after Sep 1

select c.name, sum(case when d>=date '2026-09-01' then oi.total_cents else 0 end) after_c, sum(case when d<date '2026-09-01' then oi.total_cents else 0 end) before_c, sum(case when d>=date '2026-09-01' then oi.total_cents else -oi.total_cents end) change_c
from (select o.id, (o.created_at at time zone (select timezone from merchants))::date d from orders o where o.state='completed') x
join order_items oi on oi.order_id=x.id join items i on i.id=oi.item_id join categories c on c.id=i.category_id
where x.d>=date '2026-08-04' and x.d<date '2026-09-29' and c.name in ('Signature Drinks','House Favorites','Matcha','Seasonal Specials','Bubble Tea','Flights')
group by c.name order by change_c

3,805 tokens in (3,121 from cache), 480 out

Code
Guard passed
- ✓ parses
- ✓ single statement
- ✓ select only
- ✓ no writes
- ✓ no unsafe functions
- ✓ allowed tables, categories, items, merchants, order_items, orders
- ✓ tenant scope, merchant_id=2 on categories.merchant_id, items.merchant_id, merchants.id, order_items.merchant_id, orders.merchant_id
- ✓ row limit, injected 500
Code
Ran it: 6 rows in 269 ms
name after_c before_c change_c
House Favorites 837725 909725 -72000
Matcha 901750 923250 -21500
Bubble Tea 63235 62510 725
Flights 72000 70500 1500
Signature Drinks 2457425 2442425 15000
Seasonal Specials 971200 0 971200
Model
Answered
House Favorites lost the most. Its sales were $8,377.25 in the four weeks after the September 1 launch (Sept 1–28, 2026), against $9,097.25 in the four weeks before (Aug 4–31). That is a drop of $720.00. The windows come from , and the drink categories from .

Matcha was the only other category that fell, by $215.00, from $9,232.50 to $9,017.50. Signature Drinks, Bubble Tea and Flights rose slightly. Seasonal Specials went from $0 to $9,712.00, because it isn't on the menu before September 1.

These figures are item sales on completed orders, using the restaurant's local time. They don't subtract discounts.
4,461 tokens in (3,121 from cache), 258 out
Code
Citations checked
Every section cited was one that was retrieved: Definitions §3, Seasonal Specials; Definitions §4, Drinks.

name	after_c	before_c	change_c
House Favorites	837725	909725	-72000
Matcha	901750	923250	-21500
Bubble Tea	63235	62510	725
Flights	72000	70500	1500
Signature Drinks	2457425	2442425	15000
Seasonal Specials	971200	0	971200

answered

$0.0142to answer, $0.0310 without the prompt cache
8.2 s7.2 s in the model
7steps, 11,516 tokens in, 798 out

Since the Seasonal Specials came back on September 1, which drink category has lost the most sales compared with the four weeks before?

What should happen

Each run

What the grader found in run 2

The query I wrote

The query the model wrote

What it answered

Asked

Looked up the policy

Retrieved 4 sections

Proposed a query

Guard passed

Ran it: 6 rows in 269 ms

Answered

Citations checked