Evaluation case sql-09, from the sql suite

Which of the Seasonal Specials is worth keeping on the menu all year?

Run 1 passRun 2 pass

Asked as Ridgeline Coffeemarginowner-definitionmenu-engineering

What should happen

Answered, and the answer contains the figure from the query I wrote.

Worth keeping means the most total margin last season, Definitions §3; total margin is the vendor's. Maple Pecan Cold Brew, $7,183; the best seller by units is also it, but Apple Cider Chai is second by units and last by margin.

Each run

Pick a run to see what the grader found, what the model answered, and every step as it ran. Steps marked model were decided by Claude; steps marked code ran as fixed logic; a person appears only where someone approved or rejected something.

What the grader found in run 2

Passed.

The query I wrote

The question read the way the owner means it. Run on the frozen database for the figure the grader expects.

SELECT i.name
FROM order_items oi
JOIN orders o ON o.id = oi.order_id
JOIN items i ON i.id = oi.item_id
JOIN categories c ON c.id = i.category_id
WHERE o.state = 'completed'
  AND c.name = 'Seasonal Specials'
  AND (o.created_at AT TIME ZONE 'America/Denver')::date BETWEEN '2025-09-01' AND '2025-12-31'
GROUP BY i.name
ORDER BY sum(oi.quantity * (oi.unit_price_cents - i.cost_cents)) DESC
LIMIT 1

The query the model wrote

Its last query, laid out for reading. The trace has it as sent. The guard added the restaurant filter and the row limit before it ran.

SELECT
  i.name,
  sum(oi.quantity) units,
  sum(oi.quantity * (oi.unit_price_cents - i.cost_cents)) margin_cents,
  count(
    DISTINCT CASE
      WHEN o.channel IN ('doordash', 'uber_eats', 'grubhub') THEN o.id
    END
  ) mkt_orders
FROM
  order_items oi
  JOIN items i ON i.id = oi.item_id
  JOIN categories c ON c.id = i.category_id
  JOIN orders o ON o.id = oi.order_id
WHERE
  c.name = 'Seasonal Specials'
  AND o.state = 'completed'
  AND (
    o.created_at at time zone(
      SELECT
        timezone
      FROM
        merchants
    )
  ) >= '2025-09-01'
  AND (
    o.created_at at time zone(
      SELECT
        timezone
      FROM
        merchants
    )
  ) < '2026-01-01'
GROUP BY
  i.name
ORDER BY
  margin_cents DESC

What it answered

The Maple Pecan Cold Brew is the one to keep all year. Your Seasonal Specials definition says to pick the special that earned the most total margin last season, September through December 2025 . I counted completed orders only. Margin is the price paid minus food cost, multiplied by units.

The Maple Pecan Cold Brew earned $7,183.20 on 1,460 units. The others:

Special	Margin	Units
Peppermint Mocha	$6,644.40	1,356
Pumpkin Spice Latte	$6,414.24	1,288
Apple Cider Chai	$6,255.16	1,412

No commission is deducted. Ridgeline isn't on any delivery marketplace , and no marketplace orders contain these items.

answered

$0.0174to answer, $0.0399 without the prompt cache
9.8 s8.3 s in the model
9steps, 15,535 tokens in, 883 out

Human
Asked
Which of the Seasonal Specials is worth keeping on the menu all year?
Model
Looked up the policy
“Seasonal Specials definition, which items or category; keep on menu criteria”
3,236 tokens in (3,121 from cache), 69 out
Code
Retrieved 4 sections
Searched this restaurant's policies only. Threshold 0.4; the model saw only the sections above it.
- 0.73 Definitions §3, Seasonal Specials
- 0.46 Definitions §4, Drinks
- 0.45 Definitions §2, Regulars
- 0.42 Customer Messaging Policy §5, Offers in Messages
Model
Looked up the policy
“marketplace commission rate DoorDash Uber Eats Grubhub”
3,762 tokens in (3,121 from cache), 124 out
Code
Retrieved 1 section
Searched this restaurant's policies only. Threshold 0.4; the model saw only the sections above it.
- 0.53 Online Ordering and Loyalty Policy §2, Delivery Marketplaces
- 0.32 House Policy §2, Remakes and Comps (below threshold)
- 0.32 Online Ordering and Loyalty Policy §3, Pickup Orders (below threshold)
- 0.32 House Policy §5, Online Orders (below threshold)

Model

Proposed a query

Total margin per Seasonal Specials item, Sep-Dec 2025, completed orders

select i.name, sum(oi.quantity) units, sum(oi.quantity*(oi.unit_price_cents-i.cost_cents)) margin_cents, count(distinct case when o.channel in ('doordash','uber_eats','grubhub') then o.id end) mkt_orders
from order_items oi join items i on i.id=oi.item_id join categories c on c.id=i.category_id join orders o on o.id=oi.order_id
where c.name='Seasonal Specials' and o.state='completed'
and (o.created_at at time zone (select timezone from merchants)) >= '2025-09-01' and (o.created_at at time zone (select timezone from merchants)) < '2026-01-01'
group by i.name order by margin_cents desc

4,004 tokens in (3,121 from cache), 375 out

Code
Guard passed
- ✓ parses
- ✓ single statement
- ✓ select only
- ✓ no writes
- ✓ no unsafe functions
- ✓ allowed tables, categories, items, merchants, order_items, orders
- ✓ tenant scope, merchant_id=2 on categories.merchant_id, items.merchant_id, merchants.id, order_items.merchant_id, orders.merchant_id
- ✓ row limit, injected 500
Code
Ran it: 4 rows in 42 ms
name units margin_cents mkt_orders
Maple Pecan Cold Brew 1460 718320 0
Peppermint Mocha 1356 664440 0
Pumpkin Spice Latte 1288 641424 0
Apple Cider Chai 1412 625516 0
Model
Answered
The Maple Pecan Cold Brew is the one to keep all year. Your Seasonal Specials definition says to pick the special that earned the most total margin last season, September through December 2025 . I counted completed orders only. Margin is the price paid minus food cost, multiplied by units.

The Maple Pecan Cold Brew earned $7,183.20 on 1,460 units. The others:

Special Margin Units
Peppermint Mocha $6,644.40 1,356
Pumpkin Spice Latte $6,414.24 1,288
Apple Cider Chai $6,255.16 1,412

No commission is deducted. Ridgeline isn't on any delivery marketplace , and no marketplace orders contain these items.
4,533 tokens in (3,121 from cache), 315 out
Code
Citations checked
Every section cited was one that was retrieved: Definitions §3, Seasonal Specials; Online Ordering and Loyalty Policy §2, Delivery Marketplaces.

name	units	margin_cents
Maple Pecan Cold Brew	1460	718320
Peppermint Mocha	1356	664440
Pumpkin Spice Latte	1288	641424
Apple Cider Chai	1412	625516

answered

$0.0174to answer, $0.0399 without the prompt cache
9.8 s8.3 s in the model
9steps, 15,535 tokens in, 883 out

Which of the Seasonal Specials is worth keeping on the menu all year?

What should happen

Each run

What the grader found in run 2

The query I wrote

The query the model wrote

What it answered

Asked

Looked up the policy

Retrieved 4 sections

Looked up the policy

Retrieved 1 section

Proposed a query

Guard passed

Ran it: 4 rows in 42 ms

Answered

Citations checked