All 39 cases

Evaluation case sql-11, from the sql suite

How many catering orders did we do in September?

Asked as Mekong Noodle House

Run 1 not runRun 2 not runRun 3 not runRun 4 not runRun 5 not runRun 6 not runRun 7 failRun 8 pass

What should happen

The agent should answer, and the answer should have 22 in it, give or take 1%. I got that figure by running the query I wrote on the frozen database.

A catering order is one with an item from the Catering category on the menu. Nothing on the order itself says catering: no channel, no label, no discount. Three ad hoc asks on 1 November, about October, all searched the order's labels, found nothing, and said the data cannot identify catering orders. The first version asked about October, the month in progress on the frozen clock, and the agent rightly stopped at the 24th while the gold did not.

Each run

Pick a run to see what the grader found, what the model answered, and every step as it ran. Steps marked model were decided by Claude; steps marked code ran as fixed logic; a person appears only where someone approved or rejected something.

What the grader found in run 8

Passed.

What the run 8 trace shows

  • Passed. The first query listed the Catering category and its six items. The second counted completed September orders with one of them: 22.
  • Four lines in the prompt. Without them, the September wording found the menu two times in three and the October wording never. With them, three of three and this run.

The query I wrote

The question read the way the owner means it. Run on the frozen database for the figure the grader expects.

SELECT count(DISTINCT o.id)
FROM orders o
JOIN order_items oi ON oi.order_id = o.id
JOIN items i ON i.id = oi.item_id
JOIN categories c ON c.id = i.category_id
WHERE o.state = 'completed'
  AND c.name = 'Catering'
  AND o.created_at >= timestamp '2026-09-01' AT TIME ZONE 'America/Chicago'
  AND o.created_at <  timestamp '2026-10-01' AT TIME ZONE 'America/Chicago'

The query the model wrote

Its last query, laid out for reading. The trace has it as sent. The guard added the restaurant filter and the row limit before it ran.

SELECT
  count(DISTINCT o.id) orders,
  sum(o.gross_sales_cents) gross
FROM
  orders o
  JOIN order_items oi ON oi.order_id = o.id
  JOIN items i ON i.id = oi.item_id
WHERE
  i.category_id = 9
  AND o.state = 'completed'
  AND o.created_at >= timestamp '2026-09-01' AT TIME ZONE(
    SELECT
      timezone
    FROM
      merchants
  )
  AND o.created_at < timestamp '2026-10-01' AT TIME ZONE(
    SELECT
      timezone
    FROM
      merchants
  )

What it answered

We did 22 catering orders in September 2026. I counted completed orders that include at least one item from the Catering menu category, using the restaurant's local time. Those orders had $3,634.62 in gross sales. That total is the whole order, so it includes any non-catering items on the same order.

The house policies don't define "catering" beyond the menu category .

answered

  • $0.0129to answer, $0.0373 without the prompt cache
  • 7.5 s7.2 s in the model
  • 10steps, 15,749 tokens in, 582 out
  1. Human

    Asked

    How many catering orders did we do in September?

  2. Model

    Looked up the policy

    “catering order definition”

    3,495 tokens in (3,393 from cache), 83 out

  3. Code

    Retrieved 2 sections

    Searched this restaurant's policies only, by meaning and by words. Threshold 0.4 on the meaning score; the model saw the sections above it, and the one section whose words matched best.

    • 0.56 House Policy §6, Catering
    • 0.31 House Policy §4, Discounts (shown for its words: the best word match, under the threshold)
    • 0.34 House Policy §1, Scope and Authority (not shown)
    • 0.31 Definitions §3, The Lunch 20 Promotion (not shown)
  4. Model

    Proposed a query

    Find catering category/items

    select c.id, c.name as category, count(i.id) items from categories c left join items i on i.category_id=c.id where c.name ilike '%cater%' group by 1,2

    3,862 tokens in (3,393 from cache), 130 out

  5. Code

    Guard passed

    • ✓ parses
    • ✓ single statement
    • ✓ select only
    • ✓ no writes
    • ✓ no unsafe functions
    • ✓ allowed tables, categories, items
    • ✓ tenant scope, merchant_id=1 on categories.merchant_id, items.merchant_id
    • ✓ row limit, injected 500
  6. Code

    Ran it: 1 row in 6 ms

    idcategoryitems
    9Catering6
  7. Model

    Proposed a query

    Count completed September orders containing catering items

    select count(distinct o.id) orders, sum(o.gross_sales_cents) gross from orders o join order_items oi on oi.order_id=o.id join items i on i.id=oi.item_id where i.category_id=9 and o.state='completed' and o.created_at >= timestamp '2026-09-01' AT TIME ZONE (SELECT timezone FROM merchants) and o.created_at < timestamp '2026-10-01' AT TIME ZONE (SELECT timezone FROM merchants)

    4,047 tokens in (3,393 from cache), 248 out

  8. Code

    Guard passed

    • ✓ parses
    • ✓ single statement
    • ✓ select only
    • ✓ no writes
    • ✓ no unsafe functions
    • ✓ allowed tables, items, merchants, order_items, orders
    • ✓ tenant scope, merchant_id=1 on items.merchant_id, merchants.id, order_items.merchant_id, orders.merchant_id
    • ✓ row limit, injected 500
  9. Code

    Ran it: 1 row in 29 ms

    ordersgross
    22363462
  10. Model

    Answered

    We did 22 catering orders in September 2026. I counted completed orders that include at least one item from the Catering menu category, using the restaurant's local time. Those orders had $3,634.62 in gross sales. That total is the whole order, so it includes any non-catering items on the same order.

    The house policies don't define "catering" beyond the menu category .

    4,345 tokens in (3,393 from cache), 121 out

  11. Code

    Citations checked

    Every section cited was one that was retrieved: House Policy §6, Catering.

answered

  • $0.0129to answer, $0.0373 without the prompt cache
  • 7.5 s7.2 s in the model
  • 10steps, 15,749 tokens in, 582 out