All 38 cases

Evaluation case sql-06, from the sql suite

What was our average tip in September 2026?

Run 1 passRun 2 pass

Asked as Mekong Noodle Housetip-nullambiguous

What should happen

Answered, and the answer contains the figure from the query I wrote.

No qualifier this time. Marketplace orders carry no tip at all (null), so treating null as zero gives $1.36 instead of $4.95.

Each run

Pick a run to see what the grader found, what the model answered, and every step as it ran. Steps marked model were decided by Claude; steps marked code ran as fixed logic; a person appears only where someone approved or rejected something.

What the grader found in run 2

Passed.

The query I wrote

The question read the way the owner means it. Run on the frozen database for the figure the grader expects.

SELECT avg(tip_cents)
FROM orders
WHERE state = 'completed'
  AND date_trunc('month', created_at AT TIME ZONE 'America/Chicago') = '2026-09-01'

The query the model wrote

Its last query, laid out for reading. The trace has it as sent. The guard added the restaurant filter and the row limit before it ran.

SELECT
  count(*) AS n_orders,
  count(tip_cents) AS n_with_tip,
  avg(tip_cents) AS avg_tip_cents,
  avg(tip_cents) FILTER (
    WHERE
      tip_cents > 0
  ) AS avg_nonzero
FROM
  orders
WHERE
  state = 'completed'
  AND (
    created_at at time zone(
      SELECT
        timezone
      FROM
        merchants
    )
  ) >= '2026-09-01'
  AND (
    created_at at time zone(
      SELECT
        timezone
      FROM
        merchants
    )
  ) < '2026-10-01'

What it answered

Our average tip in September 2026 was about $4.95. That covers completed orders from September 1 to 30, using local time.

Only 2,190 of the 7,985 completed orders had a tip recorded. The other orders have no tip figure, because some channels don't pass tips through. I left those out of the average instead of counting them as $0.

answered

  • $0.0057to answer, $0.0170 without the prompt cache
  • 3.8 s3.7 s in the model
  • 4steps, 6,775 tokens in, 342 out
  1. Human

    Asked

    What was our average tip in September 2026?

  2. Model

    Proposed a query

    Average tip on completed orders in September 2026, non-null tips only, local time

    select count(*) as n_orders, count(tip_cents) as n_with_tip, avg(tip_cents) as avg_tip_cents, avg(tip_cents) filter (where tip_cents>0) as avg_nonzero from orders where state='completed' and (created_at at time zone (select timezone from merchants)) >= '2026-09-01' and (created_at at time zone (select timezone from merchants)) < '2026-10-01'

    3,223 tokens in (3,121 from cache), 235 out

  3. Code

    Guard passed

    • ✓ parses
    • ✓ single statement
    • ✓ select only
    • ✓ no writes
    • ✓ no unsafe functions
    • ✓ allowed tables, merchants, orders
    • ✓ tenant scope, merchant_id=1 on merchants.id, orders.merchant_id
    • ✓ row limit, injected 500
  4. Code

    Ran it: 1 row in 57 ms

    n_ordersn_with_tipavg_tip_centsavg_nonzero
    79852190495.2283105022831495.2283105022831
  5. Model

    Answered

    Our average tip in September 2026 was about $4.95. That covers completed orders from September 1 to 30, using local time.

    Only 2,190 of the 7,985 completed orders had a tip recorded. The other orders have no tip figure, because some channels don't pass tips through. I left those out of the average instead of counting them as $0.

    3,552 tokens in (3,121 from cache), 107 out

answered

  • $0.0057to answer, $0.0170 without the prompt cache
  • 3.8 s3.7 s in the model
  • 4steps, 6,775 tokens in, 342 out