All 38 cases

Evaluation case sql-07, from the sql suite

What did we net from DoorDash delivery orders in September 2026 after their commission?

Run 1 passRun 2 pass

Asked as Mekong Noodle Housesql+policycommission

What should happen

Answered, and the answer contains the figure from the query I wrote.

The 25% rate is in the marketplace terms, not the database, and net payout takes discounts off too (the vendor's definition). Commission on gross alone gives $37,452 instead of $34,863.

Each run

Pick a run to see what the grader found, what the model answered, and every step as it ran. Steps marked model were decided by Claude; steps marked code ran as fixed logic; a person appears only where someone approved or rejected something.

What the grader found in run 2

Passed.

The query I wrote

The question read the way the owner means it. Run on the frozen database for the figure the grader expects.

SELECT round(sum(gross_sales_cents * 0.75 - discount_cents))
FROM orders
WHERE state = 'completed'
  AND channel = 'doordash'
  AND fulfillment_type = 'delivery'
  AND date_trunc('month', created_at AT TIME ZONE 'America/Chicago') = '2026-09-01'

The query the model wrote

Its last query, laid out for reading. The trace has it as sent. The guard added the restaurant filter and the row limit before it ran.

SELECT
  count(*) n,
  sum(gross_sales_cents) gross,
  sum(discount_cents) disc,
  sum(gross_sales_cents * 0.25) comm
FROM
  orders
WHERE
  channel = 'doordash'
  AND fulfillment_type = 'delivery'
  AND state = 'completed'
  AND (
    created_at AT TIME ZONE(
      SELECT
        timezone
      FROM
        merchants
    )
  ) >= '2026-09-01'
  AND (
    created_at AT TIME ZONE(
      SELECT
        timezone
      FROM
        merchants
    )
  ) < '2026-10-01'

What it answered

For September 2026 (local time), DoorDash delivery orders netted about $34,862.65 after commission. This covers 1,269 completed orders.

  • Gross sales were $49,935.79.
  • Discounts were $2,589.19.
  • Commission was about $12,483.95, which is 25% of gross sales. That is the DoorDash delivery rate .

Net is gross minus discounts minus commission. DoorDash orders have no processing fee deducted.

answered

  • $0.0106to answer, $0.0274 without the prompt cache
  • 6.2 s5.9 s in the model
  • 7steps, 10,930 tokens in, 557 out
  1. Human

    Asked

    What did we net from DoorDash delivery orders in September 2026 after their commission?

  2. Model

    Looked up the policy

    “DoorDash commission rate”

    3,237 tokens in (3,121 from cache), 83 out

  3. Code

    Retrieved 2 sections

    Searched this restaurant's policies only. Threshold 0.4; the model saw only the sections above it.

    • 0.62 Marketplace Terms §2, Commission and Pricing
    • 0.56 Definitions §5, Marketplaces
    • 0.39 Marketplace Terms §6, Promotions (below threshold)
    • 0.37 Marketplace Terms §3, Missing or Incorrect Items (below threshold)
  4. Model

    Proposed a query

    DoorDash delivery Sept 2026 gross, discounts, orders

    select count(*) n, sum(gross_sales_cents) gross, sum(discount_cents) disc, sum(gross_sales_cents*0.25) comm from orders where channel='doordash' and fulfillment_type='delivery' and state='completed' and (created_at AT TIME ZONE (SELECT timezone FROM merchants)) >= '2026-09-01' and (created_at AT TIME ZONE (SELECT timezone FROM merchants)) < '2026-10-01'

    3,685 tokens in (3,121 from cache), 253 out

  5. Code

    Guard passed

    • ✓ parses
    • ✓ single statement
    • ✓ select only
    • ✓ no writes
    • ✓ no unsafe functions
    • ✓ allowed tables, merchants, orders
    • ✓ tenant scope, merchant_id=1 on merchants.id, orders.merchant_id
    • ✓ row limit, injected 500
  6. Code

    Ran it: 1 row in 15 ms

    ngrossdisccomm
    126949935792589191248394.75
  7. Model

    Answered

    For September 2026 (local time), DoorDash delivery orders netted about $34,862.65 after commission. This covers 1,269 completed orders.

    • Gross sales were $49,935.79.
    • Discounts were $2,589.19.
    • Commission was about $12,483.95, which is 25% of gross sales. That is the DoorDash delivery rate .

    Net is gross minus discounts minus commission. DoorDash orders have no processing fee deducted.

    4,008 tokens in (3,121 from cache), 221 out

  8. Code

    Citations checked

    Every section cited was one that was retrieved: Marketplace Terms §2, Commission and Pricing.

answered

  • $0.0106to answer, $0.0274 without the prompt cache
  • 6.2 s5.9 s in the model
  • 7steps, 10,930 tokens in, 557 out