I built an agent that answers a restaurant owner's questions. Then I tried to find out what happens when it is wrong.

The runs

1. Run 1, October 30, 2026

   First run: 38 questions with the answer written down first

   • Ten sql questions where the answer turns on a definition: lift, new customer, regular, summer, net payout, item margin. The vendor's definitions are in the prompt. Each owner's own words are a Definitions document beside their policies, searched and cited like any other section.
   • Every gold query is its own file, headed by the question, so a reader can check it against the model's.
   • Fifteen policy questions, including two where the nearest section is the wrong one. Eight proposals that code has to check against the rules. Three questions that must be declined, one about the other restaurant, one that needs no search.
   • Twenty-four months of data with named menus, food costs, one price per item per channel, a promotion, loyalty rewards, seasonal items and regulars who stop coming, so these questions have answers in the data.
   • Retrieval threshold 0.50, a number picked from ten captured runs before this set existed.

   Result. 34 of 38, $0.44 for the run. One wrong-but-safe query in ten. Two rules hidden by the retrieval threshold. One citation slip caught by code. The threshold moves to 0.40 next.

2. Run 2, October 31, 2026this page

   Second run: the retrieval threshold moves from 0.50 to 0.40

   • One number changed, the similarity score a policy section needs before the model sees it, from 0.50 to 0.40. Nothing else: same questions, same gold, same model, same prompt.

   Result. 33 of 38, $0.46 for the run. One case fewer, and a different set of failures. The rule hidden at 0.47 is now found. A question the policies do not cover now pulls in a near section, which the model dismisses correctly and my grader counts as a fail anyway. The citation slip happened a second time. Recall 86% to 93%, precision 46% to 29%. The threshold stays. The next change is the grader's rule for not-covered cases and the citation label, not the model.

Experiment 1 of 4

Does it read a word the owner's way?

The question

A query can pass all eight safety checks and still answer a different question from the one the owner asked. I wanted to know how often that happens, because nothing in the system would tell me.

What I ran

10 questions where a word has to be read the owner's way: lunch, regular, summer, lift, net payout. For each one I wrote the query myself, ran it on the frozen database, and kept the figure. The model never sees my query. The grader reads every number in the model's answer and passes the case if one of them is within tolerance of mine.

What came back

9 of 10. The miss is a query that converted time zones in the wrong direction on a retry after a timeout, so September started five hours early. It passed the guard, ran clean, and came back $2.49 against my $2.64.

What I decided

Nothing catches this at runtime, and nothing in the guard could, because the query is safe. So I measure it instead of pretending to enforce it, and I put my query beside the model's on every case so a reader can see where the two read a word differently. The cost is that the measurement is only as good as the ten questions, and I wrote them.

WITH september AS (
  SELECT o.channel, o.fulfillment_type, o.gross_sales_cents,
         o.gross_sales_cents - o.discount_cents AS after_discount,
         coalesce(p.processing_fee_cents, 0) AS fee
  FROM orders o
  LEFT JOIN payments p ON p.order_id = o.id AND p.status = 'completed'
  WHERE o.state = 'completed'
    AND date_trunc('month', o.created_at AT TIME ZONE 'America/Chicago') = '2026-09-01'
)
SELECT round(
    avg(after_discount - fee) FILTER (WHERE channel = 'in_store')
  - avg(after_discount - gross_sales_cents * 0.25) FILTER (WHERE channel = 'doordash' AND fulfillment_type = 'delivery')
)
FROM september

WITH
  f AS (
    SELECT
      order_id,
      sum(processing_fee_cents) fee
    FROM
      payments
    WHERE
      status = 'completed'
    GROUP BY
      order_id
  )
SELECT
  o.channel,
  count(*) n,
  avg(
    o.gross_sales_cents - o.discount_cents - CASE
      WHEN o.channel = 'in_store' THEN coalesce(f.fee, 0)
      ELSE 0.25 * o.gross_sales_cents
    END
  ) AS avg_net
FROM
  orders o
  LEFT JOIN f ON f.order_id = o.id
WHERE
  o.state = 'completed'
  AND (
    o.channel = 'in_store'
    OR (
      o.channel = 'doordash'
      AND o.fulfillment_type = 'delivery'
    )
  )
  AND o.created_at >= timestamptz '2026-09-01 00:00' at time zone(
    SELECT
      timezone
    FROM
      merchants
  )
  AND o.created_at < timestamptz '2026-10-01 00:00' at time zone(
    SELECT
      timezone
    FROM
      merchants
  )
GROUP BY
  o.channel

Experiment 2 of 4

Does it find the right rule?

The question

When the owner asks what is allowed, the model searches their policies and only sees sections that score above a similarity threshold. If the right section scores under the line, the model is told nothing matched and answers as if there is no rule. I wanted to know how often that happens.

What I ran

15 policy questions, each with the section I expect cited. The first run used a threshold of 0.50, a number I picked on 23 October from ten captured runs because real matches scored above 0.55 and near misses below 0.49. Ten runs is not a measurement. It is a guess that happened to split cleanly. The second run changed that one number to 0.40 and nothing else.

What came back

11 of 15 both times, and the same score hides a trade. At 0.50 two misses were the same failure: the right section scored 0.47 and 0.28, the model was shown nothing, and it said the policies do not cover it. At 0.40 the 0.47 case passes. In its place a question the policies really do not cover, health inspections, pulls in the Allergies section at 0.41, and the model cites it to say it does not apply, which my grader counts as a fail. Recall went from 86% to 93%. Precision went from 46% to 29%.

What I decided

The two errors are still not equal. An extra section shown is mostly ignored, and when the model did lean on one it said out loud that it did not apply. A hidden one makes the model say there is no rule, and the owner has no way to see through that. So 0.40 stays. What it bought was one rescued rule. What it cost was 17 points of precision and one grade, and what it exposed is that my rule for a not-covered case, nothing shown and nothing cited, is stricter than the behaviour I want. That rule is mine to fix and I have not decided how. The miss at 0.28 is unchanged. No threshold reaches it. That one is the wording of the search, and it is still open.

Experiment 3 of 4

Does it cite only what it was shown?

The question

An answer that cites a section the model was never shown is an answer made up. A code step compares every citation in the answer with the sections retrieved. I wanted to see it fire.

What I ran

Every policy case runs through it. The grader recomputes the check from the answer text, so a regrade uses today's parser and not the one from the day of the run.

What came back

It fired twice across the two runs, both times on an answer that was right. The model cited the first section with its full label, then shortened the next ones to [§5] or [§4, Frequency], and the step failed them. A formatting slip, caught by code, and counted against the model because that is what the check is for.

What I decided

This is the check doing its job, and it is also the check being literal. A reader knows which document [§4, Frequency] means when only one document was shown. I could let the step accept a short label when it matches exactly one retrieved section, or tell the model in the prompt to write the full label every time. I lean to the second, because the check is the thing I trust and I would rather not teach it to guess. Not done yet. What the step still does not do is read. It knows a cited section was shown, not that the section says what the answer claims. That is the third kind of wrong, the one with no code behind it, and I have not built it because I do not yet know how to check support without a second model whose own errors I would then have to measure.

Experiment 4 of 4

Does code hold the line on actions?

The question

When the model proposes an action, code re-derives every rule from the limits file and the database before a person sees it. I wanted to know whether anything that breaks a rule gets through.

What I ran

8 proposals: disputes inside and outside their window, campaigns over and under the discount cap, an audience the model counted wrong, a comp over the manager's limit. And 5 questions that must be declined, scoped to the asking restaurant, or answered without a search.

What came back

8 of 8, and 5 of 5. The checker counted 563 customers with verified consent where the model said everyone with a phone number, 29,762 of them. It blocked a 25% campaign under a 20% cap and a dispute filed 26 days after a 7-day window.

What I decided

This is the strongest part of the system, and the reason is structural, not clever: the model's proposal is an input, and the verdict comes from code reading the same rules a person would. It is the part worth extending. The cost is that every rule has to be written down as a limit a program can check, and the owners' policies are written for people.

Every case

Limitations