Applied AI case study – Architecture

How a question becomes an answer

Every request goes through the same steps. The model decides what to try. Code decides what runs. A person decides what happens. The model cannot run a query, send a message or file a dispute. Only code and a person can, and the trace records every step.

• The model writes SQL, searches the policies and drafts actions. It cannot run a query, send a message or file a dispute.
• Code takes every query apart before it runs: no writes, one restaurant's rows only, read-only.
• Policy search is limited to one restaurant and to sections above a score threshold. Citations are checked.
• An action waits for a person. Approval produces a simulated record, not a real send.
• Every step is recorded with who did it, what it cost and how long it took.

The model picks a tool

The model gets the question and a description of the seven tables it may read. It has four tools. run_sql writes one SELECT and says what it answers. search_policy asks for the rules on a topic. propose drafts an action: a dispute, a text campaign, a comp or a note to the owner. decline turns the question down and says why. The model can call up to six tools in one request, in any order, or answer in plain text with none. None of the four can write, send or file anything. A question that tells the model to delete or change something, like the injection scenario, has no tool to do it with. tools.py

The guard

The guard takes the SQL apart with sqlglot into the parts the database would see, a parse tree, instead of reading it as text. A string match misses a write hidden behind a comment or odd casing. A parsed query cannot hide one. It runs eight checks: the query parses, it is one statement, it is a SELECT, it contains no write, it calls no function that reads files or reaches the network, it reads only the seven allowed tables, it is scoped to one restaurant, and it fits under a row limit. The scope check does not look for a restaurant filter in the query. It adds one. Every table in the query is replaced by that restaurant's rows, so a join or a subquery cannot reach another restaurant. A blocked query goes back to the model with the name of the check that failed, and the model can try again. If nothing runs, the trace ends as blocked. guard.py

The database

The query runs in a read-only transaction with a five-second statement timeout. If a write got past the guard, the database refuses it. If a query is slow, the database stops it. The model gets at most a hundred rows, and the trace keeps exactly the rows it got, so a wrong figure can be checked against what the model saw. execute.py

Policy retrieval

Each restaurant has three policy documents written as a manual, with numbered sections. Each section is stored with an embedding in Postgres, using pgvector. A search is limited to the asking restaurant before any score is computed, the same way the guard limits a query. Up to four sections come back, and only those scoring above 0.40. Sections under the bar are written to the trace but not shown to the model. When nothing clears the bar, the model is told so. I chose a threshold over a fixed top four because a question the policies do not cover should return nothing, not the nearest miss. The two restaurants' rules differ on purpose, so a section from the wrong restaurant would be a wrong decision, not just a wrong answer. search.py

Proposals and the checker

A proposal is structured: a kind, an order or customer, an amount or a discount, an audience, a draft message. Code checks it against a limits file kept beside each restaurant's policies: dispute windows per platform, the discount cap, the consent rule, the audience cap, the comp limit, the required opt-out wording. For a text campaign the checker counts the audience itself from the consent flags in the data and overwrites whatever the model claimed. A proposal ends pending, pending for the owner, or blocked with the reason. Nothing is sent, filed or comped at this point. proposals.py

The citation check

The model cites the sections it used in brackets, like [House Policy §2, Comps]. After the answer, code checks every citation against the sections that were retrieved. A citation to a section the model was not shown fails the check. So does an answer that used policy and cites nothing. The answer still goes out, and the trace shows the failed check. On the site a citation opens the section it names. loop.py

The approval step

A pending proposal waits on the trace page for a person to approve or reject it. Either choice appends a human step to the trace. Approval then appends a code step that writes a record of the action, labelled as simulated. No marketplace, phone or register is reached. This is the only way anything changes. The model has no step in it. proposals.py

The trace and the cost

Every step is a row: who did it, the model, code or a person, what it decided, how long it took. Each model turn records its tokens, including how many came from the prompt cache, and the trace prices the request from the published rates. The cost per question on the Evaluation page is the sum of those rows. The Scenarios and Evaluation pages are rendered from these rows. trace.py

Decisions

• Python and sqlglot over Node. sqlglot parses SQL into a tree and lets the guard rewrite it. The guard is the enforcement layer, so the API is written in the language that has the parser.
• No ORM. The model writes the SQL. An ORM writes SQL for you, which is the opposite of what this project does. Queries go through psycopg directly.
• Scoping by rewrite, not row-level security. Row-level security would do this job inside Postgres. I rewrite the query instead so the trace can show the filter being added. The data is generated, so there is nothing to protect.
• A threshold, not a top four. A question the policies do not cover should return nothing. The threshold is 0.40, and the Evaluation page says why that is still an open question.
• Sonnet 5.5 at medium effort. On the evaluation cases Opus passed fewer and cost two and a half times as much. Low effort did as well as medium. The evaluation chose the model.
• Prompt caching. The system prompt, the tool definitions and the table descriptions are the same for every question, about 3,120 tokens. They are cached, so every turn after the first reads them at a tenth of the price. On the first scenario's answering turn, 3,121 of 4,113 input tokens came from the cache. The trace records this on every turn and the cost per question includes it. Answers are not cached. Every question is a live run.
• Deterministic graders. No model grades another model. A case passes on a figure, a citation or a status the trace records.
• Two databases from one schema. The demo database serves the site and the reruns. The generator can append one day of orders to it at a time, and the job that would do that nightly is not built. The evaluation database is frozen so a score today compares with one next month.

What it does not do

• The two restaurants and their orders are generated. Nothing here is real sales.
• An approved action is recorded, not sent. No marketplace, SMS gateway or register is connected.
• There is no login. On the Ask page you pick a restaurant, the choice is stored in a cookie, and the server sends it to the API as the restaurant id. In a real product the id would come from the account.
• Each evaluation setting ran once. The pass rates have no spread yet.
• Tenant isolation is enforced in the application, not in the database.

Stack

• API. FastAPI, with psycopg for the database and sqlglot for the guard.
• Database. Postgres with pgvector for the policy embeddings.
• Embeddings. OpenAI text-embedding-3-small.
• Model. Claude Sonnet 5.5 at medium effort.
• Site. Next.js.
• Running it. Locally, one docker compose file runs the database, the API and the site. Deployed, the site will run on Vercel and the API and the database will run separately.