Head office often needs a one-off number: "How many leads came in last quarter, and from where?" or "Which location has the most missed appointments?" The answers live in the CRM, the software a group of exercise clinics uses to keep track of everyone who has shown interest. Each of those questions used to wait on a developer to write a database query.
Now they type the question. An AI model, Anthropic's Claude, writes the query, the app checks it and runs it, and the model writes a short answer from the results. Every answer shows the query and the assumptions behind it, and the person who asked can give it a thumbs up or down.
Limits the AI can't touch
This is a health business, so I didn't want safety to depend on the AI following instructions. These limits live in the database and in code the model can't touch:
- The AI's database account can only read about a dozen summary tables I set up for it. Health information, contact details, notes and payments aren't in any of them.
- Every query is taken apart and checked before it runs. Only a fixed list of operations is allowed, and results have to be totals and counts. A query that lists people, or tries to piece together one person's details, is rejected.
- Queries can only read, get five seconds, and return at most 200 rows.
- Once the AI writes its answer, the app checks every number in it against the actual results. If one doesn't match, the answer isn't shown.
- Answers are plain text, with no links or images. That closes off a known trick where an AI is fooled into hiding data inside a link to someone else's server.
The AI sees the question, a description of the tables, and the totals. It never sees names, contact details or health information. A question costs under ten cents.
It can't answer everything. When the data it needs isn't in its tables, it says so. Those gaps turned out to be the most useful thing it told me.
Where it stands
In its first two weeks, head office asked it 55 questions. Of the answers people rated, 21 got a thumbs up and 14 a thumbs down. What the thumbs-downs turned out to mean, and how I test the answers now, is the next post.
Rails 8.1, Postgres, the Anthropic API.