A change that fixed one question broke a different one, and every test still passed. That was how I learned that normal tests, which check that code does what it says, can't tell you whether an AI's answers got worse. The questions were going to the data chat from the previous post: an AI that answers head office's questions about the clinics' CRM by writing the database query itself.
So there's a list of about thirty real questions from head office, each saved with an answer we checked. Whenever someone changes the AI's instructions or the tables it can see, the list runs again and reports which answers changed. That report goes into the code review. A full run costs about $3.
A few of those questions also appear as worked examples in the AI's instructions, which would flatter the score. So it's reported twice: over all the questions, and over only the ones the AI hasn't seen.
Every thumbs-down gets a reason
Every answer is also saved with a fingerprint of the exact instructions and model that produced it, so ratings can be compared before and after a change. A thumbs down comes with what the person wanted. Then each miss gets a reason: missing data, an unclear table description, a wrong query, or a refusal that was actually right. An AI helps sort each week's misses, but only after it matched our own sorting on 10 of 12 examples.
The test run doesn't use an AI to grade answers. It compares them with the saved ones. Checking an AI grader properly takes hundreds of human ratings, and there aren't that many yet.
Where it stands
The first dozen thumbs-downs surprised me. None was the AI writing a wrong query over data it had. Seven were questions about data it couldn't see yet. Two were correct refusals to list people. Since then, most of the work has gone into the tables and their descriptions, and very little into the AI's instructions.
Rails 8.1, Postgres, the Anthropic API.