KindWatch 24 / 25
AI regression suite on live data
Regression tests for the AI consultant that run real model calls against the live database.
Problem
Prompt or model changes silently break answers that used to work.
What it does
- Cases described in JSON: user phrasing, expected tool call, required and forbidden content
- Fixtures picked by DB queries, so tests are portable between DB copies
- Replay mode: re-runs real production conversations and compares escalation outcomes
How it works
- php yii ai-regression → real requests, nothing written to tickets
- Replay modes: point / last / first message of the dialogue
Stack & models
Yii2 consolePostgreSQLLLM API