Monitor an AI agent on four signals: quality drift, hold rate, spend per run, and latency. Uptime alone does not show whether the answers are still good.
This is the part teams skip. An automation is launched, it works, and attention moves elsewhere. It can get worse for months before somebody notices. Most problems show up as a slow drop in quality.
Why an automation that worked stops working
Its inputs and its model keep changing. The documents change format when a supplier updates their template. A new product line appears with vocabulary the agent has never seen. The model provider ships an update and behaviour shifts a little. Volume doubles in a seasonal peak and the retry logic starts costing real money. None of these are bugs, and none of them will fail a health check.
| Signal | Healthy | What a change means |
|---|---|---|
| Quality drift | Sampled accuracy stable week to week | Inputs have moved, or the model has |
| Hold rate | Falling, then steady | Rising means new input classes it does not recognise |
| Spend per run | Flat | Rising means retries, longer inputs, or a loop |
| Latency | Slowest requests stay steady | Spikes precede timeouts and duplicate work |
Hold rate is the most useful of the four. It shows how often the agent meets work it does not recognise.
What to alert on
Alert when a rate changes. A hold rate that doubles in a day is worth a look, and it often shows up before quality drops.
- /01
Hold rate clearly above the trailing week. Something changed in the inputs.
- /02
Spend per run clearly up. Retries, longer documents, or a loop that has not tripped the limit yet.
- /03
Sampled accuracy falling. Pull a sample and read the evidence fields before touching thresholds.
- /04
Any run that hits a limit. Someone should look at it the same day.
- /05
Zero holds for a week. That usually means a threshold set too low to ever trigger.
How to check quality without reading every run
Pull a small random sample of runs each week, plus every run that held. The person who owns the process reads each decision and its evidence. This works because each record stores what the agent decided and why, so review is a matter of reading.
One named person watches it
One named person checks it on a fixed schedule. A dashboard with no named reader is as good as none.
- Sample
- A few runs at random, plus all holds
- Compare
- Against the trailing week
- Watch
- Hold rate, spend, accuracy
- Owner
- Named, by role
- Output
- Threshold changes, with reasons
Surehand operates what it deploys after launch. That is when thresholds get corrected against real inputs rather than estimates, and when the hold rate should settle as the deployment does.
GatehouseContinuous evaluation on Gatehouse, Surehand's control plane, re-checks past decisions against the current policy. See Gatehouse
Keep reading
Why do AI projects fail?
An MIT NANDA report found most generative AI pilots showed no measurable return. This note covers the work around the model.
What should an AI audit trail contain?
Seven fields, and why logs are not an audit trail. What your buyer's risk function will ask for, and what most systems cannot produce.
