What should an AI audit trail contain?

An audit trail for an AI agent is a record of each run. It shows what the agent could touch, what it decided and why, what it spent, and who answers for it. If your system cannot show that for one named run six months later, what you have is logs.

The distinction matters because your buyer's risk function will test evidence before accuracy. In regulated back offices, the questionnaire often arrives before the budget. Each question on it asks the same thing: show me what happened.

The difference between logs and a record

Logs are written for the engineer on call. Records are written for a person outside the team reconstructing a decision later. They look similar and serve opposite purposes. A log line is a fragment, written when it was handy and deleted after a few weeks. Only people who know the code can read it. A record is complete at the moment the run closes, and addressed to a reader who was not there.

What the record has to contain

The record needs seven fields. With fewer than seven there is a question a reasonable auditor can ask that you cannot answer.

1. /01

   Identity. Which agent, which version, which model and prompt revision. “Our AI” is not an answer when the behaviour changed in March.

2. /02

   Scope. The permissions in force for that run: what it could read, what it could act on, what it was forbidden to touch. Scope is what turns “it should not have done that” into a testable claim.

3. /03

   Input. The exact material the decision was made on, or a reference to it that can't change. A claim number is not the input. The document is.

4. /04

   Evidence. The specific passages, rows or fields the decision rested on, cited rather than summarised. This is the field most often missing.

5. /05

   Decision and confidence, against the threshold in force at the time. A score that cleared a bar set before the run is a decision you can defend. The same score with no recorded bar is a number.

6. /06

   Cost. Tokens, calls and money for that run, against the limit that applied. Spend is a control, and a control nobody measures is not a control.

7. /07

   Person. Who approved it, or the rule that let it proceed without one. The record names a person.

GatehouseThat is the sealed record of RUN-7F2K4, the simulated run on Gatehouse, Surehand's control plane. INV-40218 for 2,400.00 was held on a changed bank detail and declined by the AP lead. Read the run on Gatehouse ↗

Why “the model said so” fails an audit

A confidence score describes the model's output. On its own it does not explain the decision. A decision you can defend pairs the score with a threshold set before the run, and with the evidence behind the score. Recorded together, those three show that a policy was followed. Recorded apart, or not at all, every disputed run becomes your word against the customer's.

It is also why a screenshot of a chat is not evidence. It shows an answer. It does not show scope, threshold, cost or accountability, and anyone with the same prompt can produce one after the fact.

The five questions your buyer will ask

1. /01

   Show me one run end to end, from a date I choose rather than one you prepared.

2. /02

   Who could have stopped this, and did they see it before it happened?

3. /03

   What did this decision rest on, and can I read that source myself?

4. /04

   What would it have cost if this ran wrong for a week before anyone noticed?

5. /05

   If we end the contract, what do we keep, and in what format?

A system that seals runs answers all five by opening the record. A system that logs answers them by asking an engineer to reconstruct something from fragments, if it answers them at all.

How long you keep it, and who can read it

Keep the record for as long as your obligations require. Many insurance and finance obligations run to years, and the record has to outlive both the agent and the vendor. Press any vendor on that last point.

What it costs to add this later

Adding records later costs more than starting with them, and most of that cost is not engineering time. Without early records, you cannot answer questions about anything that already ran, which is the period a first audit examines. Teams learn this late. The pilot worked and the buyer said yes. Then the security review asked for six months of records nobody kept.

Every run Surehand deploys is sealed from the first day. A record rebuilt after someone asks is much weaker evidence.