How Do You Keep an Audit Trail for AI Automations?

How do you keep an audit trail for AI automations?

Record every run as a row, not a log line. Each row needs an identifier, the input, the prompt version, the model, the output, the decision taken, the cost and who could see it. Store it where a non-engineer can search it a year later, because that is when somebody asks.

Almost every AI automation we inherit has the same gap. It works, it saves hours, and nobody can say what it did last month. The moment a customer disputes an outcome, the team is reading a chat history and guessing.

This is the setup we build, and the reasoning behind each field.

What counts as an audit trail, not just logs?

Logs tell you the system ran. An audit trail tells you what it decided and why, in a form somebody can inspect without your help. The test is simple: can a colleague answer "why did this customer get that email" in under five minutes, without opening a terminal?

Most stacks have the first and not the second. Platform run histories expire, and a raw log of API calls does not record the decision in business terms.

So we treat the trail as data, not as output. One table, one row per run, queryable, with retention you chose on purpose rather than whatever your tooling defaults to.

What does the law already require?

More than most teams realise, if your system is high risk. The EU AI Act, Regulation 2024/1689, says in Article 12 that high-risk AI systems shall technically allow for the automatic recording of events over the lifetime of the system, to support post-market monitoring and deployer oversight.

The dates matter for planning. The obligations for high-risk systems listed in Annex III apply from 2 December 2027, and for those under Annex I from 2 August 2028. That is not far away for anything you are building now and expect to still run then.

Two honest caveats. Most marketing and web automations are not high-risk systems under the Act, and Article 12 does not set a retention period. We still use its logic as the design brief, because a regulator's minimum makes a decent engineering floor.

What should every AI run record?

Nine fields cover almost everything. A run identifier, a timestamp, the trigger and source record, the model name, the prompt version, the input actually sent, the raw output, the action taken, and the cost. Add the reviewer if a person approved it.

The two people skip are the ones that matter most. Prompt version is how you explain why August looks different from July. Action taken is how you separate what the model said from what your code then did with it.

Keep the field names boring and stable. A trail you rename every quarter cannot be queried across the period anyone will ask about.

How do you log the prompt without storing private data?

Split the prompt from the payload. Store the prompt template once, by version, and store only a reference plus a redacted copy of the variable content. That way you can reconstruct any run without keeping a second copy of personal data in a table nobody thought about.

Redact at write time, not at read time. Email addresses, phone numbers, contract values and free-text notes should be masked or hashed before they land, because a log table quietly becomes the least protected copy of your customer data.

Write down a retention period and enforce it with a scheduled job. We usually pick thirteen months, because it lets you compare a full year plus the current month. Our notes on versioning and testing prompts cover how we keep templates addressable.

Where should the logs live?

In your own database, not in the automation tool. Platform run histories are built for debugging this week, and they are scoped to the vendor. A table in Postgres that your team already queries survives a tooling change, and it can be joined to the CRM record that caused the run.

Tooling still helps at the edges. OpenTelemetry maintains a dedicated repository of generative AI semantic conventions covering spans, metrics and events for AI clients, along with conventions for the Model Context Protocol and specific providers. If you already run traces, use the standard names rather than inventing your own.

Keep both layers. Traces answer where the latency went. The audit table answers what happened to that customer.

How do you make the trail searchable a year later?

Index the three fields people search by: the source record, the date, and the outcome. Nobody searches by prompt text. They search by a customer, a week, or the thing that went wrong, and if those are not indexed the table becomes a warehouse with no doors.

Give it a plain interface too. A simple internal page with filters beats teaching everyone SQL, and it means support can answer the question without escalating to engineering.

Then use it yourself once a month. A trail nobody reads rots quietly: fields go stale, and half the runs turn out to be writing null into a column you renamed.

How do you log cost, not just outcomes?

Capture the usage figures the API already returns. Claude API responses report input tokens, output tokens and server tool use, including counts for web search and code execution requests. Multiply those by the published rates and you have cost per run without any estimation.

The rates make the arithmetic easy. Claude Haiku 4.5 is published at 1 dollar per million input tokens and 5 dollars per million output tokens, and Anthropic's own documented example puts roughly 10,000 support tickets at about 37 dollars using that model. Server-side web search is priced at 10 dollars per 1,000 searches.

Store the cost per run, not the monthly total. Per-run cost is what lets you kill the one workflow burning the budget instead of debating the whole line item. Our guide to controlling AI costs on a web team goes further into the tracking.

What does good look like when something goes wrong?

You find the run in one query, read the exact input and output, see which prompt version produced it, and know whether a human approved it. Then you fix the template, bump the version, and rerun the affected rows from the trail itself.

That last part is the real payoff. A proper trail is also a work queue: it tells you exactly which records ran under the broken version, so the cleanup is a filter rather than an archaeology project.

Anthropic's engineering guidance warns that autonomous systems bring higher costs and the potential for compounding errors. A trail is how you catch a compounding error while it is still only expensive. Our notes on handling AI automation failures cover the alerting side.

Where should you start this week?

Add one table and one insert. Before your automation takes its action, write the row with all nine fields in it. That single change turns an opaque system into an accountable one, and it takes less time to build than the meeting about whether it is needed.

Then set a retention job, index the three fields, and build the little internal page. Everything else, the traces, the dashboards and the alerting, is an improvement on a foundation you now have.

If you want help designing the trail before you build the automation, we are happy to walk through it. You can see how we build these systems at phoenix.studio.