Who Is Responsible When an AI Automation Gets It Wrong?

Who is responsible when an AI automation gets it wrong?

You are. Not the model provider, not the vendor whose tool you wired together, and not the automation itself. If your company sent the email, changed the record or made the decision, your company owns the outcome, and every regulatory framework we have read this year says the same thing in more careful language.

This sounds obvious written down. It is not obvious in practice, because the way most automations get built quietly distributes responsibility until nobody holds it. A marketer configures a tool, a developer connects an API, a vendor supplies the model, and the thing that goes out at three in the morning has no author at all.

We think that is the real risk in AI automation, and it is an organisational problem rather than a technical one.

Why does responsibility go missing?

Because the work is assembled rather than written. Nobody sat down and authored the behaviour, so nobody feels like they decided it. A person who writes a script knows they wrote it. A person who connects two tools and types a prompt into a box often does not feel they have shipped anything.

The prompt is the part that makes this worse. A prompt reads like an instruction to a colleague, which is a very different psychological act from writing code, and colleagues are assumed to exercise judgement. They are assumed to not send the obviously wrong email.

So the automation launches with an implicit assumption that something in the loop has common sense. Nothing in the loop has common sense. What the loop has is a person who could have checked, and usually did not know they were the person.

What does the law actually say about this?

For high risk systems, the EU AI Act is unambiguous about where the duty sits, and the structure is worth borrowing even where it does not bind you. Article 26 puts obligations on deployers, which is the company using a system, as distinct from the provider that built it.

The core sentence is the one about oversight. Deployers shall assign human oversight to natural persons who have the necessary competence, training and authority, as well as the necessary support. Read every clause in that: not just a person, but a person with competence, training, authority and support.

That last pair is where most real arrangements fail. Plenty of companies have someone watching an automation. Very few have given that person the authority to turn it off without asking permission, which means they are a witness rather than an overseer.

What else does Article 26 require?

A set of duties that read like a decent operating manual even outside the regulated cases. Deployers must monitor the system's operation according to the provider's instructions, and report risks to the provider, distributors, importers and market surveillance authorities. They must suspend use if risks emerge, and immediately notify about serious incidents.

On records, the Act requires deployers to keep the logs automatically generated by the system, to the extent those logs are under their control, for a period appropriate to the intended purpose, of at least six months, unless other Union or national law provides otherwise.

There are also transparency duties that companies rarely anticipate. Before deploying a high risk system at a workplace, employer-deployers must inform workers' representatives and affected workers. And deployers making decisions about individuals must inform those people that they are subject to the system's use.

Does any of this apply if you are not in the EU?

As law, often not. As a design specification, always. The obligations describe what it takes to be able to answer for something, and that requirement does not depend on jurisdiction.

Consider what happens without them. A customer complains about an automated decision. You cannot say who oversaw it, you did not keep the logs, the person watching had no authority to stop it, and nobody told the affected person it was automated. The legal exposure varies by country. The reputational exposure does not.

We would treat Article 26 the way we treat accessibility standards for clients outside the regulated set: as the clearest available description of doing the thing properly, written by people who thought about it longer than we have.

Is the vendor ever responsible?

For their product, yes. For your use of it, no, and this is where a lot of internal conversations go wrong after an incident.

The model provider is responsible for the model behaving as documented. The automation platform is responsible for executing what you configured. Neither is responsible for the fact that you pointed it at your customer list, and no contract you are likely to have signed says otherwise.

This distinction matters most in the meeting after something breaks, because the instinct is to look for the component that failed. Usually no component failed. The model produced plausible output, the platform delivered it faithfully, and the system did exactly what it was configured to do, which turned out to be wrong. That is a design failure with a name on it.

How do you assign responsibility in practice?

Name one person per automation, in writing, before it runs. Not a team, not a function, a person. The test for whether you have done this properly is whether that person can be woken up about it, and whether they can switch it off alone.

Write down three things next to the name. What this automation is allowed to do. What it must never do. And what happens, specifically, when it does the second thing. That last item is the one everyone skips and the one that turns an incident from a panic into a procedure.

Then review the list when people leave. An automation whose owner left the company six months ago is running unowned, which is the default state of most automations older than a year. We covered how to keep this legible in documenting AI automations so someone else can run them.

What about the logs?

Keep them, and keep them longer than you think. The EU AI Act's six month floor for high risk systems is a useful reference point, and our observation is that incident discovery routinely takes longer than teams' retention settings allow for.

The practical failure is not that logs are absent. It is that they are partial. You have the model call but not the tool result, or the tool result but not the decision, or all three but no way to connect them to the record that changed. An investigation needs the chain, not the pieces.

The NIST AI Risk Management Framework, released as AI RMF 1.0 on January 26, 2023, organises this thinking under four functions, Govern, Map, Measure and Manage, and NIST added a Generative AI Profile, publication NIST.AI.600-1, on July 26, 2024. We find the Govern framing useful precisely because it puts accountability upstream of measurement, where it belongs.

Does this mean you should automate less?

No, and we want to be clear about that, because the accountable framing is sometimes used as an argument for doing nothing. Automation is good. Most of the work we see automated should be automated, and the teams doing it are better off than the teams still copying data between tabs.

What it means is that the ownership question should be answered at the same time as the build question, rather than six months later under pressure. The cost of answering it upfront is one paragraph in a document. The cost of answering it after an incident is a week of people reconstructing what happened from memory.

There is also a quieter benefit. Automations with a named owner get retired when they stop being useful. Unowned ones run forever, quietly producing output nobody reads, which is its own kind of risk.

What would we actually do this month?

List every automation currently running that can touch a customer, a record or a payment. For most teams that list is longer than expected, because automations accumulate faster than anyone tracks them.

Then put a name against each one, confirm that person knows and can stop it, and check that the logs would let you reconstruct a bad day. Anything that fails those three checks either gets an owner this week or gets switched off until it has one. That sounds severe and it is the least severe version of the alternative.

The underlying argument is simple. Accountability is not a document you produce after something goes wrong. It is a property a system either has before it runs or does not have at all, and adding it retroactively is not possible. You can only find out whether it was there. The adjacent risks are worth understanding at the same time, and we wrote about the nastiest of them in prompt injection in AI automations and about measurement in evaluating an AI automation in production.

If you want a second opinion on the automations running around your site, we are happy to walk through it. You can find us at phoenix.studio.