When Should an AI Automation Stop and Ask a Human?

When should an AI automation stop and ask a human?

When the action is hard to reverse, when it is visible to a customer, or when the model is guessing rather than looking something up. Those three tests catch most of it. Everything else can run unattended, and should, because a checkpoint you do not need is a checkpoint nobody reads.

We build automations around client websites, so this question arrives in a very concrete form. Should the thing that drafts the weekly update post it, or queue it? Should the lead router email the prospect, or tell a human to? Teams tend to answer by instinct, and instinct pulls in two bad directions at once. Either everything gets an approval step, which quietly turns into a rubber stamp, or nothing does, and the first bad output is also the first anyone hears about it.

There is a better way to decide, and it is not complicated.

What does human in the loop actually mean?

It means a person has to act before the automation continues. That is the whole definition, and it is narrower than how the phrase gets used. A human who can see the output afterwards is not in the loop. A human who gets a Slack notification they can ignore is not in the loop either. If the automation proceeds when the person does nothing, there is no loop.

This matters because most teams we talk to believe they have human oversight when what they have is a log. A log is useful. It is not oversight. Oversight is the ability to stop something before it happens.

The useful distinction is between a gate and a receipt. A gate blocks. A receipt informs. Both have a place, and confusing them is how teams end up surprised.

Which decisions always need a person?

Three categories, and we would hold these regardless of how good the model gets. Anything that cannot be undone. Anything a customer or the public will see with your name on it. Anything that moves money or changes permissions.

Deleting records, sending an email to a real list, publishing to a live site, issuing a refund, granting access to a system, signing anything. These share a property: the cost of being wrong is paid by someone outside your team, and you cannot take it back by editing a row.

Notice that the split is almost never about how clever the task is. It is about who pays if it goes wrong.

What does the EU AI Act require here?

For high risk systems, quite a lot, and the list is worth reading even if you are nowhere near that category. Article 14 of the EU AI Act sets out what human oversight has to make possible. The text requires that a person can properly understand the capacities and limitations of the system and duly monitor its operation. It requires that they remain aware of the possible tendency of automatically relying or over-relying on the output, which the Act names as automation bias.

It also requires that a person can correctly interpret the output, can decide in any particular situation not to use the system or to otherwise disregard, override or reverse the output, and can intervene in the operation or interrupt the system through a stop button or a similar procedure that allows the system to come to a halt in a safe state.

Most marketing automations are not high risk systems and this article does not bind them. We still treat those five capabilities as the design brief, because they describe what oversight has to be able to do to mean anything. If your team cannot stop a running automation in a safe state, you do not have a stop button, you have a hope.

How do you decide where the checkpoint goes?

Put it at the last reversible moment. Not at the start, where the person has no output to judge, and not after the effect, where judging is pointless. The right place is immediately before the step that reaches the outside world.

A concrete shape. An automation reads a form submission, enriches it, writes a draft reply, picks a template, and sends. Four of those five steps are internal and reversible. The checkpoint belongs between step four and step five, with the full draft visible, and nowhere else. Putting a gate at step two, where a person approves the enrichment, produces a person clicking approve on data they have no way to evaluate.

This is also the cheapest place to put it, because by then the automation has done all the work. The person is doing the one thing people are better at, which is noticing that something is off.

Why do approval steps stop working over time?

Because they are almost always right, and being almost always right is how you teach a person to stop reading. The EU AI Act calls this automation bias, and the NIST AI Risk Management Framework treats the same problem through its Govern and Measure functions. NIST released AI RMF 1.0 on January 26, 2023, and added the Generative AI Profile, publication NIST.AI.600-1, on July 26, 2024.

In our experience the decay is fast. A checkpoint that fires forty times a week and has never once needed a rejection is not an approval step any more. It is a button. The team clicks it while reading something else.

The fix is not more training. It is fewer checkpoints on higher stakes decisions. If you cut the volume so that a rejection is a realistic outcome of any given review, attention comes back. We wrote about the related failure patterns in how to handle failure in AI automations, and the same principle holds: a safeguard that never triggers is not being tested.

How should the approval itself be designed?

Show the person what they need to judge, and nothing else. The most common design mistake we see is a Slack message that says an automation is ready to send and links to a dashboard. That is two clicks and a context switch before any judgement happens, and it guarantees the reviewer skims.

Put the full draft in the message. Put the recipient in the message. Put one thing that would make a reviewer suspicious in the message, such as the confidence the system had, or the source it used, or the fact that a field was blank and got filled by a guess.

Then give two buttons, approve and reject, and make reject cheap. If rejecting means opening a tool, writing an explanation, and reassigning a task, people will approve marginal cases to avoid the friction. Rejection should be one click and a free text box that is optional.

What should you log at every checkpoint?

The input the model saw, the output it produced, who decided, what they decided, and when. That is the minimum. Without the input you cannot reproduce a bad decision. Without the decider you cannot answer the only question that matters after an incident, which is who knew.

Log the rejections especially. Rejections are the highest value data any automation produces, because each one is a labelled example of your system being wrong in a way a human caught. Most teams throw them away. If you keep them, you get a test set for free, which is the input to evaluating an AI automation in production.

Keep the logs somewhere a non-engineer can read. Airtable and Google Sheets are both fine for this, and both are better than a log nobody opens.

Where should you start with all this?

Take your existing automations and sort them into the three categories from earlier: irreversible, externally visible, or money and permissions. Anything in those categories gets a gate at the last reversible moment. Everything else gets a receipt and no gate at all.

Then count your gates. If a single person is approving more than a handful of things a day, you have not built oversight, you have built a queue, and it will decay into a rubber stamp within a month. Cut until the number is small enough that each review is a real decision.

The teams that get this right are not the ones with the most safeguards. They are the ones who can say exactly which three things a person still decides, and why. Everything else runs, and runs unattended, and that is the point of building it.

If you are putting automations around a site we built or one you already have, and you want a second pair of eyes on where the gates belong, we are happy to walk through it. You can find us at phoenix.studio.