At the point where an action becomes hard to undo. Not at every step, and not nowhere. If a mistake costs a minute to fix, let the machine run. If it costs a customer, a contract or a reputation, a person signs off before it goes out.
That sounds obvious written down. In practice most teams pick one of two extremes. Either they review every single output, which kills the time savings and trains people to click approve without reading, or they wire the agent straight into production and find out later what it did.
This is the framework we use when we design automation for clients, why we use it, and what the regulators have already decided about the question.
It means a person has real authority over the system, not just visibility of it. Watching a dashboard is not oversight. Oversight is the ability to stop the thing, override it, and understand enough about it to know when to.
There are three distinct patterns and teams conflate them. Human in the loop means a person approves each action before it happens. Human on the loop means the system acts and a person monitors and can intervene. Human in command means a person sets the boundaries and can shut it all down.
Most workflows should not be in the loop everywhere. A content pipeline might be in the loop at publish, on the loop for drafting, and in command over the whole thing. Naming which one you mean for each step is half the design work.
Because it does not survive contact with volume. A person asked to approve forty outputs a day will read the first five carefully, skim the next ten and rubber stamp the rest. You have not added a control. You have added a person whose name is now attached to whatever the model produced.
It also destroys the economics. If the automation saves eight minutes and the review takes seven, you have built an expensive way to move work sideways. Teams then quietly stop reviewing, which is worse than never having had the step.
The honest design question is not whether to review, but which small subset must be reviewed for the whole thing to be safe. Usually that is a much smaller set than people assume, and it is not the set they would have guessed.
More than most teams realise. The EU AI Act sets out human oversight requirements for high risk AI systems in Article 14. It is worth reading even if you are nowhere near a high risk use case, because it is a well constructed checklist.
Article 14 says overseers must be enabled to properly understand the relevant capacities and limitations of the system and monitor its operation, including in view of detecting and addressing anomalies. It requires that they can correctly interpret the output, and that they can decide, in any particular situation, not to use the system or to otherwise disregard, override or reverse the output.
It also requires the ability to intervene in the operation of the system or interrupt it through a stop button or a similar procedure that allows the system to come to a halt in a safe state. For biometric identification it goes further, requiring that no action is taken unless the identification has been separately verified and confirmed by at least two natural persons.
Automation bias is the tendency to trust a machine's output more than you should, and the AI Act names it directly. Article 14 requires that overseers remain aware of the possible tendency of automatically relying or over relying on the output produced by a high risk AI system.
It is the reason a review step can look fine on a process diagram and fail completely in practice. A fluent, confident, well formatted answer reads as correct. Human reviewers correct fewer errors in machine output than they would in a colleague's draft, because the machine's output does not look uncertain.
The design fix is to make uncertainty visible. Show the source sentence a claim came from, flag low confidence extractions differently from high confidence ones, and put the thing most likely to be wrong at the top. We wrote about this in the content context in our piece on fact checking AI written content.
Anything that leaves your company, changes money, or changes a person's status. Sending an email to a customer, changing a price, closing a ticket, deleting a record, publishing to a live site, or moving someone through a hiring stage. All of those get a person.
The test we use is reversibility crossed with visibility. A mistake that is easy to undo and that only you would see is safe to automate. A mistake that is hard to undo, or that a customer sees before you do, is not. Most bad automation decisions we have unpicked failed that second half.
There is a third category worth naming. Anything where the model would need information it does not have. An agent cannot know that this particular client is already annoyed, or that a deal is politically sensitive. If context that lives only in someone's head changes the right answer, that someone is part of the workflow.
Less good than the demos suggest, and the honest benchmarks show it. The OSWorld benchmark, published by Xie and colleagues, tested multimodal agents on 369 real computer tasks. Humans completed over 72.36 percent of them. The best model in that paper managed 12.24 percent.
Scores on that benchmark have moved since it was published, and we are not going to invent a current number. The shape of the gap is the useful part. Agents are strong at language tasks and much weaker at long chains of operations in a real environment, which is exactly what most business automation is.
That gap is the argument for keeping the person at the action boundary rather than the drafting boundary. Let the model do the reading, the summarising and the drafting, where it is genuinely strong. Keep the clicking that changes state under human control, where it is not.
It shows the diff, not the output. A reviewer approving a CRM update should see which three fields changed and what they changed from, not a wall of text they have to compare from memory. The cognitive cost of review is what decides whether review actually happens.
It should also make rejecting cheap. If the only options are approve or start again from scratch, people approve. Give them edit and approve, and reject with a reason, and you get honest signal about what the system gets wrong.
Batch carefully. Approving twenty items at once is fine when they are genuinely similar and the reviewer can spot the odd one out. It is dangerous when the items are unrelated, because attention resets with every context switch and there is no attention left by item twelve.
When you have counted. Run the step with mandatory review, log every approval, edit and rejection, and after a few hundred items look at the rate. If reviewers changed nothing in ninety nine cases out of a hundred, the review is theatre and you can move to sampling.
Sampling is the middle setting people forget exists. Review one in ten at random, keep logging, and set a threshold that puts full review back on if the error rate rises. That is how you keep a control without paying for it on every item.
The NIST AI Risk Management Framework, released as AI RMF 1.0 on January 26, 2023, organises this thinking around four functions: Govern, Map, Measure and Manage. It is voluntary, and NIST says so plainly, but Measure is the function most teams skip and it is the one that tells you whether to keep the human.
They need a permission boundary, not just a review step. An agent with a write capable API token can do more damage faster than any human on your team, and it will do it without hesitating. Scope the credentials to the narrowest thing that works.
Read only by default is the right starting posture. If an agent needs to write, give it write access to one table, one repository or one CRM object type, not to the account. And log every call it makes somewhere the agent cannot edit.
We go into how we assess this before anything reaches production in our piece on evaluating AI automations. The short version is that the blast radius question comes before the accuracy question.
Map the steps, mark each one reversible or not, and put the person at the first irreversible step. Then instrument everything so that in six weeks you have data rather than opinions about where the review is earning its keep.
Resist the temptation to start with the most valuable workflow. Start with one that matters enough to be worth doing and small enough that being wrong is cheap. The point of the first build is to learn how your team actually behaves around machine output, and that lesson transfers.
If you would like help drawing that map for a workflow you are considering, we are glad to sit down with it. You can reach our team at phoenix.studio and we will tell you honestly which parts we would automate and which we would leave alone.
Tell us where you want to go. We'll tell you how we'd get you there.