Because a demo is a handful of easy cases chosen by the person building it. Production is a long tail of strange ones. The automation that summarised ten support tickets perfectly meets its first ticket written in two languages, or an empty form, and there is no plan for what happens next.
The gap is not really about model quality. It is about the absence of a test. Nobody would ship a payment flow on the strength of one successful transaction, and yet AI workflows go live on exactly that evidence every week.
So this is the process we use before an AI automation is allowed near a live system. None of it is exotic. It is mostly the discipline that ordinary software already has, applied to a component that gives different answers each time.
It means deciding in advance what a good output looks like, collecting real examples, and measuring how often the automation produces one. That is the whole idea. The difficulty is that AI outputs are rarely right or wrong in a binary way, so you have to define acceptable before you can count it.
This is different from testing a normal function. A normal function is deterministic, so one passing test proves a lot. A model can give you a slightly different answer to the same input tomorrow, which means you need many examples and a tolerance for variation rather than a single assertion.
The useful reframe is that you are not testing the model. You are testing your workflow, including the prompt, the data you feed it, the parsing of its output, and what your system does when it produces nonsense. Most failures we see live in those surrounding parts, not the model.
There is a widely used voluntary one. The National Institute of Standards and Technology published its AI Risk Management Framework on January 26, 2023, designed to better manage risks to individuals, organizations and society associated with artificial intelligence. It is intended for voluntary use rather than compliance.
Its structure is worth borrowing even for small projects. The framework has four core functions: Govern, Map, Measure and Manage. Govern is who is accountable. Map is understanding the context and what could go wrong. Measure is the evaluation itself. Manage is what you do about the risks you found.
Most teams jump straight to Measure and skip Map, which is why they end up measuring the wrong thing. Spending twenty minutes writing down what failure would actually look like for this automation, and who would be harmed by it, changes what you decide to test.
From your own history, not from invented examples. Pull fifty to a hundred real inputs the automation would have handled, and make sure they are representative rather than tidy. If a fifth of your real support tickets are one line long and badly punctuated, a fifth of your set should be too.
Then write down the correct output for each one. This is tedious and it is the step that makes everything else possible. Without agreed answers you have no way to tell an improvement from a change. Teams that skip this end up arguing about vibes for months.
Deliberately include the cases you expect to break it. Empty inputs. Inputs in another language. Something abusive. Something that looks like an instruction to the model rather than data, because prompt injection through user submitted content is a real failure mode, not a theoretical one. Our piece on AI assisted website QA covers the same idea from the testing side.
Cost, latency, and the shape of the failures. An automation that is 95% accurate but takes forty seconds is unusable in a form submission flow. One that is accurate and fast but costs more per run than the human it replaced has not saved anything.
Failure shape matters more than the headline number. Ask whether the wrong answers are obviously wrong or plausibly wrong. Obviously wrong output gets caught by a human in seconds. Plausibly wrong output gets accepted and propagates, which is far more damaging even at a lower rate.
Track spend from the first day, because it is the thing that surprises teams a month in. Our notes on controlling AI costs cover how quickly a per run cost turns into a real line item once volume arrives.
You let it do the work without acting on the result. The automation runs on live inputs, writes its output to a log or a review queue, and changes nothing. A person continues doing the job as normal, and you compare the two.
Two or three weeks of that is worth more than any benchmark, because it is measured on your actual traffic. It also surfaces the operational questions that evaluation sets never do, like what happens when the volume triples on a Monday or when an upstream system returns an empty response.
The discipline is to agree the pass condition before you start. Write down, in advance, what result would make you comfortable turning it on. Otherwise the decision gets made by whoever is most tired of waiting.
Narrow permissions first. If the automation connects to your systems through the Model Context Protocol, which its documentation describes as an open source standard for connecting AI applications to external systems, the protocol is not what makes it safe. The scope you grant it is. Give it read access to what it needs and write access only to the fields it owns.
Then an audit trail. Every action the automation takes should be recorded with the input that caused it. Platforms are starting to build this in. Webflow said at its Conf 2026 event on September 2, 2026 that it had updated its site activity log earlier in the summer to improve visibility and tracking for every AI change across a site, and that agents contributing to a release are subject to the same roles and permissions as teammates.
That second idea is the one to copy. An agent should not have powers a junior team member would not be given. If you would not let a new starter publish to production unreviewed on their first week, do not let the automation do it either.
By never fully turning off the comparison. Keep a sample of outputs going to human review on an ongoing basis, even at a low rate. A few checked items a day is enough to notice a change in quality before your customers do.
Re-run your evaluation set on a schedule too. Models get updated, your inputs change as your business changes, and a prompt that was tuned for last year's traffic slowly stops fitting. Running the same hundred cases every month turns drift into a number rather than a feeling.
Set an alarm on the boring metrics as well. A sudden change in average output length, cost per run, or the rate of empty responses usually shows up before accuracy does, and it is much cheaper to monitor.
When the cost of a quiet failure is higher than the cost of the manual work. Anything that rejects a customer, sends an external message, changes money, or makes a legal claim belongs on the do not automate list until you have a long record of good behaviour, and often after that too.
Also skip it when the volume is genuinely low. Building, evaluating and monitoring an automation is not free. If a person handles the task in an hour a week, the automation has to be nearly perfect to be worth the maintenance, and it almost never is at the start.
The honest test we apply is whether we would be comfortable explaining the failure to the affected customer. If the answer is no, we keep a person in the loop and automate the preparation instead of the decision. Our piece on fact checking AI content makes the same argument for anything published under your name.
Take whichever AI automation you are closest to shipping and write down fifty real examples with their correct answers before you touch anything else. If you cannot produce fifty real examples, that is a useful signal in itself about whether the volume justifies the build.
Then run it in shadow mode with a pass condition agreed in advance. It is slower than switching it on, and it is the difference between an automation you trust and one you quietly stop using. If you want help designing the evaluation or building the guardrails around a workflow, we are happy to go through it with you at phoenix.studio.
Tell us where you want to go. We'll tell you how we'd get you there.