How Do You Prove an AI Automation Actually Saved Time?

How Do You Prove an AI Automation Actually Saved Time?

By measuring the same task before and after, with a number you wrote down in advance. Not by asking the team how it felt. The research on this is uncomfortable: people are confidently wrong about their own productivity when AI is involved, in both directions.

Most automation programmes we see have no baseline. Somebody built a thing, everybody agreed it was useful, and nine months later nobody can say whether it earns its keep or who would notice if it stopped.

This is the framework we use to answer that question honestly, including the parts that make the automation look worse.

Why Can't You Just Ask the Team If It Helped?

Because self-reported productivity and measured productivity can point in opposite directions. METR, an AI evaluations research organisation, ran a randomised controlled trial with 16 experienced developers working on 246 real tasks in their own large open-source repositories, averaging over 22,000 stars and more than a million lines of code.

The result: "when developers are allowed to use AI tools, they take 19% longer to complete issues." Before the study those developers expected AI to speed them up by 24%. Afterwards, having just been measurably slowed down, "they still believed AI had sped them up by 20%."

METR is careful about scope, and so should we be. The study used Cursor Pro with Claude 3.5 and 3.7 Sonnet, frontier models at the time, and METR explicitly does not claim that its developers or repositories represent most software work, or that AI fails to speed up other domains. It calls the result a "snapshot of early-2025 AI capabilities in one relevant setting."

The transferable lesson is not "AI makes people slower." It is that the gap between felt speed and actual speed can be 39 percentage points wide. That is why a survey is not a measurement.

What Should You Measure Instead of Time Saved?

Pick a number that already exists in a system, not one someone has to estimate. Time saved is the hardest possible metric because nobody logs the counterfactual. Better candidates are throughput, cycle time, error rate, and queue depth.

For a content automation, that might be published pages per week and the number that needed a correction after publishing. For a lead qualification automation, the number of leads routed and the share a human reclassified. For a QA automation, defects caught before release versus after.

Each of those has a property that matters: someone else's system records it, so your automation cannot grade its own homework. If the only evidence that the automation works comes from the automation's own log, you have no evidence.

How Do You Get a Baseline If You Never Took One?

Reconstruct it from history, then be honest that it is reconstructed. Most teams have more usable history than they think: ticket timestamps, git commits, CMS publish dates, CRM record creation times, support queue exports.

Pull four to eight weeks from before the automation existed and compute the same metric you plan to track now. If seasonality matters, use the same weeks a year earlier as a sanity check. Write the baseline down in a dated note with the query you used, so the next person can reproduce it.

If there is genuinely no history, the honest move is to turn the automation off for two weeks and measure the manual process properly. That feels like a step backwards. It is much cheaper than running an unmeasured automation for another year.

What Does the Real Cost Column Include?

More than the model bill. Four costs get left out almost every time, and together they often exceed the inference spend.

First, the build. Engineer hours are the largest single cost of most automations and they are almost never counted, because they are already paid for. Second, the review. If a human checks every output, you have not removed the work, you have changed it, and reviewing someone else's draft is not free. Third, the maintenance. Prompts drift, APIs change, models get deprecated. Fourth, the failure cost: what a wrong output costs to find and fix.

We wrote separately about keeping the inference side under control in controlling AI costs on a web team, but the bill is usually the smallest of these four. The maintenance line is the one that kills automations quietly.

How Do You Account for Work the Automation Created?

Count it as a cost, in the same units as the saving. This is the step that turns a flattering number into a true one.

A summarisation automation that drafts 40 items a week and requires a human to read all 40 has not saved 40 units of work. It may have saved the writing and added the reading. Whether that is a win depends on which is slower, and you can measure that in an afternoon by timing both.

The DORA 2025 report on AI-assisted software development found a version of this at the organisation level. Google Cloud's announcement of the report states: "Unlike last year, we observe a positive relationship between AI adoption on both software delivery throughput and product performance. However, AI adoption does continue to have a negative relationship with software delivery stability." Faster and less stable is a trade, not a free win, and you only see the trade if you are measuring both sides.

What Does a Good Measurement Plan Look Like?

Five lines, written before the automation ships. We keep it deliberately short so it actually gets written.

Line one: the one metric, named, with the system it comes from. Line two: the baseline value, with the date range and the query. Line three: the threshold that would count as success, decided in advance. Line four: the review date, an actual calendar date. Line five: the name of the person who decides, and what they are allowed to decide, including switching it off.

That last line is the one people resist and the one that matters. An automation with no named owner and no shutdown authority does not get retired when it stops being useful. It just accumulates. This is the same discipline we argued for when choosing what to build in the first place, in deciding what to automate first.

How Long Should You Run It Before Deciding?

Long enough to cover a full cycle of the work, and no longer than a quarter. For weekly work, four to six weeks. For monthly reporting, three cycles. For anything seasonal, be honest that you are comparing against a moving target and widen the threshold.

Resist the urge to extend the trial because the numbers are ambiguous. Ambiguous after a fair trial is a result. It usually means the automation is neither helping nor hurting much, which makes it a maintenance liability with no upside.

The one good reason to extend is that you changed something material mid-trial. In that case, restart the clock and say so.

When Should You Switch It Off?

When it misses its threshold, when nobody can name its owner, when the underlying task has changed, or when the review burden has quietly grown to match the work it replaced. Any one of those is sufficient.

Switching off is not a failure of the programme, it is the programme working. A portfolio where nothing is ever retired is not a portfolio, it is a pile. We went into the mechanics of doing this safely in retiring automations nobody uses.

In our experience the hardest cases are the automations with one enthusiastic user. They are cheap to run and expensive to maintain, and the honest conversation is whether that one user's benefit justifies the standing cost. Sometimes it does.

What Is the Honest Way to Report This Upward?

Three numbers and one sentence. The baseline, the current value, and the total cost including build, review and maintenance. Then one sentence on what you would do differently. That is more credible than a percentage improvement with no denominator, and it survives questions.

Avoid the two common framings. Do not report time saved unless you actually measured the before case, because that number is almost always a guess dressed up as data. And do not report volume as value: publishing four times as many pages is not an outcome, it is an input.

The wider point is the one DORA reached. Its own conclusion is that "AI doesn't fix a team; it amplifies what's already there. Strong teams use AI to become even better and more efficient. Struggling teams will find that AI only highlights and intensifies their existing problems." Measurement is how you find out which kind of team you are, and it is cheaper to find out on purpose than by accident.

If you have a set of automations running and no honest read on whether they are working, that is a good problem to bring to us. We are at phoenix.studio.