Should Your AI Automation Run in Real Time or in Batches?

Should Your AI Automation Run in Real Time or in Batches?

Run it in batches unless a person is waiting. Both OpenAI and Anthropic charge half price for asynchronous batch processing, and most marketing and operations workloads have no real time requirement at all. Real time is a product decision you should have to justify, not the default.

We find teams default to synchronous calls because that is how the first tutorial worked. Then the monthly bill arrives and the conversation turns to cheaper models, when the cheaper option was sitting in the same API the whole time.

Here is the actual trade, with the numbers both providers publish.

What Does Batch Processing Actually Save?

Half. Anthropic's Message Batches API charges all usage at 50% of standard API prices, and its documentation notes this applies to input tokens, output tokens and special tokens. OpenAI's Batch API offers a 50% cost discount compared to synchronous APIs.

That is not a marginal optimisation. It is the single largest lever most teams have on their AI spend, and it requires no change to the prompt, no change to the model, and no loss of quality. The same model produces the same output.

What you give up is immediacy, and that is the whole question.

How Long Does a Batch Actually Take?

Less time than the word "batch" suggests. Anthropic says most batches finish in less than 1 hour, while results become available when all messages have completed or after 24 hours, whichever comes first. OpenAI says batches complete within 24 hours and often more quickly.

So the honest framing is not "instant versus tomorrow". It is usually "instant versus about an hour, with a 24 hour worst case". For a nightly content pipeline, a weekly report, or an enrichment job, that difference is invisible to everyone involved.

The 24 hour figure still matters, because it is the number you must design around. Both providers expire batches that do not complete in time.

What Happens If a Batch Does Not Finish?

The two providers handle it slightly differently, and the difference affects your budgeting.

Behaviour	Anthropic Message Batches	OpenAI Batch API
Discount	50% of standard prices	50% versus synchronous
Typical completion	Most under 1 hour	Within 24 hours, often sooner
Expiry window	24 hours	24 hours
Size limit	100,000 requests or 256 MB, whichever first	50,000 requests, input file up to 200 MB
Billing on expiry	Expired requests are not billed	Charged for tokens from completed requests

Anthropic's documentation also notes that processing may slow down based on current demand and your request volume, in which case more requests may expire after 24 hours. That is a real operational risk on very large jobs, and it is a reason to split work rather than submit one enormous batch.

Can You Stack Batching With Prompt Caching?

Yes, and this is the detail most teams miss. Anthropic's documentation states that the pricing discounts from prompt caching and Message Batches can stack, providing greater savings when both are used together.

The catch is that batch requests are processed asynchronously and concurrently, so cache hits are provided on a best effort basis. Anthropic says users typically experience cache hit rates ranging from 30% to 98% depending on traffic patterns, which is a wide range and worth measuring rather than assuming.

Its guidance for batches with shared context is to consider the 1 hour cache duration, because batches can take longer than five minutes and a shorter cache would expire mid job. If you are already thinking about spend, this compounds with everything in our piece on controlling AI costs.

Which Workloads Are Obviously Batch?

Anything on a schedule and anything triggered by a volume of data rather than a person. Classifying a month of support tickets. Generating meta descriptions for 4,000 CMS entries. Enriching a lead list. Running evaluations over a test set. Summarising yesterday's calls.

Content pipelines are the clearest case. Nobody is refreshing a page waiting for the draft. A nightly job that costs half as much and finishes before anyone logs on is strictly better than a synchronous one, which is how we think about automated publishing pipelines.

The test is simple. If the job could run at 3am and nobody would notice, it should.

Which Workloads Genuinely Need Real Time?

Anything with a human in the loop at that moment. A chat interface. An in product assistant. A form that qualifies a lead while the person is still on the page. An agent taking a step in a workflow where the next step depends on this answer.

There is also a category people forget: anything where latency is the feature. If your product's promise is instant, batching breaks the promise regardless of cost.

Beyond that, be suspicious. "The sales team wants to see it immediately" usually means within ten minutes, which a small frequent batch handles fine.

Is There a Middle Option?

Yes, and it is the one we reach for most: micro batching. Instead of one call per event or one giant nightly job, collect events for a few minutes and submit them together.

This gives you most of the throughput benefit, keeps latency inside what a human tolerates, and dramatically reduces the number of API calls your system has to retry, log and reconcile. It also smooths your rate limit usage, which is a real constraint that the pricing conversation tends to ignore.

The design question becomes how you collect events, which is the same plumbing decision we discuss in webhooks versus polling.

What Does Batching Cost You in Complexity?

It is genuinely more work, and pretending otherwise is how these projects go wrong. A synchronous call returns an answer. A batch returns a job ID, and now you need somewhere to store it, a way to poll or be notified, handling for partial results, and a plan for expired requests.

You also lose the simple error story. In a batch, individual requests can succeed, error, be cancelled, or expire independently, so your code has to handle four outcomes per item rather than two.

Our rule is that this complexity is worth it above a threshold of volume and below a threshold of urgency. A job running twice a day over 50 items is probably not worth the machinery. A job running nightly over 5,000 items certainly is.

How Should You Decide?

Ask three questions in order. Is a person waiting for this specific answer right now. If yes, real time, and stop. If no, does the work arrive in volume or on a schedule. If yes, batch it. If it arrives as a trickle of events with mild urgency, micro batch it.

Then check the size limits against your job. Anthropic caps a batch at 100,000 requests or 256 MB, whichever is reached first. OpenAI caps a batch at 50,000 requests with an input file up to 200 MB. Design your chunking around the lower of those, not the higher.

Finally, decide what happens to expired work before you ship, not after your first failed run.

What Would We Change First?

We would look at the single highest volume AI job in the stack and ask whether anybody actually waits for it. In most marketing and operations setups the answer is no, and moving that one job to a batch endpoint halves its cost in an afternoon.

Then we would add prompt caching to it, with a cache duration matched to how long the batch takes, and measure the hit rate rather than assume it. Those two changes together are usually the difference between an AI budget that worries people and one that does not.

If you have AI automations running and no clear picture of which ones need to be immediate, we are happy to go through them with you and sort them into the right buckets. You can find us at phoenix.studio.