How Do You Tell a Real AI Agent From a Scraper?

How Do You Tell a Real AI Agent From a Scraper?

Today you verify the published IP ranges the major providers document, and you start watching for cryptographically signed requests. A user agent string proves nothing. The industry is moving to signed requests through an IETF effort called Web Bot Auth, and that work reached a working group draft on 1 September 2026.

The practical problem is simple to state. Something calls itself ChatGPT-User and asks for your pricing page fifty times a minute. Is that a buyer's assistant doing a job, or somebody scraping you behind a borrowed name?

Until recently the honest answer was that you could not tell reliably. That is finally changing, and it is worth understanding the mechanism rather than waiting for a vendor to handle it.

Why Is the User Agent String Useless on Its Own?

Because anyone can send any string they like. The Web Bot Auth architecture draft names this directly, noting that user agent strings are easily spoofed and do not cryptographically bind a request to the provider that claims to have sent it.

Cloudflare made the same argument when it published its proposal in May 2025, calling user agent headers easily spoofable and therefore insufficient for reliable identification. It is not a subtle failure mode. Setting a header is one line of code in every HTTP library ever written.

The practical consequence is that every rule you have written against a user agent is a rule that stops honest clients and waves through dishonest ones. Your robots.txt still matters for the cooperative crawlers, and we covered that in controlling AI crawlers with robots.txt, but it is a request, not a control.

What Can You Actually Verify Today?

Published IP ranges, for the providers that publish them. OpenAI documents four separate agents and a separate address list for each one. OAI-SearchBot surfaces sites in ChatGPT's search features, OAI-AdsBot validates the safety of pages submitted as ads, GPTBot crawls content that may be used for training foundation models, and ChatGPT-User handles certain user actions when a person asks ChatGPT to visit a page.

Each of those has its own JSON file of address ranges published at openai.com, so a reverse check is mechanical. Fetch the file, cache it, compare the requesting address, and treat a mismatch as unverified rather than as an attack.

One detail in OpenAI's own documentation is easy to miss and matters for how you treat the traffic. ChatGPT-User is not used for crawling the web automatically, and robots.txt rules may not apply to those user-initiated actions. That is a person asking for your page through a tool, not a crawler harvesting it.

What Is Web Bot Auth, and How Far Along Is It?

It is the effort to replace address lists with cryptography, and it is now an IETF working group item. The protocol document, titled HTTP Message Signatures for automated traffic, became draft-ietf-webbotauth-httpsig-protocol-00 on 1 September 2026, authored by Thibault Meunier of Cloudflare and Sandor Major of Google, in a working group named Web Bot Auth.

Be precise about status, because a lot of writing about this is not. It is an active Internet-Draft. The datatracker entry shows no responsible area director assigned and no intended RFC status set. The earlier architecture draft that framed the problem, revision 05 from 2 March 2026, has already expired and is archived.

So this is a real, funded, multi-vendor standards effort that has passed the point of being one company's blog post, and it is not a finished standard. Both halves of that sentence are true.

How Does a Signed Request Prove Anything?

The agent signs the request with a private key, and you check it against a public key the provider publishes. Cloudflare's implementation uses Ed25519 signing and three headers. Signature-Input carries the validity window, the key identifier as a JSON Web Key thumbprint, and a tag of web-bot-auth. Signature-Agent points at where the public keys live. Signature carries the signature itself.

Crucially, the signature covers the authority of the target URI, meaning the domain being requested. A signature captured from a request to someone else's site cannot be replayed against yours.

The architecture draft describes three ways an origin can find the right key: an out-of-band exchange directly with the provider, a public list along the lines of the Public Suffix List, or the Signature-Agent header pointing at the provider's key directory. The directory format is JWK, with key identifiers derived from thumbprints.

The reason this is better than address lists is the reason the draft gives for abandoning them. Addresses are shared across services and companies on shared infrastructure, they change as infrastructure changes, and buying dedicated ranges to make allowlisting work is expensive. Shared secrets do not scale across the open web either, and rotating or auditing them is painful.

Which Bots Should You Treat Differently?

Sort them by what the visit is for, not by who sent it. We use three buckets. Training crawlers, which read at volume to build models. Retrieval crawlers, which read so an engine can cite you. And user-initiated fetches, where a real person asked their assistant to look at your page right now.

Those deserve genuinely different treatment. A training crawler is a licensing and policy question, which is where the pay per crawl conversation lives. A retrieval crawler is a visibility question, and blocking it is choosing to be absent from answers. A user-initiated fetch is a visitor, and rate limiting it like a scraper means rate limiting a buyer.

Most of the blunt blocking we are asked to review collapses all three into one rule, usually written during an incident. It almost always turns out to be costing the site citations it wanted.

What Should You Do When You Cannot Verify a Request?

Degrade, do not slam the door. Unverified is not the same as malicious, and the cost of getting it wrong is asymmetric. Blocking a scraper saves you some bandwidth. Blocking a buyer's assistant costs you a deal you will never hear about.

A reasonable ladder looks like this. Verified and welcome gets normal service. Unverified but well behaved gets served with tighter rate limits. Unverified and aggressive gets a challenge or a slow lane. Actively abusive gets blocked, logged, and reviewed by a human.

Our notes on rate limiting go deeper into shaping that ladder without breaking normal traffic.

How Do You Roll This Out Without Blocking Real Buyers?

Run it in observation mode first, for at least a couple of weeks. Log what each rule would have done, without doing it. Then read the list of what you would have blocked, by hand, and see how much of it you recognise.

We have yet to see a first draft of these rules that did not catch something legitimate. Uptime monitors, link checkers, a partner's integration, a preview fetcher from a chat tool your own sales team uses. The observation period is what finds those, and it costs nothing.

Only after that should any rule start enforcing, and it should enforce the softest action that solves the problem. Escalate later if the traffic escalates.

What Belongs in Your Logs?

Enough to answer the question later. For every request you classify, keep the raw user agent, the requesting address, the verification result and which method produced it, the path, and the action you took. Store the verification method explicitly, because when the signature-based checks arrive you will want to know what fraction of your traffic they cover.

That last number is the one to watch over the next year. If signed requests go from nothing to a meaningful share of your AI traffic, the address lists become a fallback rather than the primary mechanism, and you can tighten the unverified lane with much less risk.

Without that logging you will be guessing, and the guess will be made under pressure during an incident.

Where Is This Heading?

Towards a web where automated clients carry provable identity and sites decide what to do with it. The signal we take most seriously is not any single announcement, it is that Cloudflare and Google engineers are authoring the draft together in a dedicated working group rather than shipping competing proprietary schemes.

What is still unresolved is everything above the protocol. A signature proves which provider sent a request. It does not tell you whose behalf it acts on, what it intends to do with the page, or whether the person behind it is a customer. Those are policy questions, and the standard is deliberately not answering them.

If you are trying to work out which AI traffic to welcome and which to throttle, and you want rules that do not quietly block your buyers, we are happy to walk through it. You can find us at phoenix.studio.