Should You Charge AI Crawlers to Read Your Website?

Should You Charge AI Crawlers to Read Your Website?

As of September 2026, you can. Cloudflare lets site owners charge AI bots per request instead of only allowing or blocking them. For most B2B companies, we think the answer is still no. You want the citations. But the default settings changed on 15 September 2026, so you need to check what your site now does.

This is one of those changes that happens quietly at the infrastructure layer and shows up later as a traffic problem. Nobody sends you an email when a crawler stops reaching your pages. You find out months later when your brand stops appearing in AI answers and you cannot work out why.

So here is the plain version of what Cloudflare shipped, what it means for a marketing site, and what we would actually do about it.

What Is Cloudflare's Pay Per Crawl?

Pay Per Crawl is a Cloudflare feature that lets a publisher charge AI crawlers money for each request. Cloudflare built it on HTTP response code 402, the Payment Required status that sat unused in the spec for decades. Publishers set one flat per-request price across the whole site. Every crawler is then allowed, charged, or blocked.

The technical piece that makes it work is identity. A crawler has to prove who it is before it can be billed, so Cloudflare requires Web Bot Auth using Ed25519 key pairs and HTTP Message Signatures. Without that, anyone could claim to be GPTBot and either dodge a charge or run up someone else's bill.

When it launched, Pay Per Crawl gave publishers three choices for any given bot: Allow, Charge, or Block. That third option already existed. The second one was new, and it was the first time a major network operator treated crawling as a transaction rather than a permission.

What Actually Changed on 15 September 2026?

Cloudflare announced on 1 July 2026 that it was changing its crawler defaults. From 15 September 2026, new customers and new sites default to allowing search but blocking training and agent use on any page that carries ads. Mixed crawlers that cannot separate those uses get blocked on ad pages entirely.

The part that catches people out is the last clause in that announcement. Existing free customers who never touched their settings also received the new defaults. If you set up a Cloudflare site two years ago on the free plan and forgot about it, your bot policy changed without you doing anything.

If your marketing site carries no ads, this specific default does not bite you. Most B2B SaaS sites do not run ads on their own pages. But the direction of travel matters more than this one switch, and the direction is toward crawling being negotiated instead of assumed.

Why Are Publishers So Angry About AI Crawlers?

Because the old trade broke. Search engines crawled your pages and sent you visitors in return. AI answer engines crawl your pages and often send nobody, because the answer is rendered inside the chat window. The crawl still costs you bandwidth. The visit never arrives.

Cloudflare gave this a name and a number. Its crawl-to-refer ratio divides HTML requests from a platform's crawlers by HTML requests referred from that platform. In a post published on 1 July 2025, Cloudflare Radar reported Anthropic at 70,900 to 1 for the week of 19 to 26 June 2025. Cloudflare noted the caveat that native app traffic often lacks referrer headers, which can overstate these ratios.

Whatever the exact number, the shape is real. A news publisher living on display ads feels that immediately. A B2B software company whose average contract runs into five figures feels it very differently, and that difference is the whole decision.

What Is Pay Per Use and Why Did Cloudflare Change Course?

Pay Per Use pays publishers when their content creates value rather than when it is fetched. Cloudflare announced it in the same July 2026 release. With Ceramic.ai, a creator is paid every time their content appears in Ceramic's AI search results. With You.com, they are paid when premium content is accessed.

The launch partners Cloudflare named were Ceramic.ai, You.com, beehiiv, Patreon, and Condé Nast. That list tells you who this is built for. Four of the five make money from audience attention or subscriptions. None of them is a software company selling annual contracts to procurement teams.

Cloudflare gave a practical reason for the shift too. It said more than half of crawl traffic from AI crawlers is spent re-fetching pages that have not changed. Charging per fetch rewards waste. Paying per use does not.

Does Charging Crawlers Hurt Your AI Search Visibility?

Yes, almost certainly, and that is the trade. A crawler that gets a 402 or a 403 has no content to cite. If you are not in the index, you cannot be the source ChatGPT names, and you cannot be the link Perplexity puts in its sidebar. You have traded a citation for a payment that may never come.

For a B2B company this is usually a bad swap. One cited answer that puts your name in front of a buyer comparing three vendors is worth more than a fraction of a cent per crawl. We wrote about the mechanics of that in our guide to getting cited by AI search engines, and nothing about Pay Per Use changes the underlying maths.

The calculation flips when your content is the product. A research firm that sells reports, a publisher with a paywall, a creator with paid subscribers: those businesses lose real revenue when an answer engine summarises the thing people were meant to buy.

How Do You Tell Crawlers What They Can Do With Your Content?

Use the Content Signals Policy. Cloudflare announced it on 24 September 2025 as a machine-readable addition to robots.txt. It defines three signals: search, for building a search index; ai-input, for feeding content into a model at answer time; and ai-train, for training or fine-tuning models.

The syntax is short. A line such as Content-Signal: search=yes, ai-train=no sits in robots.txt and states your preference without blocking anything. Cloudflare said over 3.8 million Cloudflare-managed domains had already picked it up through managed robots.txt before the announcement.

Signals are preferences, not walls. A crawler can ignore them. But they are cheap, they are honest, and they create a written record of what you asked for. If you are editing robots.txt anyway, our guide to controlling AI crawlers in robots.txt covers the rest of the file.

Who Should Actually Turn Charging On?

Publishers whose content is the product, and almost nobody else. If a reader pays you for words, or an advertiser pays you for attention on those words, then an AI answer that replaces the visit is a direct loss and charging for the crawl is a reasonable response.

If you sell software, services, or anything with a sales conversation attached, your content is marketing. It exists to be found, quoted, and passed around. Charging for it is like putting a turnstile on your own billboard. The crawl cost is real but small, and the citation is the thing you were trying to buy.

There is a middle position we like. Allow the search and answer bots, refuse the training-only ones, and say so in your Content Signals line. You keep the visibility and you stop giving away a training corpus for free.

What Should a B2B Team Check This Week?

Four things, and they take about an hour. Open your robots.txt and read it as it is served, not as you remember writing it. Open your Cloudflare bot settings and look at what the September defaults did. Check that GPTBot, ClaudeBot, PerplexityBot, and Google-Extended can reach your key pages. Then check your logs to see whether they actually do.

That last step is the one teams skip. A rule that allows a bot is not proof the bot came. In our experience, allow rules and real crawl behaviour drift apart more often than teams expect, usually because of an old firewall or WAF rule nobody remembers adding.

While you are in there, it is worth confirming your llms.txt file still points at the pages you want models to read. These files drift.

Where Does This End Up?

Our guess is a split web. Content that makes money from attention moves behind payment and licensing deals, and Cloudflare said it has helped sign more than 50 major content licensing agreements in the past year. Content that makes money from being found stays open and competes harder to be the clearest source on its topic.

Cloudflare also said automated agents and bots now drive more than half of all web requests. If that is the shape of the web, then being readable by machines is not a side project. It is the same work as being readable by people, done once and done properly.

If you want a second pair of eyes on your crawler setup, or you are not sure whether your site is open to the engines that matter, we are happy to walk through it with you. That is the kind of thing we do every week at phoenix.studio, and it usually takes one call to find out whether anything is actually broken.