Robots.txt is now the file that decides whether AI systems can read your content. It used to control Googlebot and little else. Today it also tells GPTBot, ClaudeBot, PerplexityBot, and Google-Extended what they may take. One wrong line can hide you from AI search entirely.
For years this file was an afterthought. Most site owners never opened it. It sat in the root folder, blocked a few admin pages, and did its job quietly.
That changed fast. A growing share of the bots hitting your server now belong to AI companies, not search engines. They read your pages to train models, to answer live questions, or to fetch a page a user just asked about. Each of those jobs has a different bot behind it, and each one obeys a different line in your file.
Robots.txt is a plain text file at the root of your domain. It lists rules that tell automated crawlers which paths they may request. Each rule block names a user agent, then allows or disallows paths. Crawlers read the file before they fetch anything else on your site.
The format is older than most websites. Martijn Koster defined it in 1994 as an informal convention. It stayed informal for almost thirty years.
The IETF finally standardized it as RFC 9309 in September 2022, on the standards track. That document sets the formal syntax for user-agent matching, allow and disallow directives, wildcards, and caching. It is worth reading once if you own a site, because it settles most arguments about how the rules are supposed to be parsed.
RFC 9309 is also blunt about one thing. It states that these rules are not a form of access authorization. Robots.txt is a request, not a lock. That distinction changes how you should think about protecting anything sensitive, and we come back to it later.
Four companies send most of the AI crawler traffic you will see. OpenAI sends GPTBot, OAI-SearchBot, and ChatGPT-User. Anthropic sends ClaudeBot. Google uses Googlebot plus a separate token called Google-Extended. Perplexity sends PerplexityBot and Perplexity-User. Each name does a different job.
OpenAI''s developer documentation splits its bots by purpose. GPTBot collects content to improve generative AI foundation models. OAI-SearchBot surfaces websites inside ChatGPT search features, and OpenAI states that sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers. ChatGPT-User fires when a person asks ChatGPT to look at a page, and OpenAI notes it is not used for crawling the web in an automatic fashion.
Anthropic runs ClaudeBot as a general purpose web crawler for training data. Its help documentation says ClaudeBot follows robots.txt, and that it also supports the non-standard Crawl-delay extension if you want to slow the bot down rather than block it. Anthropic publishes an address, claudebot@anthropic.com, for site owners with questions.
Perplexity documents two agents. PerplexityBot surfaces and links websites in Perplexity search results, and Perplexity says it is not used to crawl content for AI foundation models. Perplexity-User handles requests when someone asks a question inside the platform, and Perplexity states that this agent generally ignores robots.txt rules because a real person started the request.
For most businesses, let them in. AI answer engines are becoming a real referral channel, and blocking the search side of these bots removes you from those answers. Blocking only the training bots is a defensible middle path. It will not keep you visible on its own.
We work with founders and marketers who want to be found, not hidden. If someone asks ChatGPT or Perplexity for a recommendation in your category, you want to be in that answer. Blocking OAI-SearchBot or PerplexityBot guarantees you are not. That is a strange trade for a business that already spends money on marketing.
The argument for blocking is about fairness, not visibility. Cloudflare put numbers on it in a blog post published on 1 July 2025. It compared how much AI companies crawl against how much traffic they send back. With OpenAI, Cloudflare wrote, it is 750 times more difficult to get traffic than it was with the Google of old. With Anthropic, 30,000 times more difficult. Those figures explain why publishers are angry even when their rankings are fine.
Our position is that publishers with paywalled or licensable archives have a real case for blocking. A service business with a marketing site usually does not. If your goal is to be cited rather than scraped, the work sits in the content itself, and we covered that in our guide to getting cited by AI search engines.
You write one rule block per bot. Start a line with User-agent, a colon, then the bot name. On the next line write Disallow, a colon, and a single forward slash to block the whole site. Use Allow with a forward slash to permit it. Repeat the pattern for each bot.
To block Anthropic''s crawler, Anthropic''s own documentation gives the exact pattern: a User-agent line naming ClaudeBot, then a Disallow line with a single forward slash. Anthropic adds an important detail. You have to repeat this on every subdomain you want to opt out, because a robots.txt file only covers the host it sits on.
The same shape works for GPTBot, OAI-SearchBot, PerplexityBot, and Google-Extended. You swap the name after User-agent. Precision matters far more than order. A typo in a bot name means the rule silently does nothing, and no tool will warn you that it failed.
One more file comes up in the same conversation. Some site owners now publish an llms.txt file alongside robots.txt to give AI systems a clean map of their content. The two files do different jobs, and we walked through what llms.txt is and whether it earns its place in our piece on llms.txt.
Training and answering are separate systems with separate crawlers. Blocking a training bot stops your content from feeding a future model. It does not stop a live search bot from fetching your page today to answer a question. Google makes this split explicit in its own documentation.
Google Search Central describes Google-Extended as a standalone product token that publishers use to manage whether content Google crawls may be used for training future generations of Gemini models. It also covers grounding in Gemini Apps and the Vertex AI API. Google states clearly that this token does not affect inclusion or ranking in Google Search.
That is the cleanest example of the split. You can tell Google not to train on your work while staying fully indexed. Googlebot stays in charge of Google Search, Discover, Google Images, Google Video, and Google News. Google-Extended only touches the model training side.
OpenAI draws a similar line between GPTBot and OAI-SearchBot. Block only GPTBot and you stay eligible for ChatGPT search. Block OAI-SearchBot and you do not. In our experience, most people who say they blocked AI meant to block training, and blocked their own visibility instead.
Cloudflare moved the decision from your text file to the network edge. In a press release dated 1 July 2025, Cloudflare said it became the first internet infrastructure provider to block AI crawlers accessing content without permission or compensation, by default. That turns a polite request into actual enforcement.
The mechanism differs from robots.txt in one important way. Robots.txt asks. A rule at the edge refuses the connection. A crawler that ignores your text file cannot ignore a request that never completes.
Cloudflare launched Pay Per Crawl alongside that announcement. Its AI Crawl Control documentation describes three choices per crawler. You can allow it free access to your content, charge it at a flat per-request price you set across the domain, or block it entirely with no option to pay. Cloudflare launched Pay Per Crawl in private beta.
If your site sits behind Cloudflare, check those settings before you spend an afternoon tuning robots.txt. The edge configuration wins. Nothing surfaces a conflict between the two, so a site can run for months with a robots.txt that says one thing and an edge rule that does another.
No. Robots.txt is voluntary. RFC 9309 says plainly that its rules are not a form of access authorization. Well-behaved crawlers from OpenAI, Anthropic, Google, and Perplexity honor it. A crawler that does not care simply ignores the file and requests your pages anyway.
There is a second gap that surprises people. Some agents are designed to skip robots.txt on purpose, and their makers say so in public. Perplexity documents that Perplexity-User generally ignores robots.txt rules because a user initiated the request. OpenAI notes that robots.txt rules may not apply to the user-initiated actions behind ChatGPT-User. The reasoning is that a person clicking is not a crawl.
So if a page must stay private, robots.txt is the wrong tool. Use authentication, a firewall rule, or simply do not publish the page. Anthropic also warns that blocking by IP address is unreliable, partly because it can stop a crawler from reading your robots.txt in the first place.
Read your server logs. Every request carries a user-agent string, so your logs show which bots arrived, how often, and which pages they took. That is more accurate than guessing from a third-party dashboard. Pair the logs with Google Search Console for the Googlebot side of the picture.
Look for the names in this article, then look at the ratio. If GPTBot has crawled thousands of pages and sent nothing back, the Cloudflare argument is showing up in your own data. If OAI-SearchBot never appears at all, check whether a rule is quietly blocking it.
There is a related failure worth ruling out at the same time. If your pages are assembled by JavaScript in the browser, some crawlers will fetch the URL and find an empty shell. Access is not the same as readability, and we explained that gap in our article on JavaScript rendering and SEO. Across the 150+ projects we have delivered, that gap has been a more common cause of missing AI visibility than any robots.txt rule.
Open the file at your own domain followed by slash robots dot txt and read it. Confirm that nothing blocks OAI-SearchBot, PerplexityBot, or Googlebot by accident. Decide on the training bots as a business choice rather than a default. Then check your Cloudflare settings, because the edge overrides the file.
In our view most sites need a small correction here, not a rewrite. A stale disallow rule left over from a staging site. A wildcard that catches more than it should. A subdomain with no robots.txt at all. Those take minutes to fix, and they matter a little more every month.
If you want a second set of eyes on your crawler setup, or you are not sure whether AI systems can read your site at all, let''s talk. We are happy to walk through your robots.txt and your server logs with you and tell you what we would change. You can reach us at phoenix.studio.
Tell us where you want to go. We'll tell you how we'd get you there.