Does Site Speed Affect Whether AI Crawlers Read Your Pages?

Does Site Speed Affect Whether AI Crawlers Read Your Pages?

As of September 2026, Google says plainly that speed affects how much it crawls. The AI crawlers do not say. OpenAI documents its bots in detail and publishes nothing about crawl rate or timeouts, so anyone telling you a precise speed threshold for AI citation is guessing.

That gap is the interesting part, and it is where most advice on this topic goes wrong. There is a documented answer for one crawler and an honest unknown for the others, and the two get blended into a single confident claim.

Here is what is actually written down, what follows from it, and what we would do while the rest stays unknown.

What Does Google Say About Speed and Crawling?

It is unusually direct. Google's guide to managing crawl budget states that if a site responds consistently and its response times, including latency and Time to First Byte, remain stable or improve, the limit goes up, meaning more connections can be used to crawl.

The reverse is stated just as clearly. If the site slows down, with increasing latency or longer response times, or responds with server errors using 5xx status codes or rate-limiting signals such as HTTP 429, the limit goes down and Google crawls less.

Google also makes the rendering point separately: if Google can load and render your pages faster, it might be able to read more content from your site. Note the hedge in its own wording. This is a capacity relationship, not a ranking promise.

So for Google, speed is not a ranking trick here. It is a throughput constraint. A slow site is crawled less, and a site crawled less is discovered and refreshed more slowly. Our guide to crawl budget covers when that actually bites.

What Exactly Is the Crawl Capacity Limit?

Google calls it the crawl capacity limit, also known as hostload, and defines it as limiting the total amount of time your server spends holding connections open for Google. That definition is worth reading twice, because it is about your server's time rather than a page count.

The practical implication is that two sites with the same number of pages can have very different effective crawl capacity. If each of your pages takes four times as long to respond, you get roughly a quarter of the crawling for the same server-side cost.

Capacity is only half of it. Google separates crawl demand, which it says varies based on a site's size, update frequency, page quality and relevance compared to other sites. Speed raises your ceiling. It does not create demand where there is none.

That distinction matters for small sites. If you have 200 pages and Google is crawling all of them happily, faster responses will not produce more crawling, because demand is the binding constraint rather than capacity.

Do AI Crawlers Publish the Same Rules?

Not that we could find. OpenAI's bot documentation is genuinely detailed on identity: it lists each crawler with its exact user agent string, its purpose, how to allow or block it in robots.txt, and a JSON file of its IP ranges. On crawl rate, timeouts and caching behaviour, it says nothing.

That silence is information in itself. It means any claim about how many seconds an AI crawler will wait before giving up is not coming from the vendor, and should be treated as someone's observation at best.

One operational detail is documented and worth knowing. OpenAI states that it can take roughly 24 hours from a site's robots.txt update for its systems to adjust. So a change you make today does not take effect today, in either direction.

We would apply the general principle anyway. Any crawler running at scale has to bound how long it waits for a response, because none of them can afford to hold connections open indefinitely. That reasoning is sound, but it is reasoning, not documentation, and we are not going to dress it up as a number.

Which AI Crawlers Are Actually Hitting Your Site?

For OpenAI alone there are four, and they do different jobs. OAI-SearchBot, currently identifying as version 1.4, is used to surface websites in search results in ChatGPT's search features. GPTBot, also at 1.4, is used to make OpenAI's generative AI foundation models more useful and safe.

OAI-AdsBot at version 1.0 validates the safety of web pages submitted as ads on ChatGPT. ChatGPT-User at 1.0 handles certain user actions in ChatGPT and Custom GPTs, where a user asks for a page to be visited.

Blocking them is per-agent, so you can allow the one that surfaces you in ChatGPT search while disallowing the one that trains models. That is a real strategic choice and most sites make it by accident. Our guide to controlling AI crawlers with robots.txt covers how to make it deliberately.

Why Does ChatGPT-User Change the Calculation?

Because it is not a crawl, it is a visit happening while someone waits. OpenAI documents that ChatGPT-User covers actions initiated by a user requesting a page, and notes that robots.txt rules may not apply for that reason.

A background crawler that finds your site slow can simply come back later. A user-initiated fetch cannot. If your page does not respond promptly, the assistant answers from something else, and the person on the other end never learns you existed.

That is the strongest speed argument available in this whole area, and it does not depend on any undocumented threshold. It only requires that a real person is waiting, which is exactly the situation the agent name describes.

It also means the relevant metric is your slowest realistic response, not your median. A page that is usually fast and occasionally takes eight seconds fails exactly when it matters most. Look at your slowest decile rather than your average, because that is the traffic an assistant will experience on a bad day.

What Happens If You Slow Them Down Deliberately?

For Google, there is a documented procedure and a documented cost. To reduce Googlebot's crawl rate in an emergency, Google says to return a 500, 503 or 429 HTTP status code, and that once the number of these errors is reduced, the crawl rate will automatically start increasing again.

The warning attached is specific. Google says it does not recommend doing this for a long period, meaning longer than one to two days, as it may have a negative effect on how your site appears in Google products. It adds that if Googlebot observes these status codes on the same URL for multiple days, the URL may be dropped from Google's index.

There is a side effect worth noting too. Google says that reducing crawl rate means removed pages may stay in the index longer, so a throttled site is also slower to clean up.

The AI crawlers publish no equivalent procedure. If you need to shed load from them, the documented lever is robots.txt and the IP ranges they publish, with the roughly 24 hour adjustment delay for robots.txt changes in OpenAI's case.

Does Rendering Matter as Much as Speed?

Often more. Google's own framing joins the two: if it can load and render your pages faster, it might read more content. Rendering is where a fast server can still produce a slow, empty page.

For AI crawlers the risk is sharper, because you cannot assume any given one executes JavaScript at all. A page whose content only exists after a client-side fetch may be perfectly fast and still be empty to whoever is reading it.

This is why we keep pushing prerendered or server-rendered HTML for anything that needs to be quoted. It removes the question rather than answering it. Our guide to JavaScript rendering and SEO covers the details.

How Would You Actually Test This?

With your own server logs, which are the only place the real answer lives. Filter by the documented user agent strings, cross-check against the published IP range files so you are not counting impostors, and look at response times and status codes per agent.

Three things are worth measuring. Whether any agent sees materially slower responses than human traffic, which usually means they are hitting uncached paths. Whether any of them receive 5xx or 429 responses, which for Googlebot directly reduces crawling. And whether crawl volume from each agent trends with your response times over weeks.

That last one is the closest thing to a real experiment available to a single site, and it is imperfect because you cannot hold everything else constant. Report it to yourself as a correlation, not a proof. Our guide to log file analysis covers the mechanics.

What Should You Do About It?

Serve fast, cacheable, fully rendered HTML to everyone and stop optimising per crawler. Every documented and undocumented factor in this article points the same way, and none of them reward a special path for bots.

Then do the two things that are specifically documented. Make sure you are not returning 5xx or 429 to Googlebot, because that is a stated cause of reduced crawling and, over multiple days on the same URL, of dropped indexing. And set your robots.txt rules for each AI agent deliberately, knowing changes take about a day to take effect for OpenAI's bots.

What we would not do is chase a speed target invented for AI crawlers. No vendor publishes one, and building work around a number nobody stands behind is how teams end up optimising for folklore.

What Are We Watching Next?

Whether the AI vendors start publishing crawl behaviour the way search engines eventually did. Google documented hostload because site owners needed it, and the same pressure is building now that AI crawler traffic is a real share of many logs.

Until they do, the honest position is the one we have taken here. One documented relationship, one strong argument from the user-initiated case, and an explicit unknown in the middle that nobody should be filling in with confident numbers.

If you want help reading your own logs to see how these crawlers are actually experiencing your site, we are happy to walk through it. You can find us at phoenix.studio.