Almost certainly not. Google's crawl budget documentation is aimed at sites with more than one million pages that change weekly, or more than 10,000 pages that change daily. If your site is smaller than that, crawl budget is not your problem, and time spent on it is time taken from work that would help.
We bring this up because crawl budget is one of the most oversold ideas in SEO. It sounds technical and important, so it gets recommended constantly to businesses running forty page marketing sites. Those sites almost never have a crawling problem. They have a content problem or a linking problem wearing a crawling costume.
That said, something did genuinely change in 2026, and it changes who should pay attention. Google updated how it describes crawl capacity, and the crawler landscape outside Google got a lot more crowded. Both are worth understanding even if you never touch a robots.txt file.
Google defines crawl budget as the set of URLs that Google can and wants to crawl. It has two halves. Crawl capacity limit is how much your server can handle. Crawl demand is how much Google wants your pages. A site gets crawled at the lower of those two numbers.
Crawl capacity limit is about not breaking your server. Google's documentation describes it as a limit on the total amount of time your server spends holding connections open for Google. Google starts conservative and raises the limit if your server stays fast and healthy under load.
Crawl demand is the other side. Google says demand depends on a site's size, update frequency, page quality, and relevance compared to other sites. This is the half most people ignore, and it is the half that usually explains a low crawl rate on a small site. Google is not throttled by your server. It just does not want the pages very much.
That distinction matters because the two halves have completely different fixes. A capacity problem is solved with hosting. A demand problem is solved with better content and better internal links. Confusing them is how people end up buying a bigger server for a problem no server can fix.
Google names three cases. Large sites with more than one million unique pages whose content changes about weekly. Medium or larger sites with more than 10,000 unique pages whose content changes daily. And any site with a large share of pages sitting in the Discovered, currently not indexed state in Search Console.
Google is careful to call these rough estimates rather than exact thresholds, so do not treat one million as a bright line. The spirit of the guidance is what matters. Crawl budget becomes real when the number of URLs on your site is large enough that Google has to make choices about which ones to fetch.
The third case is the one worth watching, because it catches smaller sites. If Search Console shows thousands of URLs discovered but never crawled, something is generating URLs faster than Google wants to fetch them. That is usually faceted navigation, filter parameters, or a search results page that got itself indexed. Our guide to reading Google Search Console properly covers how to find that report and what the states actually mean.
An ecommerce site with 900 products can easily generate 90,000 crawlable URLs through colour and size filters. That is the shape of a real crawl budget problem on a site that does not look large.
Because Google now states plainly that crawl capacity is shared. In documentation updated on 22 July 2026, Google says the crawl capacity limit is shared across all crawlers, and that high demand from one crawler can reduce the capacity available for others. Google runs many crawlers, not just Googlebot.
This is a meaningful clarification. Google operates separate crawlers for Search, for images, for video, for its ad products, and for its AI products. If one of them is working hard on your site, the others get less room. Your server capacity is a single pool that all of them draw from.
The practical takeaway is unglamorous. Server speed is now an SEO input in a more direct way than it used to be, because a slow server shrinks a pool that more consumers are drinking from. This is one of the few places where a hosting upgrade genuinely is the answer.
Google also states there are only two ways to increase your crawl budget at all. Add server resources, or improve the quality and relevance of your content for the Google product you care about. There is no setting, no submission, and no request form. Anyone selling you a third option is selling you something else.
They add load without adding a search ranking. GPTBot, PerplexityBot, Applebot, and Bingbot all fetch pages from your server the same way Googlebot does, and they consume the same finite server capacity. Unlike Googlebot, most of them do not feed a ranking system that sends you traffic in return.
This is worth stating carefully, because it is easy to get wrong. AI crawlers do not consume Google's crawl budget in any direct sense. Google's crawl capacity limit is about Google's own connections to your server. What AI crawlers consume is your server's actual capacity, and a server busy answering GPTBot is slower for Googlebot, which is exactly the condition that makes Google throttle itself.
So the effect is real but indirect. It runs through server health, not through some shared budget ledger. That distinction matters when you are deciding what to do about it, because it means the fix is a performance and access decision rather than an SEO one.
Cloudflare has been the loudest voice here. Its Content Independence Day campaign argues that AI crawlers should not crawl without compensating the people who made the content. That framing has moved from opinion to shipped product faster than most people expected.
Only if you have measured that they are actually a problem. Blocking is a real trade. AI answer engines cite what they can read, so a blanket block on GPTBot removes you from ChatGPT answers. For most business sites, appearing in AI answers is worth more than the server load costs.
Cloudflare has made this decision much more granular. Its current controls split AI bot traffic into three categories, which it calls Search, Agent, and Training, so you can allow the crawling that gets you cited while refusing the crawling that only trains a model. That is a much better trade than an all or nothing block.
There is a trap in there that site owners should know about. Cloudflare says multi purpose crawlers, including Googlebot, Applebot, and Bingbot, follow all of their behaviours under the new system. If you disable Training crawling, those crawlers get blocked even though you allowed Search. Blocking training on a mixed use crawler can therefore cost you search crawling too.
Cloudflare has also said that from 15 September 2026, newly onboarded domains get new defaults on pages that display ads, with Training and Agent bots blocked by default and Search bots still allowed. If you run an ad supported site on Cloudflare, that default is worth checking rather than discovering later. We go deeper on the access side of this in our guide to controlling AI crawlers with robots.txt.
Cut the URLs Google should never fetch. Google's own guidance is to consolidate duplicate content, block unimportant pages in robots.txt, return a clean 404 or 410 for permanently removed pages, keep sitemaps updated with accurate lastmod values, and avoid long redirect chains. Each one removes wasted fetches.
Faceted navigation is the biggest offender we see. Every filter combination that produces a unique crawlable URL multiplies your page count without adding a page anyone would search for. The fix is usually to stop generating links to those combinations rather than to block them after the fact.
Redirect chains are the quietest offender. Every hop in a chain is a separate fetch, and a site that has been through three redesigns often has URLs redirecting four times before landing. That is four times the crawl cost for one page. We covered how to audit and flatten those in our guide to setting up redirects without losing traffic.
The 304 Not Modified response is the most underused item on Google's list. If your server can tell Google a page has not changed since the last fetch, Google can skip downloading it. On a large site with mostly static pages, supporting 304 properly is one of the highest leverage changes available.
No, and this is the most common mistake in the whole topic. Google has to fetch a page to see the noindex tag. So a noindexed page still costs a crawl every time. Google's guidance is explicit that you should use robots.txt to block unimportant pages rather than relying on noindex.
The two tools do different jobs. Robots.txt controls crawling, which is whether Google fetches the page at all. Noindex controls indexing, which is whether a fetched page appears in results. Using the indexing tool to solve a crawling problem does not work, because the fetch has already happened by the time the tag is read.
There is a real catch worth knowing. If you block a URL in robots.txt, Google cannot fetch it, which means Google cannot see a noindex tag on it either. A page blocked in robots.txt can still appear in search results if enough other sites link to it. If you need a page gone from results, let Google crawl it and serve noindex. If you need to stop the crawling, block it and accept that removal is a separate job.
Open the Crawl Stats report in Google Search Console and compare total crawl requests against your real page count. If Google is fetching far more URLs than you have pages, you have a waste problem. If it is fetching far fewer and your server is fast, you have a demand problem, which is a content and linking issue.
Then check server response time in the same report. If average response time is climbing while crawl requests fall, that is the capacity limit doing its job, and hosting is your answer. If response time is flat and low, capacity is not the constraint and no amount of crawl budget work will help.
Honestly, most of the audits we run end at this step. The Crawl Stats report shows a healthy server, a sensible request count, and pages that Google is happy to fetch and simply does not rank. That is a relief, not a dead end, because content and internal linking problems are far more fixable than infrastructure ones.
Count your indexable URLs. If the number is under a few thousand and your server responds quickly, close the tab and go work on your content. If the number is large, or Search Console shows a pile of discovered but uncrawled URLs, start with faceted navigation and redirect chains before touching anything else.
The one thing worth doing regardless of size is checking your Cloudflare bot settings if you use Cloudflare, given the September default change. Knowing which crawlers you allow is basic hygiene now, and the mixed use crawler behaviour means a careless setting can quietly cost you search crawling.
If you are staring at Search Console trying to work out whether your crawling is genuinely broken or just boring, we are happy to take a look with you. We would rather tell you your crawl budget is fine and point you at the real problem than sell you a month of technical work you do not need. Come find us at phoenix.studio.
Tell us where you want to go. We'll tell you how we'd get you there.