Because most reports tell you what got indexed, not what got visited. Your server knows every single request that arrived, including the ones from crawlers that never resulted in an indexed page. That record is sitting in your access logs, and almost nobody reads it.
We think of it as the difference between a summary and a receipt. Search Console gives you a summary Google chose to share. The log file is the receipt for every request that actually hit your server.
That gap matters more now, because Google is no longer the only crawler you care about. AI answer engines send their own bots, and they behave differently.
Log file analysis is reading your web server's access logs to see exactly which pages were requested, by whom, and when. Every request writes a line containing the IP address, the timestamp, the URL, the response code, and the user agent string. Filtering those lines for crawler user agents shows you real crawl behaviour.
It is the least glamorous SEO technique and one of the most honest. There is no sampling, no modelling, and no estimate. It is a list of things that happened.
The format is standard enough that the same approach works on Nginx and Apache, and most content delivery networks including Cloudflare and AWS CloudFront can hand you a similar log stream.
Three things stand out. You learn which URLs crawlers waste time on, you learn how often they return to your important pages, and you learn what response codes they actually receive. Search Console reports on Google only, and it reports after processing. Logs show every bot, unfiltered.
The wasted crawl finding is usually the most useful. We regularly see crawlers hammering faceted URLs, old paginated paths, and parameter variants that nobody wants indexed, while genuinely important pages get visited rarely.
Response codes are the second win. A page can look fine in a browser and return a soft error or a redirect chain to a bot. Logs show you the status code the crawler received, not the one you assume it got.
Crawl frequency on your money pages is the third. If your pricing page is visited once a month and your archive is visited daily, something in your internal linking is pointing attention at the wrong place. That connects directly to how you think about crawl budget.
Never trust a user agent string, because anyone can set one. Google's documentation describes verifying Googlebot with a reverse DNS lookup on the requesting IP address, checking that it resolves to googlebot.com, google.com, or googleusercontent.com, then running a forward DNS lookup on that domain to confirm it matches the original IP.
Google also publishes IP ranges you can match against automatically. Its documentation names common-crawlers.json for standard crawlers like Googlebot, special-crawlers.json for crawlers like AdsBot, and separate files for user-triggered fetchers. The ranges are published in CIDR format.
This matters because scrapers routinely pretend to be Googlebot to get past crude blocking rules. If you make decisions based on unverified user agents, you are counting traffic that was never Google at all.
Anthropic publishes a list of authorised crawler IP addresses for the same purpose. Its documentation also warns against blocking by IP address, because doing so can stop the crawler reading your robots.txt file in the first place.
The named ones from the major model providers, and it is worth knowing that each company runs several bots with different jobs. Training crawlers, search crawlers, and user-triggered fetchers are separate agents with separate rules, and blocking one does not block the others.
OpenAI documents four. GPTBot is described as crawling content that may be used to train its generative AI foundation models. OAI-SearchBot is used to surface websites in search results in ChatGPT's search features. OAI-AdsBot validates the safety of pages submitted as ads. ChatGPT-User handles certain user actions in ChatGPT and Custom GPTs.
Anthropic documents three. ClaudeBot collects web content that could contribute to model training, Claude-User supports user-initiated web requests inside Claude conversations, and Claude-SearchBot navigates the web to improve search result quality.
The distinction has a real consequence. If you want to stay out of training data but still appear in AI search answers, you block the training crawler and allow the search crawler. We walked through the actual syntax in our guide to controlling AI crawlers with robots.txt.
One caveat from OpenAI's own documentation is worth reading twice. It notes that for ChatGPT-User, robots.txt rules may not apply, because those requests are initiated by a person rather than by an automated crawl.
Your most important pages get visited most often, your response codes are almost entirely 200s, and crawl volume is roughly steady rather than spiking and vanishing. There is no universal correct number of crawls. The pattern matters far more than the total.
The clearest warning sign is a large share of requests hitting URLs you do not care about. If half your crawl budget goes to filtered listing pages, that is half the attention your service pages are not getting.
A second warning sign is a rising count of 404 and 301 responses to bots. Redirect chains in particular waste crawl on every hop, and they tend to accumulate quietly after a migration.
A sudden drop in crawl frequency after a site change is worth taking seriously. It often points at something structural like a blocked path or a broken sitemap rather than a content problem.
Often not directly, and this is the honest limitation. Many fully managed website platforms do not hand site owners raw server access logs, so the technique can be unavailable on exactly the setups where it would be easiest to fix things. Check what your host actually exposes before planning work around it.
The usual workaround is to put a content delivery network in front of the site. Running through Cloudflare, for example, gives you a request log at the edge that contains the same information your origin server would have written.
If neither is possible, you are not helpless. You can get partial answers from Search Console's crawl stats report for Google, and you can infer AI crawler behaviour from referral patterns, which we covered in our guide to measuring AI search traffic.
We would rather tell a client the technique does not apply to their setup than sell them an analysis built on incomplete data.
For a small site, a text editor and basic command line filtering will do. For anything larger, a dedicated tool is worth it. Screaming Frog makes a Log File Analyser built for this specific job, and it handles the parsing and bot verification that becomes tedious by hand at volume.
Whatever you use, the first step is always the same. Filter to verified crawler requests only, then group by URL and by response code. Almost every useful finding falls out of that one view.
Do not start by building a dashboard. Start by answering one question, such as which URLs Googlebot requested most last month. Dashboards built before you know the question tend to measure the wrong thing beautifully.
Usually not as a routine, but often as a one-off. On a fifty page site, crawl budget is not your constraint and you will learn little. On a site with thousands of URLs, or one that just went through a migration, it is one of the highest value hours you can spend.
Our honest position is that this is a diagnostic tool, not a monitoring habit. We reach for it when something is wrong and the usual reports do not explain it.
The other good trigger is a big structural change. After a migration or a CMS change, logs tell you within days whether crawlers found the new structure, long before rankings move enough to be readable.
First, find out whether you can get your access logs at all. That single question decides everything else. If you can, pull one month, filter to verified crawler requests, and group them by URL to see where the attention is going.
If you cannot get logs, do not force it. Use Search Console crawl stats for Google, keep your robots.txt rules deliberate about which AI crawlers you allow, and spend the effort on internal linking instead.
If you want help reading what your logs are telling you, or deciding which crawlers to let in at all, we are happy to walk through it. Reach us at phoenix.studio and we will take a look with you.
Tell us where you want to go. We'll tell you how we'd get you there.