Every request to a web server can be recorded in an access log: the time, the URL, the status code, the user agent and the IP address. Filtering those logs for crawler requests shows what bots actually do on your site, rather than what a third-party tool simulates. It is the most direct way to study crawl budget, find crawl waste, confirm fixes, and see how AI crawlers are behaving.
What a log line contains
Formats vary by server and CDN, but a typical line includes the client IP, timestamp, request method and URL, status code, response size, referrer and user agent. On sites served through a CDN or edge platform, the useful logs are often at the CDN, because the origin server never sees cached requests.
66.249.66.1 - - [14/Oct/2026:08:12:44 +0000] "GET /integrations/salesforce HTTP/1.1" 200 48211 "-" "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"Verify crawlers before trusting the data
Anyone can fake a user agent. Google documents two ways to verify Googlebot: a reverse DNS lookup on the IP that resolves to a googlebot.com, google.com or googleusercontent.com hostname, followed by a forward lookup that matches; or checking the IP against the JSON lists of crawler IP ranges Google publishes. Bing and several AI vendors publish their own verification methods or IP ranges. Discard unverified traffic claiming to be a known crawler.
Questions logs can answer
The value is in specific, comparable questions.
- Which sections does Googlebot crawl most and least?
- How many requests go to parameter URLs, redirects, 404s or 5xx errors?
- How long after publishing is a new page first crawled?
- Are important pages being crawled at all, and how often?
- Which AI crawlers visit, what do they request, and do they respect robots.txt?
- Did a fix, such as a robots.txt rule or a redirect cleanup, change crawler behavior?
Getting and handling logs
Logs can be large and often include personal data such as IP addresses, so involve whoever owns infrastructure and privacy. Many teams export only lines with known crawler user agents. Dedicated log analysis tools, general-purpose tools like a data warehouse or spreadsheet for small sites, and CDN analytics dashboards can all work. Search Console's Crawl Stats report is a lighter alternative for Googlebot only.
Example
Finding where Googlebot spends its time
A fictional HR software company, Staffloop, exports a month of CDN logs filtered to verified Googlebot requests. Grouping by URL pattern shows a large share of requests going to calendar pages under /events/ with date parameters stretching years into the future, while new comparison pages get few visits.
Blocking the infinite calendar parameters in robots.txt and adding links to the comparison pages from the main navigation shifts crawling. The next month's logs confirm the change.
Common mistakes
Analyzing origin logs behind a CDN
Cached responses never reach the origin. Use CDN or edge logs, or combine both.
Trusting user agents without verification
Scrapers often pretend to be Googlebot. Verify by DNS or published IP ranges before drawing conclusions.
Questions
Usually not. Search Console's Crawl Stats report covers most needs. Logs pay off on large sites, sites with crawl problems, and when you want to understand AI crawler activity.
Enough to cover normal variation, often several weeks. Keep before and after windows when measuring a change.
Yes, for crawlers that identify themselves. Some AI fetches happen through user-triggered agents with different user agents, and vendor documentation on this varies, so expect gaps.
