Googlebot is the generic name for Google's main web crawler for Search. It finds URLs through links, sitemaps and other sources, requests them from your server, and hands the responses to Google's systems for rendering and indexing. Almost all sites are now crawled by the smartphone version of Googlebot, which identifies itself as a mobile Chrome browser.
How Googlebot works
Googlebot keeps a queue of URLs to visit. It takes URLs from links on pages it has already crawled, from XML sitemaps, from redirects and from other discovery sources. For each URL it first checks robots.txt for the host, then requests the page, records the status code and headers, and stores the HTML.
Pages that rely on JavaScript are then queued for rendering by Google's Web Rendering Service, which runs a recent version of Chromium. The rendered result is what Google uses to read content and links. Google documents that Googlebot reads only the first 15 MB of an HTML or supported text file, which matters only for unusually large pages.
Googlebot Smartphone and other Google crawlers
Since the move to mobile-first indexing, Googlebot Smartphone does the main crawling for nearly every site. A desktop variant still exists for a small number of cases. Google also runs other crawlers with their own user agent tokens, such as Googlebot-Image, Googlebot-Video and Storebot-Google, plus special-purpose fetchers that act on a user's request.
Google-Extended is a separate robots.txt token. It does not crawl; it lets you say whether content Googlebot fetches may be used for some of Google's generative AI products. Blocking it does not affect Search crawling or ranking, according to Google's documentation.
Verifying that a visit really came from Googlebot
Anyone can put Googlebot in a user agent string, and scrapers often do. To confirm a visit is genuine, run a reverse DNS lookup on the IP address, check that the hostname ends in googlebot.com, google.com or googleusercontent.com, then run a forward lookup to confirm it resolves back to the same IP. Google also publishes JSON files listing its crawler IP ranges, which is easier to automate.
$ host 66.249.66.1
1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.
$ host crawl-66-249-66-1.googlebot.com
crawl-66-249-66-1.googlebot.com has address 66.249.66.1Example
Spotting fake Googlebot traffic in the logs
Suppose an analytics SaaS called Ledgerfern sees a spike of requests labeled Googlebot hitting its pricing page thousands of times a day. Its engineer checks a sample of IPs against Google's published ranges and finds none match. The traffic is a scraper. The team rate-limits it at the CDN, and real Googlebot activity, confirmed by reverse DNS, carries on as before.
Common mistakes
Blocking by user agent alone
Firewall rules that trust or block the string Googlebot can be fooled in both directions. Verify by IP or DNS before treating traffic as Google.
Serving Googlebot something different
Showing crawlers different content from users is cloaking and breaks Google's spam policies. Personalization is fine; a different page for bots is not.
Questions
There is no fixed schedule. Google adjusts crawl frequency based on how often pages change, how important they appear and how well the server copes. The Crawl stats report in Search Console shows the actual pattern for your property.
Not directly. You can request indexing of individual URLs with the URL Inspection tool and submit sitemaps, but overall crawl rate is set by Google. You can only signal it to slow down, for example by returning 503 or 429 responses.
