Google Crawlers and User Agents The Complete List Explained

Google sorts its bots into three groups. Common crawlers are used across its products, with Googlebot as the main one, and Google says they always respect robots.txt rules for automatic crawls. Special-case crawlers serve specific products that have an agreement with the crawled site. AdsBot is the example Google gives, and it can ignore the global * robots.txt rule with the ad publisher’s permission. User-triggered fetchers work when a person starts the fetch, like a tool that checks your site. Google Site Verifier is the example.

Google also separates crawlers from fetchers. A crawler automatically discovers and scans sites. A fetcher works more like wget, usually making a single request on a user’s behalf. If you want the wider story of how pages get found, my how Google Search works guide covers the stages.

// source: Google Search Central, Google’s common crawlers. Google says the list is not exhaustive.

Two details deserve attention. Google-Extended is a control token, with no crawler of its own. Google says it governs whether crawled content may be used to train future Gemini models and for grounding, and that it does not affect Search inclusion or ranking. And Google-InspectionTool powers the testing tools, so it has no effect on Search itself.

Google documents a smartphone and a desktop version of Googlebot. Both contain that “compatible; Googlebot/2.1” phrase. There are two rare variants as well: the short Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) and Googlebot/2.1 (+http://www.google.com/bot.html).

The full strings include a Chrome version written as “Chrome/W.X.Y.Z”. Google says this is a placeholder for the Chrome version, which increases over time. So when you search your logs, use wildcards, not an exact version. The image and video crawlers use their own tokens in the string, such as Googlebot-Image/1.0.

Google’s crawlers identify themselves by three signals: the user agent header, the source IP address, and the reverse DNS hostname of that IP. For common crawlers, Google says they generally come from IP ranges in its published common-crawlers.json file, and their hostnames look like crawl-***-***-***-***.googlebot.com or geo-crawl-***-***-***-***.geo.googlebot.com.

So the rule is simple. If something claims to be Googlebot, check its IP and reverse DNS before you whitelist it or panic about it. I walk through the exact commands in my guide to how to verify Googlebot.

1. Look at your logs or Crawl stats. Note which Google user agents appear. Most small sites see mainly Googlebot.
2. Decide per crawler. Do you want your images in Google Images? Then do not block Googlebot-Image. Want Search visibility? Do not block Googlebot.
3. Write robots.txt rules with the right token. Google says each crawler may have several tokens and a rule needs to match only one. My robots.txt syntax rules guide shows how.
4. Handle AI training separately. Google-Extended is the control token for Gemini training and grounding, and it is independent of Search.
5. Verify suspicious visitors. Anything claiming to be Google that fails the DNS check is not Google.

// check order summarized from Google’s verification guide. Details are in my verify-Googlebot post.

• Use the URL Inspection tool on a live page and confirm Google can fetch it. The tool itself runs as Google-InspectionTool.
• After changing robots.txt, test the rule for the exact token you edited.
• Check your logs a few days later for the crawlers you expect, using wildcard patterns.
• Verify any odd visitor with the DNS method before blocking or allowing it.

Crawl access is a core part of technical SEO, and a cheap SEO company like mine will check it in the first audit. Related: the crawl stats report shows what Google actually fetched.

• Google Search Central: Google’s common crawlers (tokens, user agent strings, hostnames)
• Google Search Central: Overview of Google crawlers and fetchers (three categories, identification)
• Google Search Central: Verifying Googlebot and other Google crawlers (robots.txt behavior by type)

// last checked 11 Nov 2026. The example table is a layout, not a real log. Advice on which crawlers to allow is my opinion.