Bot user agents list
Googlebot uses Googlebot/2.1 inside a Chrome-style user agent, and most other search and AI crawlers follow the same pattern. The table lists the exact strings each operator publishes, what the bot does and how to verify it. A user agent alone proves nothing: always confirm by IP or DNS.
Bot user agents list
Every string below is copied from the operator's own documentation (link in the last column), checked 2026-10-27. W.X.Y.Z stands for the current browser version, which changes over time, so match it with a wildcard in your rules and log searches.
Search engine crawlers
| Bot | User agent | What it does | How to verify | Docs |
|---|---|---|---|---|
| Googlebot Smartphone | Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) | Main Google Search crawler (mobile-first indexing) | Reverse DNS + IP list IP list | source |
| Googlebot Desktop | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36 | Google Search crawler, desktop | Reverse DNS + IP list IP list | source |
| Googlebot-Image | Googlebot-Image/1.0 | Images for Google Images and Search | Reverse DNS + IP list IP list | source |
| Googlebot-Video | Googlebot-Video/1.0 | Videos for Google Search | Reverse DNS + IP list IP list | source |
| Bingbot (desktop) Microsoft | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/W.X.Y.Z Safari/537.36 | Bing search crawler, desktop | Reverse DNS + IP list IP list | source |
| Bingbot (mobile) Microsoft | Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) | Bing search crawler, mobile | Reverse DNS + IP list IP list | source |
| Applebot Apple | Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) ... (Applebot/0.1; +http://www.apple.com/go/applebot) (token in the user agent) | Search features in Siri, Spotlight and Safari | Reverse DNS + IP list IP list | source |
Ad crawlers
| Bot | User agent | What it does | How to verify | Docs |
|---|---|---|---|---|
| AdsBot-Google | AdsBot-Google (+http://www.google.com/adsbot.html) | Checks ad landing page quality, desktop. Ignores robots.txt rules for user agent * | Reverse DNS + IP list IP list | source |
| AdsBot-Google-Mobile | Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; AdsBot-Google-Mobile; +http://www.google.com/mobile/adsbot.html) | Checks ad landing page quality, mobile. Ignores robots.txt rules for user agent * | Reverse DNS + IP list IP list | source |
AI crawlers and assistants
| Bot | User agent | What it does | How to verify | Docs |
|---|---|---|---|---|
| Google-Extended | Google-Extended (robots.txt token) | robots.txt token only, no separate HTTP user agent. Controls use of content for training Gemini models | No reliable method - | source |
| GPTBot OpenAI | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot | Crawls content that may be used to train OpenAI models | Published IP list IP list | source |
| OAI-SearchBot OpenAI | Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot | Surfaces websites in ChatGPT search | Published IP list IP list | source |
| ChatGPT-User OpenAI | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot | Visits a page when a ChatGPT user asks. Not an automatic crawler | Published IP list IP list | source |
| ClaudeBot Anthropic | ClaudeBot (token in the user agent) | Collects content that could be used to train Anthropic models. Full user agent not published | Published IP list IP list | source |
| Claude-User Anthropic | Claude-User (token in the user agent) | Visits a page when a Claude user asks. Full user agent not published | Published IP list IP list | source |
| Claude-SearchBot Anthropic | Claude-SearchBot (token in the user agent) | Improves Claude search results. Full user agent not published | Published IP list IP list | source |
| PerplexityBot Perplexity | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot) | Surfaces and links websites in Perplexity search. Not used for model training | Published IP list IP list | source |
| Perplexity-User Perplexity | Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user) | Visits a page when a Perplexity user asks. Generally ignores robots.txt | Published IP list IP list | source |
| Applebot-Extended Apple | Applebot-Extended (robots.txt token) | robots.txt token only, does not crawl. Controls use of Applebot data for Apple model training | No reliable method - | source |
SEO tool crawlers
| Bot | User agent | What it does | How to verify | Docs |
|---|---|---|---|---|
| AhrefsBot Ahrefs | Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/) | Backlink and SEO index crawler | Reverse DNS + IP list IP list | source |
| SemrushBot Semrush | SemrushBot (token in the user agent) | SEO index crawler. Semrush does not publish IP blocks or a full user agent string | No reliable method - | source |
How to verify a bot
Search engines publish two ways to prove a request is theirs. The first is a reverse DNS lookup that resolves back to the same IP. The second is a published list of IP ranges. AI companies mostly publish only IP lists. Our Verify Googlebot tool checks Google and Bing IPs both ways, and the IP ranges page has the lists to download.
Tokens such as Google-Extended and Applebot-Extended never appear in your logs. They only work in robots.txt, where they tell Google and Apple whether crawled content may be used for AI training.
How to allow them in your firewall
Rules that match on the user agent are easy to bypass, because a scraper can send any of the strings above. If your site is behind Cloudflare, use its known bots field instead, which Cloudflare sets only after verifying the bot:
(cf.client.bot)Put a Skip rule with this expression above your blocking rules. To allow only search engines, the verified bot category field narrows it down: cf.verified_bot_category eq "Search Engine Crawler". Search crawlers stay allowed, and anything that only pretends to be Googlebot still hits your filters. Deciding what to do with AI crawlers is a separate question, covered in which AI crawlers to block.
FAQ
What is the Googlebot user agent?
Googlebot Smartphone, the crawler that does most of the work since mobile-first indexing, sends a Chrome on Android user agent with (compatible; Googlebot/2.1; +http://www.google.com/bot.html) at the end. The desktop crawler uses the same token in a desktop Chrome string.
Why does the Chrome version in Googlebot's user agent change?
Googlebot renders pages with a current version of Chromium, and the user agent reports that version. Google shows it as W.X.Y.Z in its documentation for this reason.
Do AI assistants like ChatGPT-User follow robots.txt?
Not always. OpenAI and Perplexity say their user-triggered agents may not apply robots.txt rules, because a person asked for the page. Their crawlers for search and training are the ones you control through robots.txt.
Can I trust a bot that uses the Googlebot user agent?
No. Check its IP against Google's published ranges or run a reverse and forward DNS lookup. Many scrapers copy the Googlebot user agent for exactly this reason.