Google crawlers
This directory covers 11 bots that Google runs. Each has its own robots.txt token, so a site can allow one and disallow another. Last checked against Google's documentation on .
| Bot | Type | What it does | robots.txt token |
|---|---|---|---|
| AdsBot-Google | Other Google crawlers | Checks the quality of ad landing pages for Google Ads. | AdsBot-Google |
| Google-CloudVertexBot | Other Google crawlers | Crawls sites whose owners asked for it when building Vertex AI Agents. | Google-CloudVertexBot |
| Google-Extended | AI robots.txt control tokens | A robots.txt token that controls Gemini training and grounding use. | Google-Extended |
| Google-InspectionTool | Other Google crawlers | Fetches pages for Search Console URL inspection and the Rich Results Test. | Google-InspectionTool |
| Googlebot-Image | Search engine crawlers | Crawls images for Google Images and image features in Search. | Googlebot-Image |
| Googlebot-News | Search engine crawlers | Controls whether your pages appear in Google News. | Googlebot-News |
| Googlebot-Video | Search engine crawlers | Crawls video for the video features of Google Search. | Googlebot-Video |
| Googlebot | Search engine crawlers | Crawls pages for Google Search, Images, Video, News and Discover. | Googlebot |
| GoogleOther | Other Google crawlers | Fetches public pages for Google product teams, for example for research crawls. | GoogleOther |
| Mediapartners-Google | Other Google crawlers | Reads the pages of sites that show Google ads, so the ads match the content. | Mediapartners-Google |
| Storebot-Google | Other Google crawlers | Crawls product and store pages for Google Shopping. | Storebot-Google |
How to verify a request from Google
A user agent is a string any client can send, so the name alone proves nothing. Check an address from your logs, or apply the operator's own method:
- AdsBot-Google and Mediapartners-Google
- Run a reverse DNS lookup on the source IP and check that the name ends in google.com (Google uses rate-limited-proxy-….google.com for special-case crawlers), then run a forward lookup on that name and confirm it returns the same IP. Or check the IP against Google's special-case crawler ranges.https://developers.google.com/static/crawling/ipranges/special-crawlers.json
- Google-CloudVertexBot, Google-InspectionTool, Googlebot-Image, Googlebot-Video, Googlebot, GoogleOther and Storebot-Google
- Run a reverse DNS lookup on the source IP and check that the name ends in googlebot.com (Google uses crawl-….googlebot.com and geo-crawl-….geo.googlebot.com), then run a forward lookup on that name and confirm it returns the same IP. Or check the IP against Google's common crawler ranges. A googleusercontent.com name does not count: any Google Cloud virtual machine can have one.https://developers.google.com/static/crawling/ipranges/common-crawlers.json
- Google-Extended
- There is nothing to verify: no Google crawler sends Google-Extended as its user agent. A request that does is not from Google.
- Googlebot-News
- No request carries Googlebot-News as its user agent. The crawl comes from Googlebot, so verify it as Googlebot: run a reverse DNS lookup on the source IP, check that the name ends in googlebot.com, then run a forward lookup on that name and confirm it returns the same IP. Or check the IP against Google's common crawler ranges.https://developers.google.com/static/crawling/ipranges/common-crawlers.json
Sources
- https://developers.google.com/search/docs/crawling-indexing/google-special-case-crawlers
- https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers
- https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers
- https://developers.google.com/crawling/docs/crawlers-fetchers/google-special-case-crawlers