Bot directory
55 crawlers and fetchers that reach public websites: who runs each one, what it does, and its robots.txt token. Each record cites the operator's own documentation where the operator publishes any. Last checked . Have an address from your logs? Check whether it is the crawler it claims to be.
55 bots.
AI training and dataset crawlers
Crawl the web for content that may be used to train models or to build a reusable dataset.
| Bot | Operator | What it does | robots.txt token |
|---|---|---|---|
| Amazonbot | Amazon | Crawls to improve Amazon products; may be used to train Amazon AI models. | Amazonbot |
| CCBot | Common Crawl | Builds Common Crawl's open web archive, free for anyone to reuse. | CCBot |
| ClaudeBot | Anthropic | Collects web content that could contribute to Claude model training. | ClaudeBot |
| GPTBot | OpenAI | Crawls content that may be used to train OpenAI's foundation models. | GPTBot |
| Meta-ExternalAgent | Meta | Crawls for training AI models or improving Meta products. | meta-externalagent |
AI search crawlers
Index pages so an AI assistant can find, cite and link them in its answers.
| Bot | Operator | What it does | robots.txt token |
|---|---|---|---|
| Claude-SearchBot | Anthropic | Indexes pages to improve Claude's search results. | Claude-SearchBot |
| DuckAssistBot | DuckDuckGo | Crawls in real time to support DuckDuckGo's AI-assisted answers. | DuckAssistBot |
| Meta-WebIndexer | Meta | Crawls pages to improve the search results of Meta AI. | meta-webindexer |
| MistralAI-Index | Mistral AI | Indexes pages for Mistral search. | MistralAI-Index |
| OAI-SearchBot | OpenAI | Surfaces websites in ChatGPT's search features. | OAI-SearchBot |
| PerplexityBot | Perplexity | Surfaces and links websites in Perplexity's search results. | PerplexityBot |
AI user-triggered fetchers
Fetch a page because a person asked an assistant to. Several operators say robots.txt may not apply.
| Bot | Operator | What it does | robots.txt token |
|---|---|---|---|
| ChatGPT-User | OpenAI | Visits a page when a ChatGPT or Custom GPT user's request needs it. | ChatGPT-User |
| Claude-User | Anthropic | Fetches a page when a Claude user's question needs it. | Claude-User |
| Meta-ExternalFetcher | Meta | Fetches individual links on behalf of Meta product users. | meta-externalfetcher |
| MistralAI-User | Mistral AI | Fetches a page when a Mistral user's action needs it. | MistralAI-User |
| Perplexity-User | Perplexity | Fetches a page when a Perplexity user's request needs it. | Perplexity-User |
AI robots.txt control tokens
Names you write in robots.txt to control AI use. No crawler sends them as a user agent.
| Bot | Operator | What it does | robots.txt token |
|---|---|---|---|
| Applebot-Extended | Apple | A robots.txt token that opts content out of Apple model training. | Applebot-Extended |
| Google-Extended | A robots.txt token that controls Gemini training and grounding use. | Google-Extended |
Search engine crawlers
Build the indexes behind web search results.
| Bot | Operator | What it does | robots.txt token |
|---|---|---|---|
| 360Spider | 360 Search | Crawls pages for the web search of 360 Search (so.com). | 360Spider |
| Applebot | Apple | Crawls pages for Spotlight, Siri and Safari search features. | Applebot |
| Baiduspider | Baidu | Crawls pages for Baidu search. | Baiduspider |
| Bingbot | Microsoft | Crawls pages for Bing search. | bingbot |
| Diffbot | Diffbot | Crawls the web for the Diffbot Knowledge Graph and Diffbot's web search services. | Diffbot |
| DuckDuckBot | DuckDuckGo | Crawls pages for DuckDuckGo search. | DuckDuckBot |
| Googlebot-Image | Crawls images for Google Images and image features in Search. | Googlebot-Image | |
| Googlebot-News | Controls whether your pages appear in Google News. | Googlebot-News | |
| Googlebot-Video | Crawls video for the video features of Google Search. | Googlebot-Video | |
| Googlebot | Crawls pages for Google Search, Images, Video, News and Discover. | Googlebot | |
| Kagibot | Kagi | Crawls pages for the Kagi search engine. | Kagibot |
| PetalBot | Huawei | Crawls pages for Petal Search and Huawei Assistant. | PetalBot |
| SeznamBot | Seznam.cz | Crawls pages for Seznam, a Czech search engine. | SeznamBot |
| YandexBot | Yandex | Yandex's main search indexing crawler. | YandexBot |
| Yeti | Naver | Crawls pages for Naver search in South Korea. | Yeti |
| YouBot | You.com | Crawls and indexes pages for the You.com search engine. | YouBot |
Other Google crawlers
Single-purpose Google crawlers for shopping, ads, testing tools and Vertex AI.
| Bot | Operator | What it does | robots.txt token |
|---|---|---|---|
| AdsBot-Google | Checks the quality of ad landing pages for Google Ads. | AdsBot-Google | |
| Google-CloudVertexBot | Crawls sites whose owners asked for it when building Vertex AI Agents. | Google-CloudVertexBot | |
| Google-InspectionTool | Fetches pages for Search Console URL inspection and the Rich Results Test. | Google-InspectionTool | |
| GoogleOther | Fetches public pages for Google product teams, for example for research crawls. | GoogleOther | |
| Mediapartners-Google | Reads the pages of sites that show Google ads, so the ads match the content. | Mediapartners-Google | |
| Storebot-Google | Crawls product and store pages for Google Shopping. | Storebot-Google |
SEO and link-index crawlers
Build backlink and site-audit databases for SEO tools. None of them affects search rankings.
| Bot | Operator | What it does | robots.txt token |
|---|---|---|---|
| AhrefsBot | Ahrefs | Builds the link database behind Ahrefs and the Yep search engine. | AhrefsBot |
| AhrefsSiteAudit | Ahrefs | Crawls a site for an Ahrefs Site Audit project. | AhrefsSiteAudit |
| Barkrowler | Babbar | Collects links and page metadata for Babbar. | barkrowler |
| DataForSeoBot | DataForSEO | Adds links to DataForSEO's backlink database. | DataForSeoBot |
| DotBot | Moz | Gathers data for the Moz Link Index. | dotbot |
| MJ12bot | Majestic | Maps links between websites for Majestic's link index. | MJ12bot |
| Rogerbot | Moz | Crawls a site for the site audits of a Moz Pro Campaign. | rogerbot |
| Screaming Frog SEO Spider | Screaming Frog | Audits a site's technical SEO from the machine that runs it. | Screaming Frog SEO Spider |
| SemrushBot | Semrush | Collects web data for Semrush's SEO tools. | SemrushBot |
Link preview bots
Fetch a page when someone shares its link, to build the preview card.
| Bot | Operator | What it does | robots.txt token |
|---|---|---|---|
| Discordbot | Discord | Visits a link shared in a Discord message to build its preview. | Discordbot |
| facebookexternalhit | Meta | Builds link previews when content is shared on Meta's apps. | facebookexternalhit |
| Pinterestbot | Indexes pages behind Pins and keeps Pin details up to date. | Pinterestbot | |
| Slackbot-LinkExpanding | Slack | Fetches a page's meta tags when someone posts its link in Slack. | Slackbot-LinkExpanding |
| Twitterbot | X | Fetches card markup when a link is shared on X. | Twitterbot |
Crawlers without operator documentation
The operator publishes no page describing the crawler, so its identity cannot be checked.
| Bot | Operator | What it does | robots.txt token |
|---|---|---|---|
| Bytespider | Unconfirmed, attributed to ByteDance | A crawler with no operator documentation. | Bytespider |
Bots by operator
An operator that runs several bots gives each one its own robots.txt token. These pages put one operator's bots side by side.
- Google (11)
- OpenAI (3)
- Anthropic (3)
- Meta (4)
- AI crawlers only