AI crawler list
AI companies run different bots for different jobs: training models, indexing pages for AI search, and fetching a page when a person asks an assistant. Each has its own robots.txt token, so you can allow one and disallow another. Each fact comes from the operator's own documentation, last verified .
AI training and dataset crawlers
Crawl the web for content that may be used to train models or to build a reusable dataset.
| Bot | Operator | What it does | robots.txt token |
|---|---|---|---|
| Amazonbot | Amazon | Crawls to improve Amazon products; may be used to train Amazon AI models. | Amazonbot |
| CCBot | Common Crawl | Builds Common Crawl's open web archive, free for anyone to reuse. | CCBot |
| ClaudeBot | Anthropic | Collects web content that could contribute to Claude model training. | ClaudeBot |
| GPTBot | OpenAI | Crawls content that may be used to train OpenAI's foundation models. | GPTBot |
| Meta-ExternalAgent | Meta | Crawls for training AI models or improving Meta products. | meta-externalagent |
AI search crawlers
Index pages so an AI assistant can find, cite and link them in its answers.
| Bot | Operator | What it does | robots.txt token |
|---|---|---|---|
| Claude-SearchBot | Anthropic | Indexes pages to improve Claude's search results. | Claude-SearchBot |
| DuckAssistBot | DuckDuckGo | Crawls in real time to support DuckDuckGo's AI-assisted answers. | DuckAssistBot |
| Meta-WebIndexer | Meta | Crawls pages to improve the search results of Meta AI. | meta-webindexer |
| MistralAI-Index | Mistral AI | Indexes pages for Mistral search. | MistralAI-Index |
| OAI-SearchBot | OpenAI | Surfaces websites in ChatGPT's search features. | OAI-SearchBot |
| PerplexityBot | Perplexity | Surfaces and links websites in Perplexity's search results. | PerplexityBot |
AI user-triggered fetchers
Fetch a page because a person asked an assistant to. Several operators say robots.txt may not apply.
| Bot | Operator | What it does | robots.txt token |
|---|---|---|---|
| ChatGPT-User | OpenAI | Visits a page when a ChatGPT or Custom GPT user's request needs it. | ChatGPT-User |
| Claude-User | Anthropic | Fetches a page when a Claude user's question needs it. | Claude-User |
| Meta-ExternalFetcher | Meta | Fetches individual links on behalf of Meta product users. | meta-externalfetcher |
| MistralAI-User | Mistral AI | Fetches a page when a Mistral user's action needs it. | MistralAI-User |
| Perplexity-User | Perplexity | Fetches a page when a Perplexity user's request needs it. | Perplexity-User |
AI robots.txt control tokens
Names you write in robots.txt to control how an operator uses your content for AI. No crawler sends them as a user agent.
| Bot | Operator | What it does | robots.txt token |
|---|---|---|---|
| Applebot-Extended | Apple | A robots.txt token that opts content out of Apple model training. | Applebot-Extended |
| Google-Extended | A robots.txt token that controls Gemini training and grounding use. | Google-Extended |
Opt out of AI training, stay in AI search
This group disallows the training crawlers and control tokens above. It does not mention the AI search crawlers. Check each bot's page before you copy it: some names also feed products you may want.
User-agent: Amazonbot
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: GPTBot
User-agent: meta-externalagent
Disallow: /robots.txt only asks. Several user-triggered fetchers say they may not follow it, and anyone can send a crawler name. Verify a crawler or browse the full bot directory.