AI crawler list
AI companies send different bots for different jobs: training models, indexing pages for AI search, and fetching a page when a user asks. Each has its own robots.txt token, so you can allow one and refuse another. Facts come from each operator's own documentation, last verified .
AI training and dataset crawlers
Crawl the web for content that may be used to train models or to build a reusable dataset.
| Bot | Operator | What it does | robots.txt token |
|---|---|---|---|
| Amazonbot | Amazon | Crawls to improve Amazon products; may be used to train Amazon AI models. | Amazonbot |
| CCBot | Common Crawl | Builds Common Crawl's open web archive, free for anyone to reuse. | CCBot |
| ClaudeBot | Anthropic | Collects web content that could contribute to Claude model training. | ClaudeBot |
| GPTBot | OpenAI | Crawls content that may be used to train OpenAI's foundation models. | GPTBot |
| Meta-ExternalAgent | Meta | Crawls for training AI models or improving Meta products. | meta-externalagent |
AI search crawlers
Index pages so an AI assistant can find, cite and link them in its answers.
| Bot | Operator | What it does | robots.txt token |
|---|---|---|---|
| Claude-SearchBot | Anthropic | Indexes pages to improve Claude's search results. | Claude-SearchBot |
| DuckAssistBot | DuckDuckGo | Crawls in real time to support DuckDuckGo's AI-assisted answers. | DuckAssistBot |
| Meta-WebIndexer | Meta | Crawls pages to improve the search results of Meta AI. | meta-webindexer |
| MistralAI-Index | Mistral AI | Indexes pages for Mistral search. | MistralAI-Index |
| OAI-SearchBot | OpenAI | Surfaces websites in ChatGPT's search features. | OAI-SearchBot |
| PerplexityBot | Perplexity | Surfaces and links websites in Perplexity's search results. | PerplexityBot |
AI user-triggered fetchers
Fetch a page because a person asked an assistant to. Several operators say robots.txt may not apply.
| Bot | Operator | What it does | robots.txt token |
|---|---|---|---|
| ChatGPT-User | OpenAI | Visits a page when a ChatGPT or Custom GPT user's request needs it. | ChatGPT-User |
| Claude-User | Anthropic | Fetches a page when a Claude user's question needs it. | Claude-User |
| Meta-ExternalFetcher | Meta | Fetches individual links on behalf of Meta product users. | meta-externalfetcher |
| MistralAI-User | Mistral AI | Fetches a page when a Mistral user's action needs it. | MistralAI-User |
| Perplexity-User | Perplexity | Fetches a page when a Perplexity user's request needs it. | Perplexity-User |
AI robots.txt control tokens
Names you write in robots.txt to control AI use. No crawler sends them as a user agent.
| Bot | Operator | What it does | robots.txt token |
|---|---|---|---|
| Applebot-Extended | Apple | A robots.txt token that opts content out of Apple model training. | Applebot-Extended |
| Google-Extended | A robots.txt token that controls Gemini training and grounding use. | Google-Extended |
Opt out of AI training, stay in AI search
This group refuses the training crawlers and control tokens above and leaves the AI search crawlers alone. Check each bot's page before you copy it: some names also feed products you may want.
User-agent: Amazonbot
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: GPTBot
User-agent: meta-externalagent
Disallow: /robots.txt only asks. Several user-triggered fetchers say they may not follow it, and anyone can send a crawler name. Verify a crawler or browse the full bot directory.