Anthropic crawlers
This directory covers 3 bots that Anthropic runs. Each has its own robots.txt token, so a site can allow one and disallow another. Last checked against Anthropic's documentation on .
| Bot | Type | What it does | robots.txt token |
|---|---|---|---|
| Claude-SearchBot | AI search crawlers | Indexes pages to improve Claude's search results. | Claude-SearchBot |
| Claude-User | AI user-triggered fetchers | Fetches a page when a Claude user's question needs it. | Claude-User |
| ClaudeBot | AI training and dataset crawlers | Collects web content that could contribute to Claude model training. | ClaudeBot |
How to verify a request from Anthropic
A user agent is a string any client can send, so the name alone proves nothing. Check an address from your logs, or apply the operator's own method:
- Claude-SearchBot, Claude-User and ClaudeBot
- Check that the source IP is in the ranges Anthropic publishes for its crawlers.https://claude.com/crawling/bots.json