What is ClaudeBot?
ClaudeBot is Anthropic's training crawler. Anthropic says it collects web content that could contribute to training its generative AI models, and that restricting ClaudeBot signals that a site's future material should be excluded from training datasets.
Last verified 2026-09-30 against the operator's documentation. All bots
ClaudeBot at a glance
- Operator
- Anthropic
- Type
- AI training and dataset crawlers
- Purpose
- Collects web content that could contribute to Claude model training.
- robots.txt token
ClaudeBot- How to verify
- Check that the source IP is in the ranges Anthropic publishes for its crawlers.
- Operator documentation
- https://support.claude.com/en/articles/8896518
How often is ClaudeBot impersonated?
For the median organisation, 23% of requests claiming to be ClaudeBot failed verification.
Measured between 3 July 2026 and 28 September 2026 across organisations protected by Centinel. We only publish a share when at least five organisations each saw enough of these requests. Pooled across all of their requests, the share was 4%: a few large organisations carry most of the traffic, so the median organisation is the better guide to what yours will see.
This directory measures requests between 3 July 2026 and 28 September 2026. The fake crawler report covers its own, earlier window (late June to 21 September 2026), so its figures differ from the ones here. Read it for the method.
Have a request that claims to be ClaudeBot? Check its IP address.
Should you block ClaudeBot?
- Allow it if
- you are content for your pages to be used in Claude model training.
- Block it if
- you do not want your content used for training. Use Crawl-delay instead if the only issue is load.
- What blocking changes
- It signals that your future content should be excluded from Anthropic's training datasets. Claude-SearchBot and Claude-User are controlled separately.
To block it, add this group to your robots.txt. Anthropic says its bots honor robots.txt and support the non-standard Crawl-delay extension.
User-agent: ClaudeBot
Disallow: /robots.txt only asks. It does nothing against a client that ignores it or only pretends to be ClaudeBot. Stopping those takes verification at your edge.
ClaudeBot: common questions
See which bots reach your site, and which of them are who they claim to be.
What is ClaudeBot?
ClaudeBot is Anthropic's training crawler. Anthropic says it collects web content that could contribute to training its generative AI models, and that restricting ClaudeBot signals that a site's future material should be excluded from training datasets.
How do I verify that a request is really ClaudeBot?
Check that the source IP is in the ranges Anthropic publishes for its crawlers. The user agent alone proves nothing: any client can send it.
How do I block ClaudeBot in robots.txt?
Add a group for User-agent: ClaudeBot with Disallow: /. Anthropic says its bots honor robots.txt and support the non-standard Crawl-delay extension. robots.txt only asks. A client that ignores it, or pretends to be ClaudeBot, has to be stopped at your edge.
What happens if I block ClaudeBot?
It signals that your future content should be excluded from Anthropic's training datasets. Claude-SearchBot and Claude-User are controlled separately.
Related bots
- Claude-SearchBot (Anthropic): Indexes pages to improve Claude's search results.
- Claude-User (Anthropic): Fetches a page when a Claude user's question needs it.
- GPTBot (OpenAI): Crawls content that may be used to train OpenAI's foundation models.
- Meta-ExternalAgent (Meta): Crawls for training AI models or improving Meta products.
- Amazonbot (Amazon): Crawls to improve Amazon products; may be used to train Amazon AI models.
- CCBot (Common Crawl): Builds Common Crawl's open web archive, free for anyone to reuse.