What is ClaudeBot? Anthropic's Crawler Explained

A security lead pulls a week of access logs and finds 40,000 requests with ClaudeBot in the user agent. The robots.txt file says nothing about Anthropic. The origin servers are slower on category pages. Someone asks the obvious question: is this really Anthropic, and should we let it in?

That question has two halves, and most teams only ask the first. The first half is identity: did this request come from Anthropic at all? The second is purpose. Anthropic runs more than one Claude user agent, and the one in your logs may not be the one you meant to allow or block.

This article answers what ClaudeBot is in plain terms, separates it from its two siblings, and walks through a verification and policy approach you can apply this week. It also explains where general detection methods such as JA4 fingerprints and residential proxy checks help, and where they stop. Those are general techniques. They are not ClaudeBot behavior, and this post does not claim otherwise.

ClaudeBot in 50 words: what it is and what it is not

ClaudeBot is Anthropic's web crawler for collecting public web content that can be used to train Claude models. It announces itself with a user agent token, and Anthropic says it honors robots.txt. It is not a search index crawler, not a live browsing session, and not proof of anything on its own.

That last point matters most. A name in a header is a claim. It becomes a fact only after you check it against something the requester cannot easily fake.

The second thing to settle is scope. ClaudeBot is the training-data crawler. Blocking it in robots.txt tells Anthropic you do not want your pages collected for model training. It says nothing about whether Claude can fetch a page when a user pastes in a link, or whether your pages show up in Claude's search results. Those are different agents with different rules, covered next.

Treat ClaudeBot as a policy decision about training, not as a generic "Anthropic traffic" bucket. Teams that lump everything Claude-related into one rule usually end up either blocking more than they intended or allowing more than they realized.

If you want the wider context of how publishers are drawing these lines, the AI crawler policy for publishers template is a good companion.

ClaudeBot, Claude-SearchBot and Claude-User: three agents, three purposes

Anthropic documents three separate user agents on its crawler help page. Each one exists for a different reason, and each can be controlled separately in robots.txt.

Read the table as three policy levers, not one. A publisher that wants no training use but still wants to appear in AI search answers will treat ClaudeBot and Claude-SearchBot differently. An ecommerce team worried about price scraping may care most about Claude-User, because a person's request triggers it rather than a scheduled crawl.

Here is the position I would defend: decide per agent, and write the reason next to each rule. "Block ClaudeBot: no training use" is a defensible line. "Block all Claude" is a shrug you will revisit after the first angry email from your SEO team.

Verify the current token names and behavior against Anthropic's page before you ship a rule, because vendors rename and add agents. The same caution appears in our look at AI browser agents, where one vendor withdrew three agent products. Anything on this list can change.

How to verify a request really is ClaudeBot

Verification means checking the claim against evidence the claimant does not control. Work through it in this order.

1. Read Anthropic's current documentation. Check whether Anthropic publishes IP ranges or another verification method for ClaudeBot, and use what it publishes. If a published list exists, it is your strongest signal. If it does not, say so in your runbook rather than inventing a method.
2. Establish the real client address. Behind a CDN or load balancer, the peer your server sees is the proxy, not the crawler. Trust forwarded headers only from proxies you control. Get this wrong and every check below runs against the wrong IP.
3. Compare the source address to the published ranges. A match supports the claim. A mismatch on a request that says ClaudeBot is a strong sign of impersonation.
4. Check behavior. Real crawlers fetch robots.txt, follow it, and show steady patterns. Impersonators often skip robots.txt, hit login or checkout paths, or arrive in bursts from unrelated networks.
5. Log the verdict with its evidence. Record which check passed or failed, plus the time of the lookup. IP range lists change.

The same discipline applies to any crawler claim. Our write-up on the fake Googlebot problem walks through reverse DNS and range checks for Google, and the logic transfers: claim first, then independent proof.

Keep an explicit "unverified" state. When you cannot confirm and cannot refute, do not round up to "real" or down to "fake". Route unverified traffic to a challenge or a rate limit, and revisit the verdict when you have more evidence.

Why a user agent string alone proves nothing

A user agent is a text header the client chooses. Anyone can send ClaudeBot in it with one line of code. A scraper operator who wants to slip past an allowlist has every reason to try, because many sites wave named AI crawlers through.

This is not hypothetical. Centinel's published research found that on a typical news site, 84% of requests claiming to be GPTBot were fake. That figure describes GPTBot, not ClaudeBot, and nobody should quote it as a ClaudeBot number. The lesson carries over anyway: wherever a trusted name earns access, the name gets forged.

Think about the incentive. If your policy says "allow ClaudeBot, challenge unknown bots", the cheapest attack is to rename your unknown bot. A rule keyed to the string rewards exactly the traffic you wanted to stop.

So the user agent has one legitimate job: it tells you which verification to run. "Claims ClaudeBot" routes the request to the ClaudeBot check. It never replaces the check.

Two practical habits help. First, log claimed identity and verified identity as separate fields, so you can count the gap between them. That gap is your impersonation rate, and it is the number that justifies enforcement budget. Second, never write an allow rule on a string alone. If a rule says "allow if user agent contains ClaudeBot" with no second condition, delete it today.

What JA4 and residential proxy detection can and cannot tell you

These two methods get pitched as magic. They are useful, and they have limits. Neither is a ClaudeBot-specific signal, and nothing here describes how Anthropic's crawler behaves at the network level.

JA4 fingerprints. JA4 is a method for fingerprinting the TLS client handshake, published by FoxIO. It lets you group connections by how the client software negotiates TLS. Used well, it exposes a mismatch: a request that claims to be one kind of client but handshakes like another. It cannot tell you that a handshake belongs to Anthropic, because you would need a trusted reference sample, and a fingerprint can also change with a library update or a browser release. Treat a JA4 mismatch as one observation, not a verdict. Impersonation libraries also exist specifically to copy handshakes.

Residential proxy detection. A residential proxy sends traffic out through an address on a consumer connection. IP reputation then looks clean, because the address belongs to an ordinary home. Classifying an address as residential tells you the last hop. It does not tell you who chose the URL, and it can sweep in a bystander whose device is relaying traffic without their knowledge. Our guide on how to detect bots using residential proxies covers testing this properly.

The working rule: a fake ClaudeBot request behind a residential address is the case where range checks matter most. If the address is not in Anthropic's published set, the residential label is irrelevant. The claim already failed.

Combine independent signals, enforce on the protected route, and keep the unknown state. No single fingerprint earns a block.

How DataDome and Cloudflare Bot Management approach the problem differently

Both vendors are established, and they take different shapes. The differences are worth understanding before you decide where verification should live.

Cloudflare Bot Management is part of the Cloudflare platform. It scores requests and offers a verified bot concept, and its AI crawler controls sit inside the same dashboard and plan tiers as the rest of its edge services. If your traffic already runs through Cloudflare, that integration is the draw. The tradeoff is that the controls are tied to that platform and its plan structure. Our Cloudflare pricing checklist lists the questions to put in an enterprise quote.

DataDome is a dedicated bot protection vendor that you integrate alongside your existing stack, and it publishes extensive material on bot categories and detection. It is a separate product with its own integration and its own policy layer.

For your decision, the useful comparison is not which vendor is "better". Ask these questions of any option, including Centinel:

• Does it verify named crawlers against published evidence, or trust the string?
• Can you set policy per agent, so ClaudeBot and Claude-User get different treatment?
• Does it work with the CDN you already run, or require you to move?
• What does it do with traffic it cannot classify?
• Can you roll back a rule quickly?

Our best bot management software comparison puts ten tools side by side on criteria like these. Run your own two-week test on a real route before you sign anything.

Setting a policy: robots.txt, enforcement and where Centinel fits

Start with robots.txt, then admit its limit. A robots.txt file is a request. Anthropic says ClaudeBot honors it, and per-agent rules look like this:

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

That example opts out of training collection while leaving search indexing and user-triggered fetches open. Change it to match your own decision. Confirm the exact tokens on Anthropic's page first.

Now the limit. robots.txt binds only the crawlers that choose to read it. A scraper wearing a ClaudeBot costume ignores it by definition. Policy without enforcement is a sign on an unlocked door.

Enforcement is where Centinel Analytica fits. Centinel tells real visitors from scrapers and applies allow, challenge or block policy through your existing CDN or edge. The practical effect for this topic: a request that claims to be ClaudeBot gets judged on more than its header.

A sensible rollout looks like this:

1. Write the per-agent decision and its reason.
2. Publish the matching robots.txt.
3. Log claimed versus verified identity.
4. Run challenge mode on unverified claimants before you block.
5. Test with real visitors and approved monitors, using a method like bot false positive testing.

Blocking more sessions is not success if legitimate users can no longer finish their task.

Conclusion

Settle three things this week. Decide per agent whether ClaudeBot, Claude-SearchBot and Claude-User each get allowed, challenged or blocked, and write the reason beside each rule. Delete any allow rule that depends on a user agent string alone. Then measure the gap between requests that claim to be ClaudeBot and requests you can actually verify.

My prediction: that gap will be larger than your team expects, and it will be the strongest argument you have for enforcement over a robots.txt line.

Or book the 30-minute demo to walk through allow, challenge and block policy on top of the CDN you already run.

Frequently asked questions

What is ClaudeBot and does it respect robots.txt?

ClaudeBot is Anthropic's web crawler for collecting public content that may be used to train Claude models. Anthropic states that it honors robots.txt, so a Disallow rule for the ClaudeBot token opts you out. Confirm the current token on Anthropic's crawler page. Impersonators ignore robots.txt, which is why enforcement matters.

What is the difference between ClaudeBot, Claude-SearchBot and Claude-User?

They are three separate Anthropic user agents. ClaudeBot collects content for model training. Claude-SearchBot indexes content for Claude's search results. Claude-User fetches a page when a person asks Claude to open a URL. Each can be controlled separately, so decide a policy per agent rather than blocking all Claude traffic.

How do I tell if a request claiming to be ClaudeBot is fake?

Do not trust the user agent. Establish the real client IP behind your proxies, then compare it with any ranges Anthropic publishes, and check behavior such as whether robots.txt is fetched. Centinel Analytica tells real visitors from scrapers, then applies allow, challenge or block through your existing CDN. Keep an unverified state for unclear cases.

Should I block ClaudeBot?

It depends on your goal. Block it if you do not want your content used for model training. Leave Claude-SearchBot and Claude-User open if you want to appear in Claude search answers and user-requested fetches. Write the reason beside each rule, and test with a challenge before hard blocks.

Can JA4 fingerprints identify ClaudeBot?

Not on their own. JA4 can help highlight potential inconsistencies in incoming requests. It cannot prove a handshake belongs to Anthropic without a trusted reference, and fingerprints shift with updates. Use it as one observation alongside published range checks and behavior.