The short answer
GPTBot, ClaudeBot, and Bytespider are labels used for AI crawler traffic, but the evidence available for each label is not equal.
- OpenAI documents GPTBot's purpose,
robots.txttoken, and published IP ranges. - Anthropic documents separate crawler tokens, including ClaudeBot, Claude-SearchBot, and Claude-User, and links to published IP ranges.
- Cloudflare lists Bytespider as a ByteDance AI crawler. A ByteDance operator page was not available in this source pack, so treat the label as third-party documentation rather than an operator-verified identity.
A user-agent string is not an identity check. It is a claim made by the requester. Use the claim to select a verification method, then decide what that verified or unverified requester may access.
Compare the published evidence
This table describes the documentation available on September 4, 2026. It does not claim that every request using one of these names is genuine.
| Claimed crawler | Published purpose | Published identity evidence | robots.txt documentation | Source confidence |
|---|---|---|---|---|
| GPTBot | OpenAI says GPTBot crawls content that may be used to train its generative AI foundation models. | OpenAI publishes a current JSON list of GPTBot IP ranges. | OpenAI documents the GPTBot product token and Disallow rules. | Operator-primary |
| ClaudeBot | Anthropic describes ClaudeBot as a crawler used for model development. | Anthropic links to a JSON file containing IP addresses used by its crawlers. | Anthropic documents ClaudeBot separately from Claude-SearchBot and Claude-User. | Operator-primary |
| Bytespider | Cloudflare identifies Bytespider as an AI crawler operated by ByteDance. | No operator-primary identity source was found for this source pack. | No operator-primary guidance was found for this source pack. | Third-party catalogue; verify independently |
Sources: OpenAI crawler documentation, OpenAI's GPTBot IP list, Anthropic crawler documentation, and Cloudflare's AI crawler catalogue.
Keep purpose, identity, and policy separate
These are different questions:
- Purpose: What does the operator say the crawler does?
- Identity: What evidence connects this request to that operator?
- Policy: What may the requester access once you have evaluated the evidence?
Mixing them creates brittle rules. A documented purpose does not prove the current request is genuine. A genuine crawler identity does not automatically grant access to every route. An unverified request does not automatically require a site-wide block.
Centinel's guide to verifying AI agents covers the verification sequence in more detail.
Verify a GPTBot claim
OpenAI publishes separate tokens for different products. GPTBot is documented for training-related crawling. OAI-SearchBot supports search, and ChatGPT-User represents some user-initiated requests. Do not collapse those purposes into one rule just because they share an operator.
For a request claiming to be GPTBot:
- Record the claimed user-agent and source IP.
- Fetch OpenAI's current GPTBot IP list through a controlled, cached process.
- Check whether the source IP belongs to a published range.
- Apply the policy for the requested resource, not a blanket operator decision.
Published IP ranges can change. Bound the refresh interval, retain the last known-good list, and define failure behavior before depending on a remote file in enforcement.
On the median site in our crawler verification data, 84% of requests claiming to be GPTBot failed OpenAI's published check. See the GPTBot user agent and IP ranges.
Verify a ClaudeBot claim
Anthropic documents several crawler identities with different purposes. ClaudeBot is not interchangeable with Claude-SearchBot or Claude-User. A rule intended for model-development crawling should not silently govern search indexing or user-directed retrieval.
Anthropic links to an IP-address list for its crawlers. Use that operator-primary source as one part of verification, with the same operational safeguards used for any remote allowlist: a bounded refresh, validation of the downloaded data, and a deliberate response when the source is unavailable. To check one address from your logs by hand, verify a ClaudeBot or GPTBot IP against the operators' published lists.
Anthropic also documents robots.txt directives for its crawler tokens. Those directives express a preference to a conforming crawler; they do not authenticate the request at your edge. See the ClaudeBot user agent and IP ranges.
Treat a Bytespider claim as unresolved until verified
Cloudflare's crawler catalogue associates Bytespider with ByteDance. That is useful classification evidence, but it is not operator-primary identity documentation.
Because a ByteDance operator page was not in the source pack, this article does not prescribe an IP-range check or claim a universal robots.txt behavior for Bytespider. Start with observation:
- record the user-agent claim and source network data;
- examine request paths, cadence, protocol characteristics, and session behavior;
- check for a current operator source before creating a trusted allow rule;
- choose a response based on resource sensitivity and your tolerance for unresolved identity.
This is deliberately less tidy than a one-line allow-or-block answer. The available evidence does not justify pretending the identity question is settled. The Bytespider bot profile lists what is documented about it.
What robots.txt can and cannot do
The Robots Exclusion Protocol defines how a crawler can discover and interpret site-owner rules. RFC 9309 standardizes that protocol. It is a coordination mechanism, not an authentication or authorization layer.
A robots.txt rule can state that a named product token should not crawl a path. It cannot prove that a request carrying that name belongs to the stated operator, and it cannot force a non-conforming client to comply.
Use robots.txt when you want to communicate crawler preferences. Use request-time controls when access must be enforced.
Turn the evidence into a resource policy
| Evidence state | Low-sensitivity public resource | Sensitive or costly resource |
|---|---|---|
| Operator claim verified | Apply the policy assigned to that crawler and purpose. | Require the explicit rule for that resource; do not inherit a broad allow. |
| Named crawler, identity unresolved | Observe, rate-limit, challenge, or deny according to risk. | Default to the bounded failure behavior your team approved. |
| Claim conflicts with published evidence | Treat the name as spoofable input and evaluate the remaining signals. | Do not grant trusted access from the user-agent claim. |
Write the decision before implementing the rule. The AI-agent access policy guide provides a practical template.
Where Centinel fits
Centinel helps collect browser, network, session, and crawler evidence so teams can distinguish a declared name from a better-supported identity. It can then contribute that evidence to route-specific allow, observe, challenge, rate-limit, or deny decisions.
The exact enforcement options depend on the integration point. A browser-side deployment does not control every non-browser request, and a server or edge integration should be reviewed against the traffic path it is expected to cover. See the bot management overview for the product boundary and evaluation checklist.
The durable rule is simple: document the purpose, verify what you can, preserve uncertainty where you cannot, and apply policy per resource.