To block an AI crawler, first decide whether you want to state a crawl
preference or deny the HTTP request. Use robots.txt for documented crawlers
that honor the Robots Exclusion Protocol. Use a server, application, or edge
rule when the request itself must be stopped.
There is no universal "block AI" switch. A rule that matches GPTBot will not
cover a scraper pretending to be Chrome. A broad browser challenge may stop
some automation while also interrupting readers. The right control depends on
the client, the resource, and the cost of a wrong decision.
Choose the control by what you need to enforce
Start with the outcome, not the product feature.
| Goal | First control to consider | What it does not solve |
|---|---|---|
| Tell a documented crawler not to fetch selected paths | robots.txt | A client that does not honor the protocol |
| Deny a verified named crawler at the request boundary | Exact user-agent claim plus the operator's published verification method | Unidentified automation and traffic outside the published ranges |
| Reduce excessive retrieval without a full block | Rate limit scoped to the resource and the strongest available identity | Distributed clients if the key is only source IP |
| Protect subscriber, account, or internal content | Authentication and application authorization | Public-page scraping |
| Stop a suspicious interactive session | Bounded browser challenge followed by a short-lived decision | Direct HTTP clients and legitimate clients that cannot run the challenge |
| Apply several signals before serving public content | Request classification at the edge or origin | The need to choose a fallback for unknown or failed classifications |
Do not deploy every control because it appears in the table. Each additional layer creates a new failure mode. Use the smallest combination that enforces the policy you actually chose.
Control matrix
| Control | Enforcement point | Main coverage | Common bypass or limit | Reader impact | Operational work | Fast rollback |
|---|---|---|---|---|---|---|
robots.txt | Crawler reads a public file | Named clients that honor the protocol | It is not access authorization | None for normal browsing | Keep tokens and paths current | Remove or change the group |
| User-agent rule | Server, application, or edge | Requests making an exact named claim | The header can be copied or removed | Low if narrowly scoped | Maintain tokens and exceptions | Disable the rule |
| Verified-network rule | Server or edge | Named clients with a published IP or DNS method | Lists change; not every client publishes proof | Low if identity mapping is exact | Refresh and audit evidence | Restore the prior list or policy |
| Rate limit | Application, gateway, or edge | Repeated requests sharing the selected key | Weak keys split one actor or combine unrelated users | Depends on threshold and route | Tune keys, windows, and budgets | Raise limit or bypass affected routes |
| Authentication and authorization | Application | Restricted resources and state-changing actions | Does not stop access to public pages | Expected login or entitlement flow | Maintain access model | Restore prior permission rule |
| Browser challenge | Browser-facing edge or application route | Interactive clients that must execute the check | Some automation can complete it; non-browser clients cannot | Potentially high | Prevent loops and accessibility failures | Switch to observe-only or bypass route |
The enforcement point matters less than the decision quality. An edge rule can save origin work, but an application still owns subscriber entitlements and permissions for state-changing actions.
Block named crawlers with robots.txt
The Robots Exclusion Protocol lets a service publish groups of rules for crawler product tokens. A full-site preference for two documented model-development crawlers looks like this:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
Use the exact current token from the operator's documentation. OpenAI documents
separate controls for GPTBot and OAI-SearchBot, so blocking the training
crawler does not require blocking the search crawler. OpenAI crawler
documentation
Anthropic likewise documents separate rules for ClaudeBot,
Claude-SearchBot, and Claude-User. Anthropic crawler
documentation
Place the file at /robots.txt on each hostname covered by the policy. Test the
served response rather than only the file in your repository. A CDN, framework,
or redirect can change what a crawler receives.
RFC 9309 says crawlers are
requested to honor the rules and states that those rules are not access
authorization. The RFC also warns that listing a private-looking path in
robots.txt makes the path public. Use authentication or another application
security control for content that must stay restricted.
For syntax, matching, caching, and failure behavior, use the dedicated
robots.txt guide.
Enforce a named-client block at the request boundary
If the server must deny a named crawler, a user-agent string alone is too weak. Any HTTP client can copy it. Combine the exact client claim with the verification method published for that client.
The policy logic should be explicit:
claim = match an exact configured crawler token
identity = verify the source using that client's configured method
if claim is blocked and identity is verified:
deny with the configured response
else if claim exists and identity failed:
apply the spoofed-claim policy
else if verification is unavailable:
apply the preselected unknown-state fallback
else:
continue to the next policy check
Take the source address from trusted infrastructure, not an unsanitized
X-Forwarded-For header. Fetch range lists only from configured operator URLs.
Bound download size, redirects, time, and accepted entries. Keep a last
known-good list when a refresh fails.
Do not merge every range published by one company into a company-wide block. An operator may use separate networks for training crawlers, search crawlers, and user-triggered fetchers. How to verify AI agents covers these identity states and the operator-specific methods.
Rate-limit crawler traffic
Rate limiting is useful when some access is acceptable but its volume or pace is not. The hard part is the key.
Possible keys include:
- verified crawler identity;
- authenticated account or API credential;
- first-party session;
- source network or IP address;
- route group;
- a bounded combination of these fields.
An IP-only limit is easy to operate, but it can group unrelated clients behind one address and miss a distributed client using many addresses. A verified operator identity is better for named crawler budgets. Account or session keys are stronger for authenticated workflows.
Define the window, burst allowance, route scope, and response before launch.
The 429 Too Many Requests status means the client sent too many requests in a
given period. RFC 6585
allows a Retry-After header and deliberately leaves the counting method and
client identity to the server.
Do not rate-limit every route equally. A cached public article, an expensive search endpoint, and a login action have different costs and abuse risks.
Enforce restricted content in the application
Use authentication and authorization for subscriber archives, account pages, paid APIs, preview systems, and internal routes. A crawler name should never grant access that the request's credentials do not have.
Apply the same rule to agentic clients. If an authorized user delegates a task to an agent, the agent still acts within that user's permissions. Verify every state-changing action, validate its input, and keep the normal abuse controls in place.
This control is less glamorous than fingerprinting, but it settles the most important question: can this request access this resource?
Use challenges only where a browser can complete them
A browser challenge can add evidence for a suspicious interactive session. It is a poor fit for a known crawler, feed reader, API client, or other software that is not supposed to execute a browser flow.
Scope challenges to routes where browser interaction is expected. Give a successful client a short-lived, integrity-protected pass. Prevent redirect and challenge loops. Define what happens when cookies, JavaScript, or accessibility requirements make the flow unavailable.
Start in observation mode. Measure how many real sessions would have been challenged and which routes produce ambiguous results. A technically working challenge can still be the wrong control if it sits in front of every article.
Validate before and after rollout
Run a small test matrix before enabling a block:
| Case | Expected result |
|---|---|
| Verified crawler covered by the block | Denied on the intended paths with the documented reason |
| Verified crawler outside the block | Preserves its previous access |
| Copied crawler user agent from an unrelated source | Does not inherit verified identity |
| Client without a useful identity | Uses the configured unknown-client policy |
| Published range refresh fails | Keeps the last known-good evidence and reports the dependency error |
| DNS verification times out | Uses the chosen temporary-failure fallback |
| Client supplies its own forwarding header | Cannot change the source identity recorded at the trusted boundary |
| Reader on an affected route | Completes the normal page or challenge flow without a loop |
| Restricted resource without credentials | Remains denied regardless of crawler identity |
After rollout, watch request counts, response codes, bytes, cache behavior, origin load, challenge completion, and reader-facing errors. Break results down by policy reason. A falling crawler count can mean the block worked, the client changed identity, or the classifier stopped recognizing it.
Keep one canary route or a small traffic percentage in observe-only mode when the platform allows it. This gives you a comparison while the rule settles.
Roll back safely
Every blocking rule needs an owner, expiry or review date, and one-step rollback. Keep the previous rule set available. Record which routes and client classes the change affects.
Roll back or switch to observation when:
- approved search or partner traffic is denied;
- reader errors or challenge loops rise;
- the verifier loses its current evidence source;
- an overly broad key groups unrelated clients;
- the origin or edge rule adds unacceptable failure behavior.
Do not delete the incident data after rollback. Preserve reason codes and the rule version so the next change addresses the actual miss.
The broader decision model is in How to control AI agent access. It separates identity, resource, behavior, and response instead of turning "AI" into one global block.
Where Centinel fits
Centinel helps teams identify and control automated traffic. During an evaluation, test how a crawler is identified, where the policy runs, what the unknown-state fallback is, and how quickly a rule can be disabled. Use traffic from your own site rather than assuming one control covers every crawler.
To build the inventory and test cases before enforcing a new block, request a site audit.