How to block AI crawlers and verify the block

Blocking an AI crawler takes two rules. A robots.txt group tells the documented crawler to stay out, and the large operators say their crawlers follow it. A request rule at your server or edge refuses everything robots.txt cannot reach: clients that copy a crawler's name, fetchers the operators exempt, and scrapers that send no name at all. Before you block, know what an AI crawler is and which ones you want.

The group below covers the training crawlers of OpenAI and Anthropic and the training opt-outs of Google and Apple.

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

Pick the method for what you want to stop

You want to	Use	It will not stop
Keep your pages out of model training	`robots.txt` groups for training tokens	A client that ignores the file, or one that copies the name
Refuse a named crawler's requests	A rule that matches the name and checks the operator's published IP list or DNS	Automation that sends a browser user agent
Slow a crawler you otherwise accept	A rate limit keyed on verified identity, per route group	A client spread over many addresses, if the key is the IP
Keep subscriber or account content private	Authentication and authorization in the application	Scraping of public pages
Stop automation that arrives as a browser	Detection on browser and connection evidence, with a challenge where a browser is expected	Plain HTTP clients on routes without the check

Start with the first two rows. Add another when your logs show the traffic it addresses, because each extra control is one more thing that can block a reader or a search crawler by mistake.

Step 1: write the `robots.txt` groups by purpose

Operators publish one token per purpose, so you can refuse training and keep search. OpenAI separates GPTBot (training) from OAI-SearchBot (ChatGPT search), and says a site that disallows OAI-SearchBot will not be shown in ChatGPT search answers (OpenAI crawlers). Anthropic separates ClaudeBot from Claude-SearchBot the same way (Anthropic crawlers). Perplexity says PerplexityBot is a search crawler and is not used to collect content for foundation models (Perplexity crawlers).

Google-Extended and Applebot-Extended work differently. Neither one fetches pages. Googlebot and Applebot do the crawling, and the extended token only controls whether the content may be used for Gemini training and grounding or for Apple's foundation models (Google's common crawlers, About Applebot). Disallowing either one leaves your search listing alone. It also means a firewall rule that matches Google-Extended in the user-agent header does nothing useful: Google never sends that string, so every request carrying it is a copy.

Put the file on every host you serve. Anthropic asks for the rule on each subdomain you want to opt out. The robots.txt guide for AI bots has the matching rules, path examples, and the search tradeoff for each token.

Step 2: know what `robots.txt` leaves open

When a person asks an assistant to read a page, the request comes from a different client, and operators treat it as the person's request. OpenAI says of ChatGPT-User that "robots.txt rules may not apply". Perplexity says Perplexity-User "generally ignores robots.txt rules". Meta says meta-externalfetcher may bypass the file, and Amazon says Amzn-User may not follow every directive (Meta web crawlers, Amazonbot). Anthropic is the exception in this group: it documents Claude-User as a token site owners can disallow. If you want these fetches refused, you need the request rule in step 3.

Any HTTP client can also send GPTBot as its user agent. In our crawler verification data, which covers news and media sites, 84% of requests that called themselves GPTBot on the typical site failed OpenAI's published IP check. A copy has no reason to read your robots.txt.

OpenAI, Perplexity, Meta, and Amazon each say a robots.txt change can take about 24 hours to apply. Amazon's crawlers may also work from a copy of the file up to 30 days old.

RFC 9309 says the rules "are not a form of access authorization", and every path you list is readable by anyone. Content that must stay restricted needs a login.

Step 3: refuse the request at the server or edge

A rule that matches the user-agent string alone refuses the honest crawler and is bypassed by changing one header. Tie the name to the proof its operator publishes:

claim = match an exact configured crawler token
identity = verify the source using that client's published method

if claim is blocked and identity is verified:
deny with the configured response
else if claim exists and identity failed:
apply the spoofed-claim policy
else if verification is unavailable:
apply the preselected unknown-state fallback
else:
continue to the next policy check

The proof differs by operator. OpenAI and Perplexity publish one JSON IP list per client, Anthropic publishes a single list for all three of its bots, and Google and Apple support both IP ranges and forward-confirmed reverse DNS. How to verify AI agents walks through each method, and the crawler IP checker tests one address by hand.

Check for four mistakes:

The rule reads the client address from X-Forwarded-For as the client sent it. Take the address from your own proxy or CDN.
The range list is fetched once and never refreshed. Refresh it on a schedule, bound the download size and time, and keep the last good copy when a refresh fails.
All of a company's ranges are merged into one block. That removes the search crawler along with the training crawler.
The block covers /robots.txt. Anthropic warns that blocking its addresses "may not work correctly or persistently guarantee an opt-out", because the bot can no longer read the file. Serve /robots.txt to everyone.

Your CDN may have a switch for this. Cloudflare, for example, lets a site block bots by the behavior it assigns them (Search, Agent, or Training), including some unverified ones (Cloudflare: Block AI bots). A switch like that acts on the vendor's classification, so check which clients it covers before you rely on it.

We do not publish Nginx, Apache, or WAF snippets here. A rule copied from an article has not been tested against your proxy chain, and a wrong client address makes it match the wrong requests.

Step 4: limit what you do not want to block

Some crawlers are welcome at a lower volume. Anthropic's bots honor the non-standard Crawl-delay directive, and Amazon says its crawlers do not. For everything else, a rate limit does the job, and its key decides whether it works. An IP key is simple to run, but it groups unrelated visitors behind one address and misses a client that rotates addresses. A verified crawler identity is the better key for a named crawler, and an account or session is better on logged-in routes.

Set the window, the burst allowance, and the routes before launch, and give an expensive search endpoint a tighter budget than a cached article. The response is 429 Too Many Requests. RFC 6585 lets you add a Retry-After header and leaves the counting method to the server.

A browser challenge belongs only on routes where a browser is expected. A feed reader, an API client, or a real crawler cannot complete one, and a challenge in front of every article interrupts readers. Fixing challenge loops covers the failure cases.

Step 5: test the block

Check that the file is served, then that the request rule answers a claimed crawler:

curl -i https://example.com/robots.txt
curl -I -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot" https://example.com/

The first command should return 200 and your groups. The robots.txt checker reads the same file and lists the verdict for each AI bot token. The second comes from your own machine, so it is a copied name: it should get your spoofed-claim response, whatever you chose for that. It cannot show how the verified crawler is treated. For that, wait a day and read the log lines whose source address is on the operator's list.

Before you enforce, run these cases:

Case	Expected result
Verified crawler covered by the block	Denied on the intended paths, with the reason logged
Verified crawler outside the block	Same access as before
Copied crawler name from an unrelated address	Treated as a spoofed claim
Client with no useful name	Gets the unknown-client policy
Range list refresh fails	Last good list stays in use and the failure is reported
Client sends its own forwarding header	Recorded source address does not change
Reader on an affected route	Page loads, with no challenge loop
Restricted page without credentials	Denied, whatever the crawler name

After rollout, watch requests, status codes, bytes, and reader-facing errors, split by policy reason. A falling count for one crawler name has three possible causes: the block worked, the client changed its name, or your classifier stopped recognizing it. Check the verified share for that name, and whether the same source addresses now appear under another name.

Roll back

Give every blocking rule an owner, a review date, and a one-step way to turn it off, and keep the previous rule set. Switch to observe-only when an approved search or partner crawler is denied, when reader errors or challenge loops rise, or when the verifier loses its range source. Keep the logged reasons and the rule version afterwards, so the next attempt fixes the cause.

Blocking scrapers that send no crawler name

The steps above handle clients that name themselves, truthfully or otherwise. A scraper that runs a real or modified browser sends a browser user agent, and no crawler rule sees it. Finding it takes evidence from the browser and the connection. Centinel helps teams identify and control that traffic. The scraping and AI crawler use case describes how the two are handled together, and the integration options show where the decision can run.

If you are not sure which clients your current rules would cut off, request a site audit with the routes and the crawlers you want to keep.