robots.txt is where a site publishes crawl preferences. It is useful for cooperating crawlers, including most AI crawlers, but it is not an access-control file. A requester can ignore it or copy a trusted crawler's name.
This guide shows how to turn a policy into a small, testable file and how to detect the gap between a published preference and observed traffic. It covers the file only. To deny requests at your server or edge, use how to block AI crawlers.
Start with the policy
Do not begin by copying a long list of crawler names. First decide what each kind of automated use may do with each kind of resource.
| Resource | Search indexing | Model-development crawling | AI search or grounding | User-directed retrieval |
|---|---|---|---|---|
| Public documentation | Decide | Decide | Decide | Decide |
| Public product pages | Decide | Decide | Decide | Decide |
| Expensive search endpoints | Usually needs request-time controls | Usually needs request-time controls | Usually needs request-time controls | Usually needs request-time controls |
| Account or private routes | Enforce outside robots.txt | Enforce outside robots.txt | Enforce outside robots.txt | Enforce outside robots.txt |
The labels are intentionally separate. OpenAI, for example, documents GPTBot, OAI-SearchBot, and ChatGPT-User for different purposes. Anthropic documents ClaudeBot, Claude-SearchBot, and Claude-User separately. Your rules should reflect the use you intend to permit, not just the company behind a token.
Use the AI-agent access policy guide to define the decision before editing the file.
What the protocol does
The Robots Exclusion Protocol defines a file named /robots.txt at the service root. The file contains groups of user-agent lines and rules. Disallow identifies paths the matching crawler is asked not to access. Allow permits a more specific path within a broader disallow rule.
For a matching group, the most specific path rule wins. A group for a named product token is more targeted than the * group. Test overlaps rather than relying on the order in which you happened to write the lines.
The protocol also defines retrieval and caching behavior. A crawler may cache a valid file, so a change need not appear in its behavior immediately. If availability matters, serve a small valid file reliably and monitor the response status.
Use current operator tokens
Treat operator documentation as time-sensitive input. Check it when you create a rule and on a scheduled review date.
OpenAI
OpenAI's crawler documentation distinguishes separate product tokens, including:
GPTBotfor model-development crawling;OAI-SearchBotfor search;OAI-AdsBotwith its own control;ChatGPT-Userfor certain user-initiated requests.
Blocking the training crawler does not require blocking the search crawler. OpenAI says ChatGPT-User requests are initiated by users and robots.txt rules may not apply to them, so decide user-directed retrieval separately.
OpenAI also publishes IP ranges for its crawlers. Those ranges help verify a request, but they do not belong in robots.txt itself.
To ask GPTBot not to crawl the site:
User-agent: GPTBot
Disallow: /
Anthropic
Anthropic's crawler documentation distinguishes ClaudeBot, Claude-SearchBot, and Claude-User. Anthropic also links to crawler IP addresses and notes how its crawler interprets its documented tokens. Unlike OpenAI's user-triggered fetcher, Anthropic says its bots honor robots.txt, including the Claude-User control. Do not assume one operator's policy applies to another.
To disallow model-development crawling while leaving the other purposes to their own explicit policy:
User-agent: ClaudeBot
Disallow: /
Google's common crawler reference documents Google-Extended as a control token. Google-Extended does not have a separate HTTP user-agent string; Google says existing Google user agents perform the crawl.
Google also states that the Google-Extended control does not affect inclusion or ranking in Google Search. That makes it different from blocking Googlebot itself.
User-agent: Google-Extended
Disallow: /
These examples are not a universal block list. They demonstrate how to translate a decided policy into operator-documented tokens.
Weigh search visibility first
Blocking a search crawler has a visibility tradeoff. OpenAI says opting out of OAI-SearchBot removes the site from ChatGPT search answers, though navigational links can remain. Anthropic says disabling Claude-SearchBot prevents its search indexing and may reduce visibility. These are the operators' own descriptions. Allowing a crawler does not guarantee traffic from it.
Restrict a crawler to part of the site
Suppose GPTBot may crawl /docs/public/ but not the rest of /docs/:
User-agent: GPTBot
Disallow: /docs/
Allow: /docs/public/
Check the result against real URLs, including encoded paths, uppercase variants, query strings, and any redirects that expose the same resource elsewhere. A path rule is only useful when it matches the URLs your application actually serves.
Avoid listing private or secret paths as a security measure. The file is public, and a disallow rule does not stop a client from requesting the path.
Test the file before release
Run the same checks in preview and after deployment:
- Fetch the exact production path and retain the headers:
curl -i https://example.com/robots.txt. - Confirm the response is successful, plain text, and contains the intended version.
- Test representative allowed and disallowed URLs against the parser behavior you expect.
- Confirm redirects, CDN rules, and host aliases do not serve a different file.
- Save the last known-good file so a bad generation or deployment can be compared and rolled back.
For a quick read of the deployed file, the robots.txt checker reports what it says to each AI bot token. It tests the home page only, so step 3 still applies to other paths.
Do not generate an unbounded list at request time. Build or store a reviewed file, validate its size and syntax, and serve it without depending on an operator endpoint during each crawler request.
Monitor whether the preference changes traffic
A published rule and an enforcement decision answer different questions. Monitor both:
- the status and content returned from
/robots.txt; - requests carrying a token you disallowed;
- source IP and verification result for named crawlers;
- requested resource, response, and rate over time;
- the delay between a policy change and observed behavior.
Traffic after a change is not automatically proof that an operator ignored the rule. It may be a cached policy, a different product token, a spoofed user-agent, or a request that falls outside the changed path. Preserve these possibilities in the investigation.
The AI-agent verification guide explains how to evaluate a named claim using current operator evidence.
What robots.txt cannot enforce
robots.txt does not authenticate a client, authorize access, rate-limit requests, or challenge a session. A user-agent string is unvalidated request input. Anyone can send one.
Google-Extended shows the gap. Google never sends that token in a request, so any request with
Google-Extended as its user agent is fake, and the Disallow rule above does not stop it. Our
crawler verification data shows how often crawler names are copied.
When a resource needs technical protection, enforce the decision at a point that actually sees the relevant request. Depending on the route and integration, that may include source-network verification, protocol evidence, behavioral limits, authentication, rate controls, or a challenge. No single signal proves every kind of automation. How to block AI crawlers compares those request-time controls, including how to enforce a block against clients that copy a crawler's name.
Centinel can contribute browser, network, session, and crawler evidence to those request-time decisions. Coverage depends on the integration: browser-side collection does not govern every non-browser request, while server or edge controls must be placed on the traffic path they are meant to cover.
Use robots.txt to communicate. Use verified evidence and resource-specific rules when you need to enforce.