robots.txt is where a site publishes crawl preferences. It is useful for cooperating crawlers, but it is not an access-control file. A requester can ignore it or copy a trusted crawler's name.
This guide shows how to turn a policy into a small, testable file and how to detect the gap between a published preference and observed traffic.
Start with the policy
Do not begin by copying a long list of crawler names. First decide what each kind of automated use may do with each kind of resource.
| Resource | Search indexing | Model-development crawling | AI search or grounding | User-directed retrieval |
|---|---|---|---|---|
| Public documentation | Decide | Decide | Decide | Decide |
| Public product pages | Decide | Decide | Decide | Decide |
| Expensive search endpoints | Usually needs request-time controls | Usually needs request-time controls | Usually needs request-time controls | Usually needs request-time controls |
| Account or private routes | Enforce outside robots.txt | Enforce outside robots.txt | Enforce outside robots.txt | Enforce outside robots.txt |
The labels are intentionally separate. OpenAI, for example, documents GPTBot, OAI-SearchBot, and ChatGPT-User for different purposes. Anthropic documents ClaudeBot, Claude-SearchBot, and Claude-User separately. Your rules should reflect the use you intend to permit, not just the company behind a token.
Use the AI-agent access policy guide to define the decision before editing the file.
What the protocol does
The Robots Exclusion Protocol defines a file named /robots.txt at the service root. The file contains groups of user-agent lines and rules. Disallow identifies paths the matching crawler is asked not to access. Allow permits a more specific path within a broader disallow rule.
For a matching group, the most specific path rule wins. A group for a named product token is more targeted than the * group. Test overlaps rather than relying on the order in which you happened to write the lines.
The protocol also defines retrieval and caching behavior. A crawler may cache a valid file, so a change need not appear in its behavior immediately. If availability matters, serve a small valid file reliably and monitor the response status.
Use current operator tokens
Treat operator documentation as time-sensitive input. Check it when you create a rule and on a scheduled review date.
OpenAI
OpenAI's crawler documentation distinguishes three product tokens:
GPTBotfor model-development crawling;OAI-SearchBotfor search;ChatGPT-Userfor certain user-initiated requests.
OpenAI also publishes IP ranges for its crawlers. Those ranges help verify a request, but they do not belong in robots.txt itself.
To ask GPTBot not to crawl the site:
User-agent: GPTBot
Disallow: /
Anthropic
Anthropic's crawler documentation distinguishes ClaudeBot, Claude-SearchBot, and Claude-User. Anthropic also links to crawler IP addresses and notes how its crawler interprets its documented tokens.
To disallow model-development crawling while leaving the other purposes to their own explicit policy:
User-agent: ClaudeBot
Disallow: /
Google's common crawler reference documents Google-Extended as a control token. Google-Extended does not have a separate HTTP user-agent string; Google says existing Google user agents perform the crawl.
Google also states that the Google-Extended control does not affect inclusion or ranking in Google Search. That makes it different from blocking Googlebot itself.
User-agent: Google-Extended
Disallow: /
These examples are not a universal block list. They demonstrate how to translate a decided policy into operator-documented tokens.
Restrict a crawler to part of the site
Suppose GPTBot may crawl /docs/public/ but not the rest of /docs/:
User-agent: GPTBot
Disallow: /docs/
Allow: /docs/public/
Check the result against real URLs, including encoded paths, uppercase variants, query strings, and any redirects that expose the same resource elsewhere. A path rule is only useful when it matches the URLs your application actually serves.
Avoid listing private or secret paths as a security measure. The file is public, and a disallow rule does not stop a client from requesting the path.
Test the file before release
Run the same checks in preview and after deployment:
- Fetch the exact production path and retain the headers:
curl -i https://example.com/robots.txt. - Confirm the response is successful, plain text, and contains the intended version.
- Test representative allowed and disallowed URLs against the parser behavior you expect.
- Confirm redirects, CDN rules, and host aliases do not serve a different file.
- Save the last known-good file so a bad generation or deployment can be compared and rolled back.
Do not generate an unbounded list at request time. Build or store a reviewed file, validate its size and syntax, and serve it without depending on an operator endpoint during each crawler request.
Monitor whether the preference changes traffic
A published rule and an enforcement decision answer different questions. Monitor both:
- the status and content returned from
/robots.txt; - requests carrying a token you disallowed;
- source IP and verification result for named crawlers;
- requested resource, response, and rate over time;
- the delay between a policy change and observed behavior.
Traffic after a change is not automatically proof that an operator ignored the rule. It may be a cached policy, a different product token, a spoofed user-agent, or a request that falls outside the changed path. Preserve these possibilities in the investigation.
The AI-agent verification guide explains how to evaluate a named claim using current operator evidence.
What robots.txt cannot enforce
robots.txt does not authenticate a client, authorize access, rate-limit requests, or challenge a session. A user-agent string is unvalidated request input. Anyone can send one.
When a resource needs technical protection, enforce the decision at a point that actually sees the relevant request. Depending on the route and integration, that may include source-network verification, protocol evidence, behavioral limits, authentication, rate controls, or a challenge. No single signal proves every kind of automation.
Centinel can contribute browser, network, session, and crawler evidence to those request-time decisions. Coverage depends on the integration: browser-side collection does not govern every non-browser request, while server or edge controls must be placed on the traffic path they are meant to cover.
Use robots.txt to communicate. Use verified evidence and resource-specific rules when you need to enforce.