An AI-agent access policy should answer one question: what may this client do with this resource, given the identity and behavior you can actually verify?
That produces better rules than a site-wide "allow AI" or "block AI" switch. A search crawler fetching public stories, an assistant opening a page for a reader, and an automated browser attempting account actions do not deserve the same response.
Start with a resource matrix. Give every row an owner, review date, logging requirement, and failure behavior. Then test the policy with verified, unverified, spoofed, and unknown traffic before enforcing it.
Start with purpose and resource
Classify the request by why it exists, not by whether somebody has attached the label "AI" to it. Useful working classes include search indexing, model development, retrieval for a current answer, user-directed work, approved automation, and unidentified automation. What is AI agent traffic? defines those boundaries in more detail.
The operator documentation confirms that these purposes are not
interchangeable. OpenAI documents separate controls for GPTBot,
OAI-SearchBot, and ChatGPT-User. OpenAI crawler
documentation Anthropic likewise
documents ClaudeBot, Claude-SearchBot, and Claude-User separately.
Anthropic crawler
documentation
Purpose is still only one input. The same client may be acceptable on a public article and unacceptable on an account endpoint. List the actual resource groups before deciding what to do with any crawler name.
Write the policy as a resource matrix
The example below leans conservative. Replace YYYY-MM-DD, the owners,
identity methods, limits, and failure choices with decisions from your own
team. One row often becomes several when traffic classes need different
responses on the same resource.
| Resource | Traffic class | Identity requirement | Response | Owner | Review | Logging | Failure behavior |
|---|---|---|---|---|---|---|---|
| Public articles | Verified search or retrieval crawler; user-directed agent; unknown automation | Exact documented token plus the operator's current proof for a named allow; otherwise keep identity unknown | Allow approved classes; observe or apply a scoped limit to unknown traffic when behavior warrants it | Editorial and platform | YYYY-MM-DD | Class, identity state, route, bytes, response, rule reason | Keep the previous known-good public policy when verification is temporarily unavailable |
| Subscription archives | Subscriber or agent acting with subscriber authority | Valid first-party session or delegated credential with archive permission | Enforce the same authorization as the subscriber; limit bulk export separately | Subscription product and security | YYYY-MM-DD | Account or credential ID, scope, resource, result, reason; never log secrets | Deny protected access when authorization cannot be checked |
| Site search | Reader, user-directed agent, approved integration, or unknown automation | Session or account for higher budgets; explicit unknown state for anonymous use | Allow a bounded query budget; return 429 when the selected key exceeds it | Search or platform owner | YYYY-MM-DD | Limit key type, route cost, count, response, retry value | Fall back to the documented anonymous budget, not an unlimited query path |
| Feeds | Feed reader, search crawler, retrieval client, or unknown fetcher | Verification for named exceptions; no invented identity for an unidentified client | Serve the public feed under its cache and request budget; deny restricted feed variants without authorization | Editorial platform | YYYY-MM-DD | Feed, identity state, bytes, response, cache outcome | Serve a safe cached public response or the chosen unavailable response |
| Login and account | Reader, delegated agent, monitoring, or unidentified automation | Valid authentication and request protections; bot classification never grants account permission | Authenticate, authorize, rate-limit abusive actions, and use a reviewed step-up check when appropriate | Identity and security | YYYY-MM-DD | Attempt outcome, account-safe identifier, action, rule reason; exclude credentials | Fail closed for state-changing actions when authorization or policy dependencies are unavailable |
| APIs | Partner integration, user-delegated agent, internal job, or unknown client | Credential with the required audience and scope; bind delegated work to the user's permissions | Allow the scoped operation; use 401, 403, or 429 according to the actual failure | API owner and security | YYYY-MM-DD | Credential identifier, scope, endpoint, result, limit state; exclude tokens and bodies by default | Deny protected operations when authentication, authorization, or policy checks cannot complete |
Use the matrix to record actual decisions. If the platform cannot produce an identity state or response named in a row, change the row or the implementation before launch. A control that exists only in the policy document will fail its first real test.
Precedence needs to be written down too. Resource authorization wins over a crawler allow rule. A valid crawler identity must never unlock a subscription archive, account action, or API operation that lacks the required credentials.
Decide identity before response
A user-agent string is a claim. It is useful because it tells you what a client wants to be called, but another HTTP client can copy the same value. Record claim and verification separately.
Use distinct states:
| Identity state | Meaning | Safe policy use |
|---|---|---|
| Verified named client | The request made an exact configured claim and passed that operator's current verification method | Apply the rule for that named client and resource |
| Authenticated user or delegated client | The request has valid first-party or delegated credentials with known permissions | Authorize only the operations covered by those permissions |
| Claimed, verification unavailable | The verifier timed out, its evidence source failed, or required metadata is missing | Use the row's temporary-failure behavior; do not silently convert it to verified |
| Claimed, verification failed | The request made a named claim but did not match the configured proof | Apply the explicit failed-claim policy and record the reason |
| Unknown | The request has no useful claim or no supported proof | Use resource and behavior policy without assigning an operator identity |
Verification methods differ by operator. Google documents two methods for its crawlers: matching against its published IP ranges or performing reverse and forward DNS checks. Google crawler verification Other operators publish their own tokens, ranges, or instructions. Use the method attached to the exact client instead of combining every range from one company into a company-wide identity.
Keep verification evidence outside editorial copy. Store the source URL, retrieval time, parser version, last known-good data, and refresh result. Bound network calls, accepted response size, redirects, and parsing. A failed refresh should not erase current evidence or widen access by accident.
The implementation details and current operator table are in How to verify AI agents.
Choose the smallest useful response
Most policies need five response families: allow, observe, rate-limit, challenge, and deny. They are not an escalation ladder that every request must climb.
Allow serves the requested resource. Keep the identity state and rule reason in the log so approved traffic remains measurable.
Observe serves the normal response while recording evidence for a decision with an owner and end date. Observation without an end date becomes accidental policy.
Rate-limit constrains a defined key, route, window, and burst. The 429 Too Many Requests status means the client sent too many requests in a given
period. RFC 6585 allows
a Retry-After header and leaves the choice of counting method and client
identity to the server. An IP-only key can combine unrelated users and miss a
distributed client, so record why the selected key fits the route.
Challenge adds evidence where a browser is expected and a wrong block would harm a reader. Define accessibility, cookie, timeout, repeat, and loop behavior. Do not place a browser challenge in front of feed readers, APIs, or known crawlers that cannot complete it.
Deny refuses the request. Use authentication and authorization semantics
for protected resources rather than disguising every denial as bot detection.
RFC 9110 defines
401 for a request that lacks valid authentication credentials and requires a
WWW-Authenticate challenge. The RFC defines 403 for a request the server
understood but refuses to fulfill. Choose status and response content based on
the real condition and avoid leaking sensitive policy details.
For implementation options and rollback controls, see How to block AI crawlers.
Keep robots.txt in its proper role
robots.txt publishes rules for crawlers that follow the Robots Exclusion
Protocol. It does not authenticate a client or authorize access to protected
content.
RFC 9309 specifies how crawlers select and apply groups. Its security section says the rules are not a form of access authorization and warns against exposing private-looking paths in the file. Use authentication and application authorization for resources that must remain restricted.
The declared rule and enforced behavior should still agree. If the file asks a named crawler not to fetch a path while the request boundary deliberately allows it, the organization has two policies. Give somebody ownership of that conflict rather than assuming the crawler list will settle it.
Choose failure behavior before rollout
The uncomfortable policy decisions appear when a dependency fails. Make them before launch.
For each matrix row, decide what happens when:
- the crawler range or verification record cannot refresh;
- DNS verification times out;
- the policy service is unavailable;
- connection or browser evidence is missing;
- a rate-limit store cannot be reached;
- a challenge cannot load or repeats;
- logging is degraded;
- the client matches more than one rule.
Public and protected resources usually need different answers. Continuing with a previous known-good rule may be reasonable for a public article during a short verifier outage. An account change or protected API call should not gain permission because authorization is unavailable.
Avoid one global fail_open switch. Store the chosen fallback with the
resource rule, then test it. Define a maximum age for last known-good evidence
and what happens when that age is exceeded.
Test and review the policy
Build a staging matrix before enabling enforcement:
| Test case | Expected result |
|---|---|
| Verified named crawler on an allowed public route | Receives the intended public response and a verified reason code |
| Same crawler on a protected route without credentials | Remains unauthorized |
| Copied crawler user agent from an unrelated source | Does not inherit verified identity |
| Verification source temporarily unavailable | Uses the configured temporary-failure path |
| Verification fails | Uses the failed-claim path, which remains distinct from an outage |
| Unknown automation within the public budget | Receives the normal public response unless another scoped rule applies |
| Search client exceeds its chosen key and window | Receives the documented limit response and retry behavior |
| Browser cannot complete a challenge | Does not enter a loop and has the chosen fallback |
| Policy dependency fails during an account action | Cannot bypass authentication or authorization |
| Previous policy version is restored | Returns the tested pre-change behavior |
Start in observation mode where the route permits it. Compare rule results with approved crawlers, internal monitoring, normal readers, and confirmed abuse. Review response counts, bytes, origin work, challenge completion, reader errors, and unknown-state volume by resource.
Every rule needs an owner, version, review date, reason code, and one-step rollback. Review sooner when an operator changes its documentation, a browser or proxy changes the evidence you collect, or an incident exposes a policy gap. Remove expired exceptions. A forgotten allow rule is still an allow rule.
Evaluate the deployment, not the diagram
Before relying on Centinel or another control layer, confirm which observations and responses are available in the deployment you will operate. Ask where the decision runs, how identity evidence refreshes, what unknown means, which routes can be observed or limited, what happens during dependency failure, and how a rule is rolled back.
Use the resource matrix as the acceptance test. A demo that identifies a crawler but cannot reproduce the row's response and fallback has not yet implemented the policy.
Use the bot management evaluation checklist to carry those tests into a product review without collapsing every resource into one global bot rule.
To inventory the traffic and routes before writing enforcement rules, request a site audit.