An AI crawler audit should tell you which clients reach your site, which identities you can verify, what resources they use, and whether your access policy is working. A list of user-agent strings answers only the first part, and even then it records claims rather than verified operators.
Start with a bounded log window and the routes you care about. Keep unknown clients in the inventory so the audit does not become a report about only the bots that announce themselves.
Choose the observation point and time window
Decide whether you are auditing edge requests, origin requests, or both. A cache can answer requests without forwarding them to the origin. A control at the edge can also reject traffic before an application log exists. Comparing those totals without accounting for the boundary leads to misleading conclusions.
Use a recent complete window that covers the activity you want to understand. Record its start, end, time zone, and known collection gaps. Preserve sampling information if the logging system does not retain every request.
Collect only the fields needed for the investigation. Useful fields include a normalized route, time bucket, response status, cache outcome, bytes returned, claimed crawler name, verification result, and policy revision. Keep sensitive query parameters, cookies, and credentials out of the audit export. OWASP's logging guidance is a useful reference for deciding what to exclude.
Separate crawler purpose from its name
Different clients from the same operator can perform different jobs. OpenAI's crawler documentation distinguishes GPTBot's training-related role, OAI-SearchBot's search role, and ChatGPT-User's user-initiated visits. It also describes their separate controls and published network references.
Anthropic's crawler guidance distinguishes ClaudeBot, Claude-SearchBot, and Claude-User. Do not apply one operator's verification or robots behavior to another by analogy.
Record the purpose documented by the operator, the source, and when you checked it. Keep the raw observed name alongside the normalized label; names and versions change. The AI traffic taxonomy helps distinguish background crawling from other automated visits.
Verify identity before adding a trusted rule
Use the operator's documented method. For Google, the verification guide describes published IP ranges and reverse DNS followed by a forward lookup back to the original address. Use the correct crawler category and exact hostname boundaries, not a loose substring match.
Record separate outcomes: verified, contradicted, and unknown. A lookup timeout belongs in unknown. An address absent from a reference does not automatically establish malicious intent; check which client the reference actually covers.
Store the verification source and retrieval time. A request with a copied user-agent string must not inherit trust from another request with the same name. For a fuller procedure, see how to verify crawler and agent identities.
Build the audit table around decisions
Group by verified operator or unknown client category, normalized resource, and response. Avoid one hostname-wide total that hides which route is doing the work.
The following rows are hypothetical examples of decisions an audit could support. They are not observations from Centinel traffic.
| Finding to investigate | Evidence to retain | Decision to review |
|---|---|---|
| A verified search crawler reaches public pages successfully | Verification reference, response distribution, affected routes | Preserve intended search access |
| A verified training crawler reaches a route meant to be restricted | Route policy, robots rules, actual response, policy revision | Reconcile cooperative rules with enforced access |
| An unverified client claims a trusted crawler name | Original claim, trusted source address, verification outcome | Investigate before granting an exception |
| An unknown client repeatedly reaches expensive search routes | Bounded request history, response cost, available client evidence | Review route budgets and automation evidence |
| An approved crawler begins receiving failures after a release | Change time, rule revision, response details | Check for a policy or integration regression |
Measure workload as well as request count. A cached detail page and an origin-generated search response can impose different work even when each appears once in a counter. Keep response bytes, cache results, and available origin-work indicators separate rather than inventing a single cost score.
Compare intended policy with actual responses
RFC 9309 defines cooperative crawler instructions, not an authorization boundary. Your audit should therefore compare three records: the robots rule, the enforced route policy, and what the requester actually received.
For an approved client, check successful access to its intended routes. For a restricted client, check whether protected data was returned. A block counter alone cannot answer either question if another endpoint returns the same data.
Use the crawler access-policy guide to document the decision. If you need to change enforcement, the crawler-blocking guide covers the rollout questions.
Leave a repeatable handoff
Save the window, scope, aggregation method, verification references, and policy revision with the result. Separate confirmed findings from questions requiring another observation. Assign one owner and a next action to each proposed change.
Repeat the same audit after a policy change and compare equivalent windows. Retain missing-data and unknown categories; a smaller unknown count is meaningful only if the evidence improved, not if the query stopped including those requests.
The scraping and AI crawler use case connects this inventory to response options. Start with the route where intended access and observed behavior disagree, and keep its evidence with the policy change.