Verify AI crawlers and agents: spot fake GPTBot

To verify an AI agent, check its claimed identity against the operator's published IP ranges, forward-confirmed reverse DNS, or a supported signature scheme with trusted keys. A user-agent header alone is not proof. If no supported evidence is available, keep the identity unknown rather than granting trusted access.

First choose what you need to verify. A crawler's operator, a browser's use of automation, and a user's permission to delegate an action are different questions:

• Named crawler or fetcher: use its operator's documented verification method.
• Browser agent with no operator proof: evaluate automation and behavior; do not invent a verified identity from a plausible fingerprint.
• Agent accessing an account: require normal authentication and authorization. Operator verification does not establish the account owner's consent.

A verified crawler can still exceed a retrieval budget or request a restricted route. Verification establishes only the identity supported by the evidence; your resource policy decides what that client may do.

Start with the identity claim

Record four things before building a rule:

1. the user-agent token and full header;
2. the source IP observed at your trusted network boundary;
3. the resource being requested;
4. the verification method published for that client.

Do not take the source IP from X-Forwarded-For unless it was set or sanitized by infrastructure you control. A client can supply that header too.

Use a small set of identity results. It is easier to operate than a vague confidence score.

An Unknown result is not a soft pass. It means the request has not earned the privileges attached to a verified identity.

Verification methods published by operators

Operator methods differ by client and purpose. Store the official source, verification scope, and last successful review with every catalog entry. An operator-wide network list does not necessarily verify a specific product.

Notice what the table does not say. A user-agent match by itself never produces Verified. A request that says Googlebot but fails Google's published check is not verified Googlebot. A request that says Claude-User should not be checked against an unrelated vendor's network list.

Spoofed claims are common. On the median news site in our verification data, 84% of requests claiming to be GPTBot failed OpenAI's published check.

Verify a published IP range

IP-list verification is a membership check, but the maintenance around it is where implementations usually go wrong.

Map the exact user-agent token to a configured operator record. Fetch the range file from the URL in that record, not from a URL supplied by the request. Parse only the expected schema and valid CIDR values. Put limits on response size, redirects, request time, and the number of entries accepted.

Keep the last known-good list when a refresh fails. A network timeout should produce Unknown, not turn every matching user agent into either verified traffic or a spoof. Log the list version and the range that matched so an operator can explain the decision later.

The request check is then straightforward:

claim = identify the user-agent token
record = load the configured record for that exact client

if record has no supported verification method:
result = unknown
else if source IP is inside the client's current published ranges:
result = verified
else:
result = failed

Do not combine all ranges from one company into a company-wide allowlist unless the company documents them that way. OpenAI, for example, publishes separate range files for different clients and purposes.

Verify with forward-confirmed reverse DNS

Google, Apple, and Bing document a two-way DNS check. The forward lookup is not optional.

1. Run a PTR lookup on the source IP.
2. Check the returned hostname against the operator's exact documented suffix.
3. Resolve that hostname to its A and AAAA addresses.
4. Require the original source IP to appear in the forward result.

Check the hostname on a label boundary. crawl.example.googlebot.com may match an allowed suffix; googlebot.com.attacker.example must not. Normalize a final DNS dot before comparison, and do not accept a substring match.

The forward lookup closes a simple hole. An attacker may control the PTR record for an IP and give it a convincing name. They should not pass unless the named operator's hostname resolves back to the source address as its documentation requires.

DNS checks need bounds too. Set short timeouts, cap CNAME traversal, cache successful results according to DNS TTLs, and distinguish a mismatch from a resolver failure. A mismatch is Failed. A timeout or temporary DNS error is Unknown.

Signed requests are useful, but support is not universal

RFC 9421 defines HTTP Message Signatures. A client can sign selected request components, and a verifier can check the signature with trusted key material. The standard supplies the mechanism. An application still has to decide which components must be signed, where keys come from, which algorithms are accepted, and how replay is handled.

Web Bot Auth applies that mechanism to automated traffic. The protocol is still an IETF working-group draft, not a finished RFC. Cloudflare has a deployed implementation with a Signature-Agent key directory and documents its current requirements and limitations in the Web Bot Auth guide.

A valid signature proves that a holder of the accepted key signed the covered parts of the request. It does not prove that the operator's behavior is safe or that every unsigned part of the request can be trusted. The verifier also needs a rule that connects the key to an identity it recognizes.

Treat key discovery as untrusted input. Do not fetch an arbitrary internal or private address because a client placed it in Signature-Agent. Bound response size, redirects, fetch time, and key count. Require HTTPS and an explicit trust policy. These checks matter just as much as the signature math.

Fingerprints can challenge a claim, not authenticate an operator

TLS, HTTP/2, browser-environment, and behavioral signals help detect contradictions. A client may claim to be a browser while its connection looks like a common HTTP library. It may claim to be a scheduled crawler while its requests behave like an interactive browser session.

Those observations are useful, but they answer a different question. A TLS fingerprint does not prove that OpenAI, Anthropic, Google, or another named operator sent a request unless the operator has made that fingerprint part of its published verification method. Fingerprints change. They can also be imitated.

Use these signals to classify automation, find spoofing indicators, and decide how much scrutiny an unknown client needs. Do not silently upgrade Unidentified to Verified because its behavior looks tidy.

For the connection-layer mechanics, see TLS fingerprinting explained. For browser-controlled traffic, see How to detect browser automation.

Identity and behavior are separate decisions

Verification should feed a policy decision, not replace one.

The fallback should vary by resource. A public article, a subscription archive, a login endpoint, and an internal API do not need the same failure behavior. Choose fail-open or fail-closed behavior before the verifier has an outage.

The full policy model is in How to control AI agent access. For the enforcement context these identities feed, see the scraping and AI crawler use case.

A practical verification runbook

Start with a small set of clients that already matter to your site.

1. Inventory the claimed user agents in edge or origin logs.
2. Create one catalog record per client and purpose. Store the official source, verification method, refresh schedule, owner, and last successful update.
3. Verify the source address at the trusted network boundary.
4. Return one of the four explicit identity results with a reason code.
5. Keep identity policy separate from route and behavior policy.
6. Test a genuine request, a copied user agent, a DNS forward mismatch, a stale range file, and a verifier timeout.
7. Review operator documentation on a schedule and when traffic changes.

Also test the boring failure paths. What happens when DNS is slow? What happens when an operator publishes malformed JSON? Does a failed refresh erase the last known-good ranges? Can a request make the verifier fetch an internal URL? These are the cases that decide whether the verifier remains useful under pressure.

Before building any of this, you can run one address through the crawler verification tool. It forward-confirms reverse DNS where the operator documents that method, and falls back to the address list where the operator publishes one instead.

Where Centinel fits

Centinel is built to help teams identify and control automated traffic. During an evaluation, ask which operator records are maintained, which checks run in your supported integration, where the result is enforced, and what happens when verification evidence is unavailable. Test those answers against your own traffic rather than relying on a crawler-count headline.

The broader bot management solution checklist shows how to test crawler verification alongside browser automation, residential proxy traffic, reader impact, and rollback.

If you want to compare the identities declared in your logs with the automated traffic actually reaching your pages, request a site audit.

Sources

• Verify requests from Google crawlers and fetchers
• OpenAI crawler documentation
• Anthropic crawler documentation
• Applebot documentation
• Verify Bingbot
• Perplexity crawler documentation
• RFC 9421: HTTP Message Signatures
• IETF Web Bot Auth protocol draft
• Cloudflare Web Bot Auth implementation