A publisher operator's guide to telling real AI agents from spoofed ones. IP ranges, reverse-DNS, TLS fingerprints, request signing, and the policy on top.
What is AI agent verification?
AI agent verification confirms that a request really came from the crawler it names. A request says it is GPTBot, ClaudeBot, PerplexityBot, Googlebot, or Applebot. Verification checks if that is true. This is not bot detection. Bot detection asks if the client is a machine. Verification asks if the machine is who it claims to be.
The problem starts in your logs. A user agent string is just a header the client chose to send. Any scraper can set User-Agent to GPTBot. The line in your access log is not proof of identity. It is a claim. Verification is the gap between the claim and the truth.
Three kinds of traffic need this line drawn. Search indexers send you referral traffic and usually belong on an allowlist. Then there are AI crawlers you have licensed or chosen to allow. And there are agents acting for a real person, where the operator matters more than the automation. A wrong call costs you. You lose search ranking, miss licensing revenue, or block a customer mid-purchase.
Why AI agent verification matters right now
Volume forces the decision. Our data shows 39% of the top one million websites were accessed by AI bots by early 2026. Only 2.98% of those sites actively block them. We measured AI agent traffic growing 7,851% across 2025. When AI traffic is a rounding error, a blanket block or blanket allow is cheap. When it is a third of your requests, each choice gets expensive.
Stated rules have stopped working. In our data, ChatGPT-User reached 42% of sites that had blocked it in robots.txt. We found 30% of all AI bot scrapes ignoring robots.txt permissions outright. robots.txt was never a verification tool. It is a request, not a lock.
Spoofing is cheap. Our data shows 95% of advanced bot attacks pass passive inspection. We find 83% of simple curl-based bots slip through unnoticed. Pretending to be a named AI crawler is a weekend project. You rotate residential IPs. You copy a user agent from the vendor docs. You pick a TLS library that mimics a Chrome handshake.
The money changed last year. OpenAI, Perplexity, and Google signed licensing deals with publishers. That revenue depends on telling the licensed agent from the scraper copying it. Verification is now the meter.
Types of verification signals
Verification signals fall into four tiers. I rank them by how hard they are to spoof.
User-agent claim. Free. Any client can send any user agent. Matching a published AI bot string finds the traffic that was already trying to comply. It finds nothing else.
IP range and reverse-DNS. Cheap to check. Hard to spoof at scale. Googlebot publishes official IP ranges. It also supports reverse-DNS plus forward-DNS. You look up the PTR record on the source IP. Then you look up the A or AAAA record on that hostname. Then you confirm the hostname ends in googlebot.com. Bingbot uses the same pattern under search.msn.com. Applebot publishes its ranges. OpenAI publishes GPTBot, OAI-SearchBot, and ChatGPT-User IP ranges on platform.openai.com. Anthropic publishes ClaudeBot and Claude-User ranges. PerplexityBot has been inconsistent. The real cost is upkeep. The directories change, and last quarter's list is already stale.
Cryptographic request signing. Expensive to deploy. Almost impossible to spoof while the private key stays private. The IETF draft-ietf-httpbis-message-signatures spec gives a client a standard way to sign a request with its identity. No major AI vendor requires it yet. An emerging signed-agent proposal and Anthropic's experiments with signed agent passes are the first moves. Watch it. Do not rely on it yet.
Behavioral and fingerprint signals. Expensive for the attacker. Cheap for the defender. JA4 TLS fingerprints hash the cipher suites, extensions, and extension order a client sends in its ClientHello. We track about 15 million unique JA4s across our edge each day. HTTP/2 SETTINGS values and pseudo-header order differ by browser family, and spoofing libraries rarely copy them. Request cadence, revisit intervals, and path patterns separate a scheduled crawler from a burst of rotating-proxy traffic. Fingerprints catch the long tail that IP ranges and signatures miss.
How AI agent verification works
A production pipeline reads three layers of signal per request. It returns an identity verdict before the origin sees the body.
The first layer is static identity. The edge compares the user agent and source IP against a maintained directory of AI crawler identities. Say the source IP sits inside Anthropic's published ClaudeBot range and the user agent matches. The claim fits the public record. Say the source IP is a residential proxy and the user agent says ClaudeBot. The claim is already dead.
The second layer is cross-layer consistency. The TLS handshake exposes a JA4 fingerprint. Chrome sends a specific WINDOW_UPDATE value in its initial SETTINGS frame, about 15MB. Firefox sends about 12.5MB. GPTBot sends whatever library OpenAI uses, and it stays stable across requests. A request that claims to be Chrome, carries a Python TLS fingerprint, and uses a Go library's HTTP/2 setting has lied twice. Any single layer is spoofable. The combination is not. Spoofing toolchains do not cover every layer at once.
The third layer is behavior over time. A real ClaudeBot makes a steady number of requests per second. It revisits on a predictable cycle. It stays inside its own IP range. A spoofed ClaudeBot bursts a hundred pages in a minute. It drifts across autonomous systems. It stops when its proxy pool runs out. A rolling window of 50 to 100 requests per source is usually enough. That classifies a new fingerprint with high confidence.
Build or buy comes down to directory upkeep, not engineering. Writing a reverse-DNS lookup takes a weekend. Keeping IP ranges and user agents accurate for 50 AI crawlers over 24 months is a full-time job. Commercial bot-management vendors offer managed directories. Centinel ships fingerprints for the majors and the long tail. The data has to stay fresh.
How to identify a legitimate AI agent
Start with the operators that publish. GPTBot, OAI-SearchBot, and ChatGPT-User have IP ranges and a published user agent. ClaudeBot and Claude-User are documented at docs.anthropic.com. Googlebot and Google-Extended support reverse-DNS and publish IP ranges. Applebot is documented at support.apple.com. Bingbot is documented at bing.com/webmasters. If a request claims to be one of these and the IP sits outside the published range, it is a spoof. Block it at the edge and move on.
Some operators publish a user agent but no maintained IP range. PerplexityBot has fluctuated. Several smaller AI startups publish nothing. For these, fall back to fingerprint plus behavior. Check three things. The JA4 is stable across a rolling window. The cadence matches a training crawler. The path pattern looks like a crawl pass, not a targeted pull. If all three hold, the request is probably legitimate. Log it as unverified but consistent.
Agent-on-behalf-of-user traffic moves identification up a layer. Think of a ChatGPT agent buying a ticket, a Claude browsing session, or a custom agent built on Anthropic's Model Context Protocol. MCP servers expose capabilities to agents over authenticated channels. The bearer token sits at the MCP layer, not at HTTP. So the question changes. Did the agent present a valid token, and which operator issued it?
A publisher has to pick a lens. A newsroom CTO cares most about training and retrieval crawlers. Those consume archive content and sit at the center of the licensing talks. A DTC e-commerce platform cares most about agent-on-behalf-of-user traffic. Those agents complete purchases and belong on a verified-allow path. A SaaS docs platform cares most about search and retrieval crawlers, because its content needs to be cited. The signal mix is the same. The policy is not.
How to respond to unverified agents
Verification without a response is just a log file. Three responses cover almost every case.
Block the ones that fail basic consistency. A request with a GPTBot user agent, a residential IP, a curl-impersonate TLS fingerprint, and a burst rate of 200 pages per minute is not GPTBot. Drop it at the edge. Origin cost and licensing leakage both fall at once.
Challenge the ones in the grey zone. Say you see a new fingerprint, consistent with a headless browser, hitting pages a human reader would hit. An interstitial check or a proof-of-work challenge sorts a curious developer from a commercial scraper behind residential proxies. The challenge is cheap for a real user and expensive for a scraped pool.
Verify-and-allow the rest. This is the unpopular answer, and usually the right one. The 2.98% block rate on AI bots across the top one million sites is not a failure of bot management. It is a sign that most operators found blanket blocks cost more than they save. The right default for a verified search indexer, a verified retrieval crawler, or a verified agent with a signed pass is simple. Let it through. Log the operator. Watch the cumulative volume. Block goes on the table only when a specific operator fails verification or blows past a budget you set.
Verification is not the same as safety. A real GPTBot operator can run its crawler behind a residential proxy pool. A real Claude-User session can be driven by an abuse script at the other end. Verification confirms the operator. The operator's behavior is a separate check.
This is the layer Centinel runs for publishers who do not want to maintain their own directory. Every request is fingerprinted against 1,600+ AI agent profiles. Each one is checked for cross-layer consistency. Then it goes to a per-agent policy in under 2ms at the edge. The policy can block, allow, challenge, or charge. The block list stays current because the fingerprints stay current.
Key takeaways
- Verification is not detection. Detection asks if the client is automated. Verification asks if the automation is who it says it is. The figures above all show that gap widening fast.
- The signals stack. The user agent is free and spoofable. Published IP ranges and reverse-DNS are cheap and hard to spoof if you keep the directory fresh. TLS and
HTTP/2fingerprints catch the long tail. Cryptographic request signing is the future, not the present. One signal alone is a coin flip. The combination is a verdict. - The response is policy, not one setting. Block the clear spoofs. Challenge the ambiguous. Verify-and-allow everything that identifies itself honestly. The market's 2.98% block rate is its current answer.
- Centinel runs that decision layer at the edge. It keeps a maintained directory of AI agent fingerprints and a policy engine that fires per agent in under 2ms. For a publisher who does not want to build and maintain the directory, that is the whole verification pipeline in one component.