AI agent traffic is a new class of web traffic: training crawlers, retrieval crawlers, agentic workflows, and spoofed scrapers.
What is AI agent traffic?
AI agent traffic is automated web traffic. Software makes the request. That software acts for a large language model, a retrieval-augmented generation system, or an autonomous agent that runs tasks.
This is not classic bot traffic. A classic scraper pulls data on a schedule for its operator. An AI agent fetches a page in the moment. A model, or a person prompting a model, decides the content is needed. The goal is to answer a question, finish a purchase, or fill a context window.
Four kinds of software make it.
Training crawlers sweep the web to build training data. Examples are GPTBot, ClaudeBot, Google-Extended, and Applebot-Extended.
Retrieval crawlers fetch pages in real time to answer a user's question. Examples are PerplexityBot and OAI-SearchBot.
Agent-on-behalf-of-user traffic comes from ChatGPT agents, Anthropic's Claude for browsing, and new agentic frameworks. They submit forms and click links for a person.
A long tail of unlabeled scrapers sits in between. It includes cohere-ai, CCBot, Meta-ExternalAgent, and many startups crawling behind residential proxies.
This traffic breaks the old line between bot and visitor. The request comes from an automated client. The intent came seconds earlier from a person asking a chatbot.
Why it matters now
Volume is the first answer. AI agent traffic grew 7,851% over 2025. By early 2026, 39% of the top one million websites were accessed by AI bots. Only 2.98% of those sites block them. In our data, publisher sites see 1 AI bot visit for every 31 human visits. Two quarters earlier the ratio was 1 in 50.
Bandwidth is the second answer. We measure Anthropic's crawl-to-referral ratio at roughly 500,000 to 1 through 2025. That is half a million pages fetched for every visitor sent back. AI training crawl traffic rose 65% in six months. Every page hit costs the origin. Every archive sweep is a cache miss.
Money is the third answer. AI agents now sit between your content and the reader. An article read inside ChatGPT, summarized by Perplexity, or cited in an AI Overview earns no ad impression. It prompts no subscription. It builds no direct reader relationship. The traffic is real. The payment path is not.
Classic bot management was built for a different problem. Blocking scrapers and blocking AI agents are not the same choice. Treat them the same, and you either cut off a search channel you want, or let a training crawler through that you would rather charge.
Types of AI agent traffic
Four types show up in server logs. Each carries different commercial weight.
Training crawlers. Model companies run them to build training datasets. The main examples are GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended, Bytespider (ByteDance), Applebot-Extended, and CCBot (Common Crawl, whose data feeds many smaller model companies). These crawlers sweep broadly and return often. Their requests are the clearest candidates for licensing. The operator has a budget for data. Legal pressure around unlicensed training is rising.
Retrieval and grounding crawlers. They fetch pages at query time to ground a model's answer. The named ones are PerplexityBot, OAI-SearchBot (OpenAI's search crawler, separate from GPTBot), and ChatGPT-User. In our data, ChatGPT-User reaches 42% of sites that had explicitly blocked it. These crawlers act like search indexers. But they do not send referral traffic the way Googlebot does.
Agentic traffic. AI agents make it while acting for one human user. A ChatGPT agent checks flight prices. A Claude agent researches a paper. Browser-use and similar frameworks click through workflows for a user. The request comes from a headless browser on cloud infrastructure. It often routes through residential proxies. The behavior looks human until it doesn't.
Unlabeled and spoofed crawlers. This is the largest and messiest group. It includes cohere-ai, Meta-ExternalAgent, and many smaller operators. Commercial scraping services like BrightData, Oxylabs, and ScraperAPI sell access to rotating residential IP pools. Some training and retrieval crawlers refuse to identify themselves. Passive inspection misses 95% of advanced bot attacks. It also misses 83% of simple curl-based bots. Unlabeled traffic is where licensing revenue leaks out.
How it works
AI agent traffic is HTTP. Each request has a user agent, a TLS handshake, a set of HTTP/2 settings, and a body. Two things separate it from browser traffic: the software stack making the request, and the intent behind it.
Training crawlers are the simplest. A scheduler runs. A fetcher opens an HTTP connection. A parser pulls text and links. The results go into a dataset. GPTBot and ClaudeBot publish IP ranges and respect robots.txt in most cases. Their footprint in logs is steady: a consistent user agent, a consistent TLS fingerprint, a steady request rate.
Retrieval crawlers are stateful. A user asks a chatbot a question. The model picks which pages to fetch. PerplexityBot or OAI-SearchBot opens connections to those URLs, pulls the content, and hands it back within seconds. The pattern is bursty. Many pages from different domains load in parallel. Query volume drives it, not a crawl schedule.
Agentic traffic is the hardest to pin down. An agent running a workflow may use a patched Chromium build, a headless browser, or a direct HTTP client. The choice depends on whether the task needs JavaScript. Many route through residential proxies to dodge rate limits. Some use curl-impersonate, uTLS, or similar libraries. These copy a real browser's TLS handshake byte for byte. The user agent string is whatever the operator sends.
Spoofing rules the long tail. A scraper rotates through thousands of residential IPs. It swaps user agents per request. It uses a TLS library that copies Chrome's JA3/JA4 fingerprint. On the surface, the traffic looks like a human visitor. The mismatch shows only when you compare signals across layers: TLS handshake, HTTP/2 SETTINGS frame, behavioral pattern, and request rate.
How to identify it
User agents are the starting point, not the answer. Major operators publish their strings: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, PerplexityBot, cohere-ai, CCBot, Meta-ExternalAgent. Matching them finds the compliant traffic. That traffic was already the least likely to cause trouble.
For the rest, you need request-level signals the client cannot fully control.
TLS fingerprinting. The client hello in a TLS handshake exposes the cipher suites, the extensions, and the extension order of the library underneath. Python's requests produces one signature. curl produces another. Real Chrome produces another. JA4, and its successors JA4S and JA4H, hash those signals into a fingerprint. The hash resists extension randomization. We track roughly 15 million unique JA4 fingerprints across our edge each day. A Python TLS stack claiming to be Chrome is caught before the HTTP body is sent.
HTTP/2 settings. Chrome sends a WINDOW_UPDATE of about 15MB in its initial SETTINGS frame. Firefox sends about 12.5MB. Most HTTP libraries send zero. The pseudo-header order (:method, :authority, :scheme, :path) is fixed per browser. It does not match what libraries send by default.
Behavioral patterns. Request rate, path patterns, revisit intervals, and session coherence all count. A real user reading an article dwells. A training crawler moves at a steady rate. A spoofed scraper bursts through a hundred pages in a minute.
Cross-layer consistency. This is the decisive check. Say a request claims to be Chrome by user agent. Its TLS fingerprint comes from curl-impersonate. Its HTTP/2 settings come from a Go library. That request is an AI agent that lied twice. Any one signal is spoofable. The combination is not. The spoofing libraries do not cover every layer at once.
How to respond
Once you identify the traffic, you have three responses: block, verify, or watchlist. Pick per agent, not per source.
Block. For training crawlers you have not approved. For scrapers that ignore robots.txt. For spoofed traffic that fails cross-layer checks. Block at the edge. The origin never sees the request. Your bandwidth bill never grows because of it.
Verify and allow. For search indexers you want to appear in. For partner agents. For agent-on-behalf-of-user traffic you want through but want to audit. Pass the request with a signed trust stamp. Log the agent's identity. Track cumulative volume per operator. Googlebot, Bingbot, and verified AI-search user agents belong on an allowlist by default. That 2.98% block rate shows most operators do not cut themselves off from search.
Watchlist. For training crawlers you have not decided about yet. Centinel records every visit per agent. That gives you the audit trail to act later: block, challenge, or escalate once policy is set. The commercial talk around AI and publisher content is live. The decision stays yours, per agent.
robots.txt alone will not execute any of these. In our data, 30% of AI bot scrapes in Q4 2025 ignored explicit robots.txt permissions. The file is a courtesy notice. Enforcement lives at the edge, in a layer that inspects the request before it reaches the origin.
Key takeaways
- AI agent traffic is not classic bot traffic. It is a new category: training crawlers, retrieval crawlers, agentic workflows, and a long tail of unlabeled scrapers.
- The volume is large and growing fast. AI agent traffic grew 7,851% in 2025. AI bots now reach 39% of top sites. Our data shows a 1-in-31 AI-to-human visit ratio on publisher content.
- The response is not one setting. Block training crawlers you have not approved. Verify and allow search indexers and partner agents. Watchlist the ones you have not decided about. Drop spoofed traffic that fails cross-layer checks.
robots.txtis where the conversation starts. Enforcement happens at the edge.- Centinel identifies 1,600+ AI agent fingerprints in real time. It applies TLS and HTTP/2 signal checks that survive user agent spoofing. It runs block, verify, or monetize decisions per agent in under 2ms. That is the layer between you and the 7,851% of new traffic that did not ask permission.