AI crawlers take your content to train or run AI models, not to send you readers.
What it is
An AI crawler is an automated program that visits websites to collect data. That data trains or runs AI models. Search engine crawlers like Googlebot work differently, because they index pages to serve search results. AI crawlers harvest content instead. They build large language models, image generators, and other AI systems.
The request looks normal, and it arrives over HTTP like any other. The difference is what happens next. A reader keeps the content, but a crawler feeds it to a model. A Googlebot crawl indexes a page so a person can find it. A GPTBot crawl feeds a model that then answers the person instead.
Why AI crawlers matter now
The content AI crawlers take has real value. Your articles, product pages, and research can train a model. When they do, you get nothing back: no payment, no credit, no traffic. The model then competes with you, and it answers the same questions your content answers.
Our data has the numbers. As of early 2026, AI bots reach 39% of the top one million websites. Only 2.98% of those sites block them. Anthropic's crawl-to-referral ratio through 2025 was about 500,000 to 1. That means half a million pages fetched for every visitor sent back.
AI agents now sit between your content and the reader. Someone reads your article inside ChatGPT, Perplexity summarizes it, and an AI Overview cites it. None of that shows an ad or builds a reader relationship. The traffic is real, but the way to earn from it is not.
Types of AI crawlers
Four kinds show up in server logs. Each takes different data for a different reason.
Training crawlers. These sweep broadly to build training sets. Examples include GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended, Bytespider, Applebot-Extended, and CCBot. They pull text, code, structured data, images, and comments. They are the clearest candidates for licensing.
Retrieval and grounding crawlers. These fetch pages at query time, and the page grounds a model's answer. Examples include PerplexityBot, OAI-SearchBot, and ChatGPT-User. They act like search indexers, but they do not send referral traffic the way Googlebot does.
Agentic traffic. An AI agent creates this for one human user. A ChatGPT agent may check flight prices, or a Claude agent may research a paper. The client is usually a headless browser on cloud infrastructure, and it often routes through residential proxies.
Unlabeled and spoofed crawlers. This is the largest and messiest group. It includes cohere-ai, Meta-ExternalAgent, and commercial services like BrightData, Oxylabs, and ScraperAPI. Smaller operators rotate through residential IPs. Some target paywalled content to bypass access controls.
How AI crawlers work
An AI crawler is an HTTP client. Each request carries a user agent, a TLS handshake, a set of HTTP/2 settings, and a body. The software stack and the intent set a crawler apart from a browser.
Training crawlers are the simplest. A scheduler runs, a fetcher opens an HTTP connection, and a parser pulls text and links. The results go into a dataset. GPTBot and ClaudeBot publish their IP ranges, and they respect robots.txt in most cases. Their footprint is predictable, because the user agent, TLS fingerprint, and cadence all stay consistent.
Retrieval crawlers are stateful and bursty. Query volume drives them, not a schedule. Agentic traffic is the hardest to pin down. It may use patched Chromium, a headless browser, or a direct HTTP client. The choice depends on the task, and it often runs through residential proxies.
How to identify AI crawlers on your site
Start with user agents, but do not stop there. Major operators publish their strings, which include GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, PerplexityBot, and CCBot. Match them in your logs and you find the compliant traffic. That traffic is the least likely to cause trouble.
The rest needs signals the client cannot fully control. TLS fingerprinting (JA4) exposes the library behind the handshake. HTTP/2 SETTINGS frames separate browsers from libraries by the WINDOW_UPDATE value and pseudo-header order. Behavioral patterns split readers from crawlers. Cross-layer consistency is the deciding check.
Take one example. A request claims Chrome in its user agent. Its TLS fingerprint matches curl-impersonate. Its HTTP/2 settings match a Go library. That request is a crawler that lied twice. Centinel maintains 1,600+ crawler fingerprints that combine these layers.
How to respond to AI crawler traffic
Once you identify the traffic, you have three responses. Pick per agent, not per source.
Block. Block training crawlers you have not licensed. Block scrapers that ignore robots.txt. Block spoofed traffic that fails consistency checks. Block at the edge so the origin never sees the request.
Verify and allow. Allow search indexers you want to appear in. Allow partner agents. Allow AI-on-behalf-of-user traffic you want through but want to audit. Pass the request with a signed trust stamp, then watch cumulative volume per operator.
Watchlist. Use this for training crawlers you have not decided on yet. Centinel records every visit per agent, so you get the audit trail to act later. You can block, challenge, or escalate once you set policy.
robots.txt alone will not do any of this. 32% of AI scrapes bypass it. Enforcement lives at the edge.
Key takeaways
- AI crawlers are their own category, not search engine bots. They take content to train or ground models that compete with your site instead of sending traffic back.
- Four classes matter: training, retrieval, agentic, and unlabeled. Each needs a different policy.
- User agents alone cannot identify crawlers reliably. TLS fingerprinting,
HTTP/2signals, and cross-layer consistency checks catch the 32% that bypassrobots.txt. - Your response is block, verify, or watchlist. It is a per-agent decision enforced at the edge, not one site-wide setting.