What is an AI crawler? Types and examples

An AI crawler is an automated web client that collects or retrieves content for an AI system. It may gather pages that could be used in model development, build an index for AI search, or keep a retrieval system current.

The name describes what the operator does with the pages. On the wire an AI crawler is an ordinary HTTP client, the same as a search crawler. It usually sends a user-agent string such as GPTBot/1.4, and that string is a claim until the source address is checked against the operator's published list.

The category overlaps with search. Apple says pages fetched by Applebot for search may also be used to train its foundation models, and Google uses one crawl for Search and for Gemini unless a site opts out. User-triggered fetchers and browser agents add AI-related traffic that follows no crawl schedule.

AI crawler, search crawler, or agent?

One operator can run a client in every row and publish a separate control for each, so record the stated purpose beside the name.

Agentic traffic covers user-triggered retrieval and task-running clients.

How an AI crawler works

1. Discover URLs

The crawler starts with known URLs, sitemaps, feeds, or links found on pages it has already fetched. Site owners may also submit URLs through an operator's product.

2. Fetch a resource

The client sends an HTTP request for HTML, images, PDFs, feeds, JavaScript, or whatever else the operator's job needs. The request usually carries the operator's published user-agent token.

Support for robots.txt, crawl-rate settings, caching, and page-level directives differs by operator, so read each one's current documentation.

3. Parse or render the response

A basic fetcher can read the returned document and extract text and links. A crawler that needs client-rendered content may also execute JavaScript and load supporting assets. Access logs can show those additional requests, but they do not reveal the crawler's complete parsing pipeline.

4. Store or use the result

The operator may place the result in a search index, a retrieval system, or a collection used for model development. This step happens outside the site's infrastructure.

5. Revisit

Crawlers return to find new pages or refresh old ones. Cache validators such as ETag and Last-Modified can reduce the amount of unchanged content transferred when a crawler supports them. Google documents its use of both in its crawler overview.

Current AI crawler reference

This table was checked against operator documentation on October 6, 2026. It leaves out names whose purpose could not be sourced. Meta, Amazon, and Common Crawl also document crawlers; the AI crawler list has more names.

Sources: OpenAI, Anthropic, Perplexity, Google, and Apple.

ChatGPT-User, Claude-User, and Perplexity-User are left out of the table because their operators describe them as acting on a user's request. OpenAI and Perplexity also say robots.txt rules may not apply to them.

What appears in access logs

A GPTBot request in a combined-format access log looks like this (the address is a documentation placeholder):

203.0.113.7 - - [06/Oct/2026:09:14:02 +0000] "GET /pricing HTTP/1.1" 200 18432 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot"

Nothing in the line proves the sender. The user agent is the string OpenAI documents, and anyone can send it. The address is the only field to check, and only if your proxy or CDN recorded it. To make a line like this usable, retain:

• time, method, host, path, query, status, and response bytes;
• the complete user-agent header;
• the source IP observed by trusted infrastructure;
• request and trace identifiers;
• cache status and origin timing where available;
• the result of any identity or automation check;
• the policy action and its reason.

Verify named traffic against the method its operator publishes before you attach an allow rule to the name. How to verify AI agents explains the IP, DNS, and signature options.

Copying is common on sites the real crawler rarely visits. Our crawler verification data covers news and media sites, and on the typical site in it 84% of requests claiming to be GPTBot failed OpenAI's published check. Any request that sends Google-Extended as its user agent is fake, because Google never sends that token. A log search for Google-Extended or Applebot-Extended therefore counts no real crawler.

A request with an empty or generic user agent has no name to check. Connection and behavioral signals may establish that it is automated, but its operator and purpose can stay unknown.

What logs cannot establish

Server logs cannot directly show whether served content later entered a training dataset, appeared in a search index, grounded an answer, or was discarded. That processing happens after the response leaves the site.

Logs also do not prove compliance with a published purpose. They can show which rules were presented, which routes were requested, how much data was returned, and whether a verified client stayed within the site's access policy. Keep those observations separate from claims about downstream use.

Effects a site owner can measure

Measure effects that appear at your own boundary:

Measure referrals from AI search products separately in analytics.

Build a crawler inventory before setting policy

Start with a short inventory of clients that already appear in your logs.

1. Extract claimed crawler tokens and the routes they request.
2. Create one record per client and purpose.
3. Store the official documentation URL and last review date.
4. Record whether the identifier is an HTTP user agent or only a control token.
5. Add the published verification method.
6. Measure requests, bytes, cache behavior, and restricted-route attempts.
7. Assign an explicit policy owner and review date.

A static "all AI bots" list goes stale. OpenAI's documentation has moved to a new address, and tokens such as Claude-Web that older lists still carry are absent from Anthropic's current page. Review each record against the operator's page on the date you set. How to audit AI crawler traffic covers the measurement in detail.

Once the policy is clear, How to block AI crawlers covers the robots.txt group, the request rule, and the tests. For different responses by identity and resource, use How to control AI agent access.

FAQ

Is Googlebot an AI crawler?

Googlebot is Google's search crawler. Google can also use what it crawls for Gemini training and grounding, and the Google-Extended token in robots.txt opts a site out of that use without affecting Search.

Do AI crawlers obey robots.txt?

OpenAI, Anthropic, Google, Apple, Meta, and Amazon each say their automatic crawlers follow it. The protocol is voluntary, a client that copies a crawler's name is not bound by it, and several operators exempt user-triggered fetchers.

Does an AI crawler use AI to crawl?

No. The name refers to where the content goes. The fetching is ordinary HTTP, and some crawlers also render the page in a browser. Apple says Applebot may.

Finding the crawlers on your own site

Centinel helps teams identify and control automated traffic, including AI crawlers and scrapers. For an inventory of the automation reaching your current pages, request a site audit.

Sources

• OpenAI crawler documentation
• Anthropic crawler documentation
• Perplexity crawler documentation
• Google crawler overview
• Google's common crawlers
• Applebot documentation