What is agentic traffic?

Agentic traffic is web traffic created by software that chooses and performs steps toward a goal. A user might ask an agent to compare products, research a topic, or complete a workflow. The agent then decides which pages or APIs to visit and what to do next.

That definition excludes a lot of traffic commonly placed in the same bucket. A crawler collecting pages on a schedule is automated, but it is not acting on one user's live task. A search bot building an index is not an agentic session either. The distinction matters because these clients create different risks and need different access rules.

What agentic traffic means

There is no HTTP header that turns a request into agentic traffic. The useful definition is operational:

Agentic traffic is one or more web requests made by software that can select its next action while working toward a user or application goal.

The sequence may include page navigation, JavaScript execution, API calls, or state-changing actions such as submitting a form. It may also stop after one request if the task is simple. What makes it agentic is the decision loop, not the number of pages loaded or the presence of a large language model.

Publishers will also see requests that are related to AI products but are not agentic in this sense. OpenAI, Anthropic, and Perplexity each document separate clients for different jobs:

• model development or training;
• search and indexing;
• retrieval initiated by a user.

Those are useful distinctions even when two clients come from the same company. OpenAI states that GPTBot may collect training content, OAI-SearchBot supports ChatGPT search, and ChatGPT-User performs certain user-triggered actions. OpenAI crawler documentation

Anthropic similarly separates ClaudeBot, Claude-SearchBot, and Claude-User. Anthropic crawler documentation

Classify the trigger before the client

Start with why the request happened. The client name comes second.

Was the request triggered by a live user or application task?
|
+-- No or not known
|   |
|   +-- Building a search index?            -> search crawler
|   +-- Collecting model-development data?  -> training crawler
|   +-- Purpose cannot be established?      -> unidentified automation
|
+-- Yes
    |
    +-- Fetching content for an answer?      -> user-directed retrieval
    +-- Choosing and performing steps?       -> agentic session

This decision tree is deliberately conservative. If the trigger cannot be established, record it as unknown. Do not turn a guess about intent into a trusted identity.

The same distinction works as a traffic matrix:

The examples were checked on September 4, 2026. Treat operator documentation as maintained data, because names and stated purposes can change.

Perplexity's documentation is especially explicit: PerplexityBot supports search and indexing, while Perplexity-User may fetch a page after a user asks a question. Perplexity crawler documentation

One easy trap is Google-Extended. Google documents it as a robots.txt product token that controls certain uses of content already crawled by Google. It does not have its own HTTP request user agent. You cannot find Google-Extended traffic by searching access logs for that string. Google's crawler documentation

What the classes look like in logs

Logs show requests, not motives. A single row can tell you the claimed client, source address, route, method, status, time, and response size. It usually cannot tell you whether a human prompt, a scheduler, or another application caused the request.

Collect enough context to classify a sequence later:

• the source IP observed at a trusted network boundary;
• the full user-agent header and any verified operator identity;
• timestamp, method, host, path, status, and bytes transferred;
• a request or trace identifier;
• an authenticated account or first-party session identifier, when present;
• connection and browser signals available at your edge;
• the classification result, reason, and rule version.

Do not build sessions from IP addresses alone. Corporate gateways, mobile networks, privacy relays, and proxy services can place unrelated users behind one address. The reverse is also possible: one automated workflow may move between addresses.

A declared search or training crawler is the simplest case to label, once its identity has been verified. Named user fetchers provide evidence about the trigger too, because their operators publish their purpose. An ordinary browser session is harder. It might be a person, a browser agent, a test runner, or a scraper using browser automation.

What a user agent cannot tell you

A user-agent header is a claim supplied by the client. It can name a documented operator, but it cannot prove that operator sent the request. It also cannot prove that the request behaved according to the operator's stated purpose.

This creates two separate fields in a useful traffic record:

Keep both fields. A verified operator can run several types of client. An unverified request can imitate any published user agent. The verification methods and failure states are covered in How to verify AI agents.

Absence of a named AI user agent proves even less. Agents can use direct HTTP clients, browser automation, or integrations that identify only the software stack. Detection may establish that a session is automated without revealing who operates it or why it exists.

How to recognize an agentic session

Look at the sequence and the actions it attempts. Agentic traffic often moves through a task-shaped path: discover a page, inspect details, follow a relevant link, then call an API or submit data. The exact path depends on the delegated goal.

None of these signals is conclusive on its own:

• route transitions that follow content or application state;
• a switch from read-only requests to state-changing methods;
• browser execution followed by an API request;
• repeated attempts after validation or permission errors;
• timing and navigation patterns inconsistent with the site's human sessions.

The point is not to invent an "agent score" from five weak clues. Use connection, browser, request, account, and behavioral evidence to decide whether the client is automated. Use the request sequence to describe what the automation is doing. Leave the operator and trigger unknown when the evidence does not establish them.

This also prevents a common classification mistake. Unidentified automation is not automatically agentic. It may be a price scraper, uptime monitor, SEO tool, fraud bot, test runner, or a custom agent. Label what the evidence supports.

Match the response to the traffic class

Classification narrows the policy question. It does not answer it for every publisher or every route.

Verification does not override authorization. A genuine user-directed fetcher does not gain access to a subscriber-only article without the required credentials. An authenticated agent should not be able to submit an action its account is not permitted to perform.

The policy may allow, observe, rate-limit, challenge, or block a request. Make that decision using the resource, identity result, behavior, and business rule together. How to control AI agent access covers that layer in detail.

Measure it without one misleading "AI traffic" number

Putting every AI-related request into one chart hides the part operators need to understand. Report traffic class and identity status separately.

A useful publisher report includes:

• requests and sessions by traffic class;
• verified, failed, unknown, and unidentified identity counts;
• pages and bytes served by route group;
• cache hits, cache misses, response status, and origin work where available;
• state-changing attempts and their authorization result;
• policy actions and reason codes;
• the classifier version used for the report.

Keep raw observations separate from inferred labels. If a classification rule changes, versioning lets you reprocess historical data or at least explain why the chart moved. Otherwise a taxonomy update can look like a traffic surge.

For scheduled crawler mechanics, see What is an AI crawler?. For browser-controlled automation, see How to detect browser automation.

Where Centinel fits

Centinel helps teams identify and control automated traffic. A useful evaluation should test whether the integration can keep operator identity, traffic class, and policy outcome separate. It should also show what happens when a request cannot be classified or verified.

The bot management guide turns those checks into a vendor-evaluation and deployment checklist.

If you want to establish which classes are reaching your current pages before setting policy, request a site audit.

Sources

• OpenAI crawler documentation
• Anthropic crawler documentation
• Perplexity crawler documentation
• Google's common crawler documentation