# Centinel Analytica: full text
> Every bot record, tool, guide and comparison on https://www.centinelanalytica.com as Markdown. https://www.centinelanalytica.com/llms.txt is the index.
---
Page: https://www.centinelanalytica.com/bots
# Bot directory
55 crawlers and fetchers that reach public websites: who runs each one, what it does, and its robots.txt token. Last checked 2026-10-10.
## AI training and dataset crawlers
Crawl the web for content that may be used to train models or to build a reusable dataset.
| Bot | Operator | What it does | robots.txt token |
| --- | --- | --- | --- |
| [Amazonbot](https://www.centinelanalytica.com/bots/amazonbot) | Amazon | Crawls to improve Amazon products; may be used to train Amazon AI models. | `Amazonbot` |
| [CCBot](https://www.centinelanalytica.com/bots/ccbot) | Common Crawl | Builds Common Crawl's open web archive, free for anyone to reuse. | `CCBot` |
| [ClaudeBot](https://www.centinelanalytica.com/bots/claudebot) | Anthropic | Collects web content that could contribute to Claude model training. | `ClaudeBot` |
| [GPTBot](https://www.centinelanalytica.com/bots/gptbot) | OpenAI | Crawls content that may be used to train OpenAI's foundation models. | `GPTBot` |
| [Meta-ExternalAgent](https://www.centinelanalytica.com/bots/meta-externalagent) | Meta | Crawls for training AI models or improving Meta products. | `meta-externalagent` |
## AI search crawlers
Index pages so an AI assistant can find, cite and link them in its answers.
| Bot | Operator | What it does | robots.txt token |
| --- | --- | --- | --- |
| [Claude-SearchBot](https://www.centinelanalytica.com/bots/claude-searchbot) | Anthropic | Indexes pages to improve Claude's search results. | `Claude-SearchBot` |
| [DuckAssistBot](https://www.centinelanalytica.com/bots/duckassistbot) | DuckDuckGo | Crawls in real time to support DuckDuckGo's AI-assisted answers. | `DuckAssistBot` |
| [Meta-WebIndexer](https://www.centinelanalytica.com/bots/meta-webindexer) | Meta | Crawls pages to improve the search results of Meta AI. | `meta-webindexer` |
| [MistralAI-Index](https://www.centinelanalytica.com/bots/mistralai-index) | Mistral AI | Indexes pages for Mistral search. | `MistralAI-Index` |
| [OAI-SearchBot](https://www.centinelanalytica.com/bots/oai-searchbot) | OpenAI | Surfaces websites in ChatGPT's search features. | `OAI-SearchBot` |
| [PerplexityBot](https://www.centinelanalytica.com/bots/perplexitybot) | Perplexity | Surfaces and links websites in Perplexity's search results. | `PerplexityBot` |
## AI user-triggered fetchers
Fetch a page because a person asked an assistant to. Several operators say robots.txt may not apply.
| Bot | Operator | What it does | robots.txt token |
| --- | --- | --- | --- |
| [ChatGPT-User](https://www.centinelanalytica.com/bots/chatgpt-user) | OpenAI | Visits a page when a ChatGPT or Custom GPT user's request needs it. | `ChatGPT-User` |
| [Claude-User](https://www.centinelanalytica.com/bots/claude-user) | Anthropic | Fetches a page when a Claude user's question needs it. | `Claude-User` |
| [Meta-ExternalFetcher](https://www.centinelanalytica.com/bots/meta-externalfetcher) | Meta | Fetches individual links on behalf of Meta product users. | `meta-externalfetcher` |
| [MistralAI-User](https://www.centinelanalytica.com/bots/mistralai-user) | Mistral AI | Fetches a page when a Mistral user's action needs it. | `MistralAI-User` |
| [Perplexity-User](https://www.centinelanalytica.com/bots/perplexity-user) | Perplexity | Fetches a page when a Perplexity user's request needs it. | `Perplexity-User` |
## AI robots.txt control tokens
Names you write in robots.txt to control AI use. No crawler sends them as a user agent.
| Bot | Operator | What it does | robots.txt token |
| --- | --- | --- | --- |
| [Applebot-Extended](https://www.centinelanalytica.com/bots/applebot-extended) | Apple | A robots.txt token that opts content out of Apple model training. | `Applebot-Extended` |
| [Google-Extended](https://www.centinelanalytica.com/bots/google-extended) | Google | A robots.txt token that controls Gemini training and grounding use. | `Google-Extended` |
## Search engine crawlers
Build the indexes behind web search results.
| Bot | Operator | What it does | robots.txt token |
| --- | --- | --- | --- |
| [360Spider](https://www.centinelanalytica.com/bots/360spider) | 360 Search | Crawls pages for the web search of 360 Search (so.com). | `360Spider` |
| [Applebot](https://www.centinelanalytica.com/bots/applebot) | Apple | Crawls pages for Spotlight, Siri and Safari search features. | `Applebot` |
| [Baiduspider](https://www.centinelanalytica.com/bots/baiduspider) | Baidu | Crawls pages for Baidu search. | `Baiduspider` |
| [Bingbot](https://www.centinelanalytica.com/bots/bingbot) | Microsoft | Crawls pages for Bing search. | `bingbot` |
| [Diffbot](https://www.centinelanalytica.com/bots/diffbot) | Diffbot | Crawls the web for the Diffbot Knowledge Graph and Diffbot's web search services. | `Diffbot` |
| [DuckDuckBot](https://www.centinelanalytica.com/bots/duckduckbot) | DuckDuckGo | Crawls pages for DuckDuckGo search. | `DuckDuckBot` |
| [Googlebot-Image](https://www.centinelanalytica.com/bots/googlebot-image) | Google | Crawls images for Google Images and image features in Search. | `Googlebot-Image` |
| [Googlebot-News](https://www.centinelanalytica.com/bots/googlebot-news) | Google | Controls whether your pages appear in Google News. | `Googlebot-News` |
| [Googlebot-Video](https://www.centinelanalytica.com/bots/googlebot-video) | Google | Crawls video for the video features of Google Search. | `Googlebot-Video` |
| [Googlebot](https://www.centinelanalytica.com/bots/googlebot) | Google | Crawls pages for Google Search, Images, Video, News and Discover. | `Googlebot` |
| [Kagibot](https://www.centinelanalytica.com/bots/kagibot) | Kagi | Crawls pages for the Kagi search engine. | `Kagibot` |
| [PetalBot](https://www.centinelanalytica.com/bots/petalbot) | Huawei | Crawls pages for Petal Search and Huawei Assistant. | `PetalBot` |
| [SeznamBot](https://www.centinelanalytica.com/bots/seznambot) | Seznam.cz | Crawls pages for Seznam, a Czech search engine. | `SeznamBot` |
| [YandexBot](https://www.centinelanalytica.com/bots/yandexbot) | Yandex | Yandex's main search indexing crawler. | `YandexBot` |
| [Yeti](https://www.centinelanalytica.com/bots/yeti) | Naver | Crawls pages for Naver search in South Korea. | `Yeti` |
| [YouBot](https://www.centinelanalytica.com/bots/youbot) | You.com | Crawls and indexes pages for the You.com search engine. | `YouBot` |
## Other Google crawlers
Single-purpose Google crawlers for shopping, ads, testing tools and Vertex AI.
| Bot | Operator | What it does | robots.txt token |
| --- | --- | --- | --- |
| [AdsBot-Google](https://www.centinelanalytica.com/bots/adsbot-google) | Google | Checks the quality of ad landing pages for Google Ads. | `AdsBot-Google` |
| [Google-CloudVertexBot](https://www.centinelanalytica.com/bots/google-cloudvertexbot) | Google | Crawls sites whose owners asked for it when building Vertex AI Agents. | `Google-CloudVertexBot` |
| [Google-InspectionTool](https://www.centinelanalytica.com/bots/google-inspectiontool) | Google | Fetches pages for Search Console URL inspection and the Rich Results Test. | `Google-InspectionTool` |
| [GoogleOther](https://www.centinelanalytica.com/bots/googleother) | Google | Fetches public pages for Google product teams, for example for research crawls. | `GoogleOther` |
| [Mediapartners-Google](https://www.centinelanalytica.com/bots/mediapartners-google) | Google | Reads the pages of sites that show Google ads, so the ads match the content. | `Mediapartners-Google` |
| [Storebot-Google](https://www.centinelanalytica.com/bots/storebot-google) | Google | Crawls product and store pages for Google Shopping. | `Storebot-Google` |
## SEO and link-index crawlers
Build backlink and site-audit databases for SEO tools. None of them affects search rankings.
| Bot | Operator | What it does | robots.txt token |
| --- | --- | --- | --- |
| [AhrefsBot](https://www.centinelanalytica.com/bots/ahrefsbot) | Ahrefs | Builds the link database behind Ahrefs and the Yep search engine. | `AhrefsBot` |
| [AhrefsSiteAudit](https://www.centinelanalytica.com/bots/ahrefssiteaudit) | Ahrefs | Crawls a site for an Ahrefs Site Audit project. | `AhrefsSiteAudit` |
| [Barkrowler](https://www.centinelanalytica.com/bots/barkrowler) | Babbar | Collects links and page metadata for Babbar. | `barkrowler` |
| [DataForSeoBot](https://www.centinelanalytica.com/bots/dataforseobot) | DataForSEO | Adds links to DataForSEO's backlink database. | `DataForSeoBot` |
| [DotBot](https://www.centinelanalytica.com/bots/dotbot) | Moz | Gathers data for the Moz Link Index. | `dotbot` |
| [MJ12bot](https://www.centinelanalytica.com/bots/mj12bot) | Majestic | Maps links between websites for Majestic's link index. | `MJ12bot` |
| [Rogerbot](https://www.centinelanalytica.com/bots/rogerbot) | Moz | Crawls a site for the site audits of a Moz Pro Campaign. | `rogerbot` |
| [Screaming Frog SEO Spider](https://www.centinelanalytica.com/bots/screaming-frog-seo-spider) | Screaming Frog | Audits a site's technical SEO from the machine that runs it. | `Screaming Frog SEO Spider` |
| [SemrushBot](https://www.centinelanalytica.com/bots/semrushbot) | Semrush | Collects web data for Semrush's SEO tools. | `SemrushBot` |
## Link preview bots
Fetch a page when someone shares its link, to build the preview card.
| Bot | Operator | What it does | robots.txt token |
| --- | --- | --- | --- |
| [Discordbot](https://www.centinelanalytica.com/bots/discordbot) | Discord | Visits a link shared in a Discord message to build its preview. | `Discordbot` |
| [facebookexternalhit](https://www.centinelanalytica.com/bots/facebookexternalhit) | Meta | Builds link previews when content is shared on Meta's apps. | `facebookexternalhit` |
| [Pinterestbot](https://www.centinelanalytica.com/bots/pinterestbot) | Pinterest | Indexes pages behind Pins and keeps Pin details up to date. | `Pinterestbot` |
| [Slackbot-LinkExpanding](https://www.centinelanalytica.com/bots/slackbot-linkexpanding) | Slack | Fetches a page's meta tags when someone posts its link in Slack. | `Slackbot-LinkExpanding` |
| [Twitterbot](https://www.centinelanalytica.com/bots/twitterbot) | X | Fetches card markup when a link is shared on X. | `Twitterbot` |
## Crawlers without operator documentation
The operator publishes no page describing the crawler, so its identity cannot be checked.
| Bot | Operator | What it does | robots.txt token |
| --- | --- | --- | --- |
| [Bytespider](https://www.centinelanalytica.com/bots/bytespider) | Unconfirmed, attributed to ByteDance | A crawler with no operator documentation. | `Bytespider` |
## Bots by operator
- [Google](https://www.centinelanalytica.com/bots/operator/google) (11)
- [OpenAI](https://www.centinelanalytica.com/bots/operator/openai) (3)
- [Anthropic](https://www.centinelanalytica.com/bots/operator/anthropic) (3)
- [Meta](https://www.centinelanalytica.com/bots/operator/meta) (4)
---
Page: https://www.centinelanalytica.com/ai-crawler-list
# AI crawler list
Facts come from each operator's own documentation, last verified 2026-10-10.
## AI training and dataset crawlers
Crawl the web for content that may be used to train models or to build a reusable dataset.
| Bot | Operator | What it does | robots.txt token |
| --- | --- | --- | --- |
| [Amazonbot](https://www.centinelanalytica.com/bots/amazonbot) | Amazon | Crawls to improve Amazon products; may be used to train Amazon AI models. | `Amazonbot` |
| [CCBot](https://www.centinelanalytica.com/bots/ccbot) | Common Crawl | Builds Common Crawl's open web archive, free for anyone to reuse. | `CCBot` |
| [ClaudeBot](https://www.centinelanalytica.com/bots/claudebot) | Anthropic | Collects web content that could contribute to Claude model training. | `ClaudeBot` |
| [GPTBot](https://www.centinelanalytica.com/bots/gptbot) | OpenAI | Crawls content that may be used to train OpenAI's foundation models. | `GPTBot` |
| [Meta-ExternalAgent](https://www.centinelanalytica.com/bots/meta-externalagent) | Meta | Crawls for training AI models or improving Meta products. | `meta-externalagent` |
## AI search crawlers
Index pages so an AI assistant can find, cite and link them in its answers.
| Bot | Operator | What it does | robots.txt token |
| --- | --- | --- | --- |
| [Claude-SearchBot](https://www.centinelanalytica.com/bots/claude-searchbot) | Anthropic | Indexes pages to improve Claude's search results. | `Claude-SearchBot` |
| [DuckAssistBot](https://www.centinelanalytica.com/bots/duckassistbot) | DuckDuckGo | Crawls in real time to support DuckDuckGo's AI-assisted answers. | `DuckAssistBot` |
| [Meta-WebIndexer](https://www.centinelanalytica.com/bots/meta-webindexer) | Meta | Crawls pages to improve the search results of Meta AI. | `meta-webindexer` |
| [MistralAI-Index](https://www.centinelanalytica.com/bots/mistralai-index) | Mistral AI | Indexes pages for Mistral search. | `MistralAI-Index` |
| [OAI-SearchBot](https://www.centinelanalytica.com/bots/oai-searchbot) | OpenAI | Surfaces websites in ChatGPT's search features. | `OAI-SearchBot` |
| [PerplexityBot](https://www.centinelanalytica.com/bots/perplexitybot) | Perplexity | Surfaces and links websites in Perplexity's search results. | `PerplexityBot` |
## AI user-triggered fetchers
Fetch a page because a person asked an assistant to. Several operators say robots.txt may not apply.
| Bot | Operator | What it does | robots.txt token |
| --- | --- | --- | --- |
| [ChatGPT-User](https://www.centinelanalytica.com/bots/chatgpt-user) | OpenAI | Visits a page when a ChatGPT or Custom GPT user's request needs it. | `ChatGPT-User` |
| [Claude-User](https://www.centinelanalytica.com/bots/claude-user) | Anthropic | Fetches a page when a Claude user's question needs it. | `Claude-User` |
| [Meta-ExternalFetcher](https://www.centinelanalytica.com/bots/meta-externalfetcher) | Meta | Fetches individual links on behalf of Meta product users. | `meta-externalfetcher` |
| [MistralAI-User](https://www.centinelanalytica.com/bots/mistralai-user) | Mistral AI | Fetches a page when a Mistral user's action needs it. | `MistralAI-User` |
| [Perplexity-User](https://www.centinelanalytica.com/bots/perplexity-user) | Perplexity | Fetches a page when a Perplexity user's request needs it. | `Perplexity-User` |
## AI robots.txt control tokens
Names you write in robots.txt to control AI use. No crawler sends them as a user agent.
| Bot | Operator | What it does | robots.txt token |
| --- | --- | --- | --- |
| [Applebot-Extended](https://www.centinelanalytica.com/bots/applebot-extended) | Apple | A robots.txt token that opts content out of Apple model training. | `Applebot-Extended` |
| [Google-Extended](https://www.centinelanalytica.com/bots/google-extended) | Google | A robots.txt token that controls Gemini training and grounding use. | `Google-Extended` |
## Opt out of AI training, stay in AI search
This group refuses the training crawlers and control tokens above and leaves the AI search crawlers alone. Check each bot's page before you copy it: some names also feed products you may want.
```
User-agent: Amazonbot
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: GPTBot
User-agent: meta-externalagent
Disallow: /
```
robots.txt only asks. Several user-triggered fetchers say they may not follow it, and anyone can send a crawler name.
---
Page: https://www.centinelanalytica.com/tools
# Tools
| Tool | What it does | Where your input goes |
| --- | --- | --- |
| [Crawler IP verifier](https://www.centinelanalytica.com/verify-crawler) | Checks an address from your logs against the reverse DNS and published address ranges Google, OpenAI, Anthropic and Perplexity document themselves. | Sent to this site's server, which runs the lookups. |
| [User agent parser](https://www.centinelanalytica.com/user-agent-parser) | Names the bot a user agent string claims to be, with its operator, type and robots.txt token. | Stays in your browser. |
| [robots.txt checker](https://www.centinelanalytica.com/robots-txt-checker) | Reads a site's robots.txt and reports what it says to each AI bot, and whether the site has an llms.txt. | The site name is sent to this site's server, which fetches the two files. |
| [llms.txt generator](https://www.centinelanalytica.com/llms-txt-generator) | Turns a sitemap or a list of page URLs into an llms.txt draft, grouped by section. | Stays in your browser. |
---
Page: https://www.centinelanalytica.com/verify-crawler
# Crawler IP verifier
Checks an address from your logs against the reverse DNS and published address ranges Google, OpenAI, Anthropic and Perplexity document themselves.
- Use it at: https://www.centinelanalytica.com/verify-crawler
- Where your input goes: Sent to this site's server, which runs the lookups.
---
Page: https://www.centinelanalytica.com/user-agent-parser
# User agent parser
Names the bot a user agent string claims to be, with its operator, type and robots.txt token.
- Use it at: https://www.centinelanalytica.com/user-agent-parser
- Where your input goes: Stays in your browser.
---
Page: https://www.centinelanalytica.com/robots-txt-checker
# robots.txt checker
Reads a site's robots.txt and reports what it says to each AI bot, and whether the site has an llms.txt.
- Use it at: https://www.centinelanalytica.com/robots-txt-checker
- Where your input goes: The site name is sent to this site's server, which fetches the two files.
---
Page: https://www.centinelanalytica.com/llms-txt-generator
# llms.txt generator
Turns a sitemap or a list of page URLs into an llms.txt draft, grouped by section.
- Use it at: https://www.centinelanalytica.com/llms-txt-generator
- Where your input goes: Stays in your browser.
---
Page: https://www.centinelanalytica.com/bots/operator/google
# Google crawlers
This directory covers 11 bots that Google runs. Each has its own robots.txt token, so a site can allow one and disallow another. Last checked against Google's documentation on 2026-10-10.
| Bot | Type | What it does | robots.txt token |
| --- | --- | --- | --- |
| [AdsBot-Google](https://www.centinelanalytica.com/bots/adsbot-google) | Other Google crawlers | Checks the quality of ad landing pages for Google Ads. | `AdsBot-Google` |
| [Google-CloudVertexBot](https://www.centinelanalytica.com/bots/google-cloudvertexbot) | Other Google crawlers | Crawls sites whose owners asked for it when building Vertex AI Agents. | `Google-CloudVertexBot` |
| [Google-Extended](https://www.centinelanalytica.com/bots/google-extended) | AI robots.txt control tokens | A robots.txt token that controls Gemini training and grounding use. | `Google-Extended` |
| [Google-InspectionTool](https://www.centinelanalytica.com/bots/google-inspectiontool) | Other Google crawlers | Fetches pages for Search Console URL inspection and the Rich Results Test. | `Google-InspectionTool` |
| [Googlebot-Image](https://www.centinelanalytica.com/bots/googlebot-image) | Search engine crawlers | Crawls images for Google Images and image features in Search. | `Googlebot-Image` |
| [Googlebot-News](https://www.centinelanalytica.com/bots/googlebot-news) | Search engine crawlers | Controls whether your pages appear in Google News. | `Googlebot-News` |
| [Googlebot-Video](https://www.centinelanalytica.com/bots/googlebot-video) | Search engine crawlers | Crawls video for the video features of Google Search. | `Googlebot-Video` |
| [Googlebot](https://www.centinelanalytica.com/bots/googlebot) | Search engine crawlers | Crawls pages for Google Search, Images, Video, News and Discover. | `Googlebot` |
| [GoogleOther](https://www.centinelanalytica.com/bots/googleother) | Other Google crawlers | Fetches public pages for Google product teams, for example for research crawls. | `GoogleOther` |
| [Mediapartners-Google](https://www.centinelanalytica.com/bots/mediapartners-google) | Other Google crawlers | Reads the pages of sites that show Google ads, so the ads match the content. | `Mediapartners-Google` |
| [Storebot-Google](https://www.centinelanalytica.com/bots/storebot-google) | Other Google crawlers | Crawls product and store pages for Google Shopping. | `Storebot-Google` |
## How to verify a request from Google
- AdsBot-Google and Mediapartners-Google: Run a reverse DNS lookup on the source IP and check that the name ends in google.com (Google uses rate-limited-proxy-….google.com for special-case crawlers), then run a forward lookup on that name and confirm it returns the same IP. Or check the IP against Google's special-case crawler ranges. https://developers.google.com/static/crawling/ipranges/special-crawlers.json
- Google-CloudVertexBot, Google-InspectionTool, Googlebot-Image, Googlebot-Video, Googlebot, GoogleOther and Storebot-Google: Run a reverse DNS lookup on the source IP and check that the name ends in googlebot.com (Google uses crawl-….googlebot.com and geo-crawl-….geo.googlebot.com), then run a forward lookup on that name and confirm it returns the same IP. Or check the IP against Google's common crawler ranges. A googleusercontent.com name does not count: any Google Cloud virtual machine can have one. https://developers.google.com/static/crawling/ipranges/common-crawlers.json
- Google-Extended: There is nothing to verify: no Google crawler sends Google-Extended as its user agent. A request that does is not from Google.
- Googlebot-News: No request carries Googlebot-News as its user agent. The crawl comes from Googlebot, so verify it as Googlebot: run a reverse DNS lookup on the source IP, check that the name ends in googlebot.com, then run a forward lookup on that name and confirm it returns the same IP. Or check the IP against Google's common crawler ranges. https://developers.google.com/static/crawling/ipranges/common-crawlers.json
## Sources
- https://developers.google.com/search/docs/crawling-indexing/google-special-case-crawlers
- https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers
- https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers
- https://developers.google.com/crawling/docs/crawlers-fetchers/google-special-case-crawlers
---
Page: https://www.centinelanalytica.com/bots/operator/openai
# OpenAI crawlers
This directory covers 3 bots that OpenAI runs. Each has its own robots.txt token, so a site can allow one and disallow another. Last checked against OpenAI's documentation on 2026-09-30.
| Bot | Type | What it does | robots.txt token |
| --- | --- | --- | --- |
| [ChatGPT-User](https://www.centinelanalytica.com/bots/chatgpt-user) | AI user-triggered fetchers | Visits a page when a ChatGPT or Custom GPT user's request needs it. | `ChatGPT-User` |
| [GPTBot](https://www.centinelanalytica.com/bots/gptbot) | AI training and dataset crawlers | Crawls content that may be used to train OpenAI's foundation models. | `GPTBot` |
| [OAI-SearchBot](https://www.centinelanalytica.com/bots/oai-searchbot) | AI search crawlers | Surfaces websites in ChatGPT's search features. | `OAI-SearchBot` |
## How to verify a request from OpenAI
- ChatGPT-User: Check that the source IP is in the ranges OpenAI publishes for ChatGPT-User. https://openai.com/chatgpt-user.json
- GPTBot: Check that the source IP is in the ranges OpenAI publishes for GPTBot. https://openai.com/gptbot.json
- OAI-SearchBot: Check that the source IP is in the ranges OpenAI publishes for OAI-SearchBot. https://openai.com/searchbot.json
## Sources
- https://platform.openai.com/docs/bots
---
Page: https://www.centinelanalytica.com/bots/operator/anthropic
# Anthropic crawlers
This directory covers 3 bots that Anthropic runs. Each has its own robots.txt token, so a site can allow one and disallow another. Last checked against Anthropic's documentation on 2026-09-30.
| Bot | Type | What it does | robots.txt token |
| --- | --- | --- | --- |
| [Claude-SearchBot](https://www.centinelanalytica.com/bots/claude-searchbot) | AI search crawlers | Indexes pages to improve Claude's search results. | `Claude-SearchBot` |
| [Claude-User](https://www.centinelanalytica.com/bots/claude-user) | AI user-triggered fetchers | Fetches a page when a Claude user's question needs it. | `Claude-User` |
| [ClaudeBot](https://www.centinelanalytica.com/bots/claudebot) | AI training and dataset crawlers | Collects web content that could contribute to Claude model training. | `ClaudeBot` |
## How to verify a request from Anthropic
- Claude-SearchBot, Claude-User and ClaudeBot: Check that the source IP is in the ranges Anthropic publishes for its crawlers. https://claude.com/crawling/bots.json
## Sources
- https://support.claude.com/en/articles/8896518
---
Page: https://www.centinelanalytica.com/bots/operator/meta
# Meta crawlers
This directory covers 4 bots that Meta runs. Each has its own robots.txt token, so a site can allow one and disallow another. Last checked against Meta's documentation on 2026-10-10.
| Bot | Type | What it does | robots.txt token |
| --- | --- | --- | --- |
| [facebookexternalhit](https://www.centinelanalytica.com/bots/facebookexternalhit) | Link preview bots | Builds link previews when content is shared on Meta's apps. | `facebookexternalhit` |
| [Meta-ExternalAgent](https://www.centinelanalytica.com/bots/meta-externalagent) | AI training and dataset crawlers | Crawls for training AI models or improving Meta products. | `meta-externalagent` |
| [Meta-ExternalFetcher](https://www.centinelanalytica.com/bots/meta-externalfetcher) | AI user-triggered fetchers | Fetches individual links on behalf of Meta product users. | `meta-externalfetcher` |
| [Meta-WebIndexer](https://www.centinelanalytica.com/bots/meta-webindexer) | AI search crawlers | Crawls pages to improve the search results of Meta AI. | `meta-webindexer` |
## How to verify a request from Meta
- facebookexternalhit: Meta's crawler page gives no IP list and no reverse DNS domain. It says to add the user agent strings or the IP addresses used by the crawler to your allow list, and publishes no list of those addresses.
- Meta-ExternalAgent: Meta's crawler page gives no IP list and no reverse DNS domain for Meta-ExternalAgent.
- Meta-ExternalFetcher: Meta's crawler page gives no IP list and no reverse DNS domain for Meta-ExternalFetcher.
- Meta-WebIndexer: Meta's crawler page gives no IP list and no reverse DNS domain for Meta-WebIndexer.
## Sources
- https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/
---
Page: https://www.centinelanalytica.com/bots/360spider
# What is 360Spider?
360Spider is the web search crawler of 360 Search (so.com). 360 Search uses separate user agents for its other products: 360Spider-Image for image search and 360Spider-Video for video search. Its documentation is in Chinese.
- Operator: 360 Search
- Purpose: Crawls pages for the web search of 360 Search (so.com).
- robots.txt token: `360Spider`
- Example user agent: `Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/140.0.0.0 Safari/537.36 Edg/140.0.0.0; 360Spider`
- How to verify: 360 Search lists the IP segments of its spider on its help page and says to use that list, because the spider does not currently support a reverse DNS lookup. Compare the source IP with the segments on the page.
- Operator documentation: [https://www.so.com/help/spider_ip.html](https://www.so.com/help/spider_ip.html)
- Last verified: 2026-10-10
## How often is 360Spider impersonated?
Too few organisations saw requests claiming to be 360Spider in the measurement window to publish a share.
This directory measures requests between 14 July 2026 and 10 October 2026. The [fake crawler report](https://www.centinelanalytica.com/blog/fake-crawler-report) covers its own, earlier window (late June to 21 September 2026), so its figures differ from the ones here. Read it for the method.
## Should you block 360Spider?
- Allow it if: you have visitors who search with 360 Search.
- Block it if: you have no audience in China and the crawl adds load.
- What blocking changes: 360 Search warns site owners not to block its spider by mistake, because that causes indexing or crawling problems in 360 Search.
To block it, add this group to your robots.txt. 360 Search says it supports the main commands of the robots protocol: User-agent, Allow and Disallow.
```
User-agent: 360Spider
Disallow: /
```
robots.txt only asks. It does nothing against a client that ignores it or only pretends to be 360Spider. Stopping those takes verification at your edge.
## Related bots
- [Applebot](https://www.centinelanalytica.com/bots/applebot) (Apple): Crawls pages for Spotlight, Siri and Safari search features.
- [Baiduspider](https://www.centinelanalytica.com/bots/baiduspider) (Baidu): Crawls pages for Baidu search.
- [Bingbot](https://www.centinelanalytica.com/bots/bingbot) (Microsoft): Crawls pages for Bing search.
- [Diffbot](https://www.centinelanalytica.com/bots/diffbot) (Diffbot): Crawls the web for the Diffbot Knowledge Graph and Diffbot's web search services.
- [DuckDuckBot](https://www.centinelanalytica.com/bots/duckduckbot) (DuckDuckGo): Crawls pages for DuckDuckGo search.
- [Googlebot-Image](https://www.centinelanalytica.com/bots/googlebot-image) (Google): Crawls images for Google Images and image features in Search.
---
Page: https://www.centinelanalytica.com/bots/adsbot-google
# What is AdsBot-Google?
AdsBot-Google is the crawler Google Ads uses to check the quality of web pages used as ad landing pages. It is one of Google's special-case crawlers.
- Operator: [Google](https://www.centinelanalytica.com/bots/operator/google)
- Purpose: Checks the quality of ad landing pages for Google Ads.
- robots.txt token: `AdsBot-Google`
- Example user agent: `AdsBot-Google (+http://www.google.com/adsbot.html)`
- How to verify: Run a reverse DNS lookup on the source IP and check that the name ends in google.com (Google uses rate-limited-proxy-….google.com for special-case crawlers), then run a forward lookup on that name and confirm it returns the same IP. Or check the IP against Google's special-case crawler ranges.
- Reverse DNS: google.com
- IP ranges: [https://developers.google.com/static/crawling/ipranges/special-crawlers.json](https://developers.google.com/static/crawling/ipranges/special-crawlers.json)
- Operator documentation: [https://developers.google.com/search/docs/crawling-indexing/google-special-case-crawlers](https://developers.google.com/search/docs/crawling-indexing/google-special-case-crawlers)
- Last verified: 2026-09-30
## How often is AdsBot-Google impersonated?
For the median organisation, under 1% of requests claiming to be AdsBot-Google failed verification.
Measured between 14 July 2026 and 10 October 2026 across organisations protected by Centinel. We only publish a share when at least five organisations each saw enough of these requests. Pooled across all of their requests, the share was under 1%: a few large organisations carry most of the traffic, so the median organisation is the better guide to what yours will see.
This directory measures requests between 14 July 2026 and 10 October 2026. The [fake crawler report](https://www.centinelanalytica.com/blog/fake-crawler-report) covers its own, earlier window (late June to 21 September 2026), so its figures differ from the ones here. Read it for the method.
## Should you block AdsBot-Google?
- Allow it if: you run Google Ads that point to this site.
- Block it if: you do not advertise with Google. Remember that User-agent: * does not cover it.
- What blocking changes: Google Ads can no longer check your landing pages, which can affect ads that point to them.
To block it, add this group to your robots.txt. Google says AdsBot ignores the global User-agent: * group. A rule only applies if it names AdsBot-Google.
```
User-agent: AdsBot-Google
Disallow: /
```
robots.txt only asks. It does nothing against a client that ignores it or only pretends to be AdsBot-Google. Stopping those takes verification at your edge.
## Related bots
- [Google-CloudVertexBot](https://www.centinelanalytica.com/bots/google-cloudvertexbot) (Google): Crawls sites whose owners asked for it when building Vertex AI Agents.
- [Google-Extended](https://www.centinelanalytica.com/bots/google-extended) (Google): A robots.txt token that controls Gemini training and grounding use.
- [Google-InspectionTool](https://www.centinelanalytica.com/bots/google-inspectiontool) (Google): Fetches pages for Search Console URL inspection and the Rich Results Test.
- [Googlebot-Image](https://www.centinelanalytica.com/bots/googlebot-image) (Google): Crawls images for Google Images and image features in Search.
- [Googlebot-News](https://www.centinelanalytica.com/bots/googlebot-news) (Google): Controls whether your pages appear in Google News.
- [Googlebot-Video](https://www.centinelanalytica.com/bots/googlebot-video) (Google): Crawls video for the video features of Google Search.
---
Page: https://www.centinelanalytica.com/bots/ahrefsbot
# What is AhrefsBot?
AhrefsBot is Ahrefs' crawler. Ahrefs says it powers the database for both Ahrefs' SEO tools and Yep, a search engine.
- Operator: Ahrefs
- Purpose: Builds the link database behind Ahrefs and the Yep search engine.
- robots.txt token: `AhrefsBot`
- Example user agent: `Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)`
- How to verify: Check that the reverse DNS name ends in ahrefs.com or ahrefs.net, or check the source IP against the ranges Ahrefs publishes.
- Reverse DNS: ahrefs.com, ahrefs.net
- IP ranges: [https://api.ahrefs.com/v3/public/crawler-ip-ranges](https://api.ahrefs.com/v3/public/crawler-ip-ranges)
- Operator documentation: [https://ahrefs.com/robot](https://ahrefs.com/robot)
- Last verified: 2026-09-30
## How often is AhrefsBot impersonated?
For the median organisation, under 1% of requests claiming to be AhrefsBot failed verification.
Measured between 14 July 2026 and 10 October 2026 across organisations protected by Centinel. We only publish a share when at least five organisations each saw enough of these requests. Pooled across all of their requests, the share was under 1%: a few large organisations carry most of the traffic, so the median organisation is the better guide to what yours will see.
This directory measures requests between 14 July 2026 and 10 October 2026. The [fake crawler report](https://www.centinelanalytica.com/blog/fake-crawler-report) covers its own, earlier window (late June to 21 September 2026), so its figures differ from the ones here. Read it for the method.
## Should you block AhrefsBot?
- Allow it if: you or your partners use this tool's backlink or audit data, or the crawl load is not a concern.
- Block it if: the crawl load is noticeable and you have no use for the tool's data. Blocking it does not affect search rankings.
- What blocking changes: Ahrefs and Yep stop updating their data about your pages. Use Crawl-delay if load is the only issue.
To block it, add this group to your robots.txt. Ahrefs says AhrefsBot obeys robots.txt and Crawl-delay.
```
User-agent: AhrefsBot
Disallow: /
```
robots.txt only asks. It does nothing against a client that ignores it or only pretends to be AhrefsBot. Stopping those takes verification at your edge.
## Related bots
- [AhrefsSiteAudit](https://www.centinelanalytica.com/bots/ahrefssiteaudit) (Ahrefs): Crawls a site for an Ahrefs Site Audit project.
- [Barkrowler](https://www.centinelanalytica.com/bots/barkrowler) (Babbar): Collects links and page metadata for Babbar.
- [DataForSeoBot](https://www.centinelanalytica.com/bots/dataforseobot) (DataForSEO): Adds links to DataForSEO's backlink database.
- [DotBot](https://www.centinelanalytica.com/bots/dotbot) (Moz): Gathers data for the Moz Link Index.
- [MJ12bot](https://www.centinelanalytica.com/bots/mj12bot) (Majestic): Maps links between websites for Majestic's link index.
- [Rogerbot](https://www.centinelanalytica.com/bots/rogerbot) (Moz): Crawls a site for the site audits of a Moz Pro Campaign.
---
Page: https://www.centinelanalytica.com/bots/ahrefssiteaudit
# What is AhrefsSiteAudit?
AhrefsSiteAudit is the crawler behind the Site Audit tool of Ahrefs. Ahrefs users run Site Audit on a website to find technical SEO and on-page SEO issues.
- Operator: Ahrefs
- Purpose: Crawls a site for an Ahrefs Site Audit project.
- robots.txt token: `AhrefsSiteAudit`
- Example user agent: `Mozilla/5.0 (compatible; AhrefsSiteAudit/6.1; +http://ahrefs.com/robot/site-audit)`
- How to verify: Check that the reverse DNS name ends in ahrefs.com or ahrefs.net, or check the source IP against the ranges Ahrefs publishes.
- Reverse DNS: ahrefs.com, ahrefs.net
- IP ranges: [https://api.ahrefs.com/v3/public/crawler-ip-ranges](https://api.ahrefs.com/v3/public/crawler-ip-ranges)
- Operator documentation: [https://ahrefs.com/robot](https://ahrefs.com/robot)
- Last verified: 2026-10-10
## How often is AhrefsSiteAudit impersonated?
Too few organisations saw requests claiming to be AhrefsSiteAudit in the measurement window to publish a share.
This directory measures requests between 14 July 2026 and 10 October 2026. The [fake crawler report](https://www.centinelanalytica.com/blog/fake-crawler-report) covers its own, earlier window (late June to 21 September 2026), so its figures differ from the ones here. Read it for the method.
## Should you block AhrefsSiteAudit?
- Allow it if: you or your agency run Ahrefs Site Audit on this site.
- Block it if: nobody who works on this site uses Ahrefs Site Audit.
- What blocking changes: A Site Audit project for your site cannot crawl it.
To block it, add this group to your robots.txt. Ahrefs says AhrefsSiteAudit obeys robots.txt and Crawl-delay by default. The owner of a verified site can let it disobey robots.txt on that site.
```
User-agent: AhrefsSiteAudit
Disallow: /
```
robots.txt only asks. It does nothing against a client that ignores it or only pretends to be AhrefsSiteAudit. Stopping those takes verification at your edge.
## Related bots
- [AhrefsBot](https://www.centinelanalytica.com/bots/ahrefsbot) (Ahrefs): Builds the link database behind Ahrefs and the Yep search engine.
- [Barkrowler](https://www.centinelanalytica.com/bots/barkrowler) (Babbar): Collects links and page metadata for Babbar.
- [DataForSeoBot](https://www.centinelanalytica.com/bots/dataforseobot) (DataForSEO): Adds links to DataForSEO's backlink database.
- [DotBot](https://www.centinelanalytica.com/bots/dotbot) (Moz): Gathers data for the Moz Link Index.
- [MJ12bot](https://www.centinelanalytica.com/bots/mj12bot) (Majestic): Maps links between websites for Majestic's link index.
- [Rogerbot](https://www.centinelanalytica.com/bots/rogerbot) (Moz): Crawls a site for the site audits of a Moz Pro Campaign.
---
Page: https://www.centinelanalytica.com/bots/amazonbot
# What is Amazonbot?
Amazonbot is Amazon's web crawler. Amazon says it is used to improve its products and services and may be used to train Amazon AI models. Amzn-SearchBot and Amzn-User are separate agents for search and user actions.
- Operator: Amazon
- Purpose: Crawls to improve Amazon products; may be used to train Amazon AI models.
- robots.txt token: `Amazonbot`
- Example user agent: `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1) Chrome/W.X.Y.Z Safari/537.36`
- How to verify: Check that the source IP is in the addresses Amazon publishes for Amazonbot.
- IP ranges: [https://developer.amazon.com/amazonbot/ip-addresses/](https://developer.amazon.com/amazonbot/ip-addresses/)
- Operator documentation: [https://developer.amazon.com/amazonbot](https://developer.amazon.com/amazonbot)
- Last verified: 2026-09-30
## How often is Amazonbot impersonated?
For the median organisation, 2% of requests claiming to be Amazonbot failed verification.
Measured between 14 July 2026 and 10 October 2026 across organisations protected by Centinel. We only publish a share when at least five organisations each saw enough of these requests. Pooled across all of their requests, the share was under 1%: a few large organisations carry most of the traffic, so the median organisation is the better guide to what yours will see.
This directory measures requests between 14 July 2026 and 10 October 2026. The [fake crawler report](https://www.centinelanalytica.com/blog/fake-crawler-report) covers its own, earlier window (late June to 21 September 2026), so its figures differ from the ones here. Read it for the method.
## Should you block Amazonbot?
- Allow it if: you are content for Amazon to use your pages in its products and models.
- Block it if: you do not want your content used for Amazon's AI training.
- What blocking changes: Amazon stops crawling your pages with Amazonbot. Amzn-SearchBot and Amzn-User are controlled separately.
To block it, add this group to your robots.txt. Amazon says Amazonbot respects robots.txt and page-level noarchive and noindex tags, but not Crawl-delay.
```
User-agent: Amazonbot
Disallow: /
```
robots.txt only asks. It does nothing against a client that ignores it or only pretends to be Amazonbot. Stopping those takes verification at your edge.
## Related bots
- [CCBot](https://www.centinelanalytica.com/bots/ccbot) (Common Crawl): Builds Common Crawl's open web archive, free for anyone to reuse.
- [ClaudeBot](https://www.centinelanalytica.com/bots/claudebot) (Anthropic): Collects web content that could contribute to Claude model training.
- [GPTBot](https://www.centinelanalytica.com/bots/gptbot) (OpenAI): Crawls content that may be used to train OpenAI's foundation models.
- [Meta-ExternalAgent](https://www.centinelanalytica.com/bots/meta-externalagent) (Meta): Crawls for training AI models or improving Meta products.
---
Page: https://www.centinelanalytica.com/bots/applebot-extended
# What is Applebot-Extended?
Applebot-Extended is a robots.txt control token, not a crawler. Apple says it does not crawl web pages: disallowing it tells Apple not to use content that Applebot crawls to train its foundation models.
- Operator: Apple
- Purpose: A robots.txt token that opts content out of Apple model training.
- robots.txt token: `Applebot-Extended`
- How to verify: There is nothing to verify: Apple says Applebot-Extended does not crawl. A request that uses the name is not from Apple.
- Operator documentation: [https://support.apple.com/en-us/119829](https://support.apple.com/en-us/119829)
- Last verified: 2026-09-30
## How often is Applebot-Extended impersonated?
No real crawler sends Applebot-Extended as its user agent, so every request that claims it is fake; there is nothing to verify.
This directory measures requests between 14 July 2026 and 10 October 2026. The [fake crawler report](https://www.centinelanalytica.com/blog/fake-crawler-report) covers its own, earlier window (late June to 21 September 2026), so its figures differ from the ones here. Read it for the method.
## Should you block Applebot-Extended?
- Allow it if: you are content for Apple to use your pages in model training.
- Block it if: you want to stay in Apple search features but opt out of model training.
- What blocking changes: Apple may not use your content to train its foundation models. Your pages can still appear in Apple's search features.
To block it, add this group to your robots.txt. Apple reads this token from robots.txt. Applebot keeps crawling for search either way.
```
User-agent: Applebot-Extended
Disallow: /
```
robots.txt only asks. It does nothing against a client that ignores it or only pretends to be Applebot-Extended. Stopping those takes verification at your edge.
## Related bots
- [Applebot](https://www.centinelanalytica.com/bots/applebot) (Apple): Crawls pages for Spotlight, Siri and Safari search features.
- [Google-Extended](https://www.centinelanalytica.com/bots/google-extended) (Google): A robots.txt token that controls Gemini training and grounding use.
---
Page: https://www.centinelanalytica.com/bots/applebot
# What is Applebot?
Applebot is Apple's web crawler. Apple says it powers search features in Spotlight, Siri and Safari, and that the data it collects may also be used to help train Apple's foundation models. Applebot-Extended is the separate token for opting out of that training.
- Operator: Apple
- Purpose: Crawls pages for Spotlight, Siri and Safari search features.
- robots.txt token: `Applebot`
- Example user agent: `Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot)`
- How to verify: Run a reverse DNS lookup and check that the name ends in applebot.apple.com, then confirm with a forward lookup. Apple also publishes Applebot's IP ranges.
- Reverse DNS: applebot.apple.com
- IP ranges: [https://search.developer.apple.com/applebot.json](https://search.developer.apple.com/applebot.json)
- Operator documentation: [https://support.apple.com/en-us/119829](https://support.apple.com/en-us/119829)
- Last verified: 2026-09-30
## How often is Applebot impersonated?
For the median organisation, under 1% of requests claiming to be Applebot failed verification.
Measured between 14 July 2026 and 10 October 2026 across organisations protected by Centinel. We only publish a share when at least five organisations each saw enough of these requests. Pooled across all of their requests, the share was under 1%: a few large organisations carry most of the traffic, so the median organisation is the better guide to what yours will see.
This directory measures requests between 14 July 2026 and 10 October 2026. The [fake crawler report](https://www.centinelanalytica.com/blog/fake-crawler-report) covers its own, earlier window (late June to 21 September 2026), so its figures differ from the ones here. Read it for the method.
## Should you block Applebot?
- Allow it if: you want to appear in Spotlight, Siri and Safari suggestions.
- Block it if: you want no Apple crawling at all. Most sites that only object to training should use Applebot-Extended.
- What blocking changes: Your pages stop appearing in Apple's search features. To opt out of training only, disallow Applebot-Extended instead.
To block it, add this group to your robots.txt. Apple says Applebot follows your Googlebot rules when robots.txt does not name Applebot, and does not support Crawl-delay.
```
User-agent: Applebot
Disallow: /
```
robots.txt only asks. It does nothing against a client that ignores it or only pretends to be Applebot. Stopping those takes verification at your edge.
## Related bots
- [Applebot-Extended](https://www.centinelanalytica.com/bots/applebot-extended) (Apple): A robots.txt token that opts content out of Apple model training.
- [360Spider](https://www.centinelanalytica.com/bots/360spider) (360 Search): Crawls pages for the web search of 360 Search (so.com).
- [Baiduspider](https://www.centinelanalytica.com/bots/baiduspider) (Baidu): Crawls pages for Baidu search.
- [Bingbot](https://www.centinelanalytica.com/bots/bingbot) (Microsoft): Crawls pages for Bing search.
- [Diffbot](https://www.centinelanalytica.com/bots/diffbot) (Diffbot): Crawls the web for the Diffbot Knowledge Graph and Diffbot's web search services.
- [DuckDuckBot](https://www.centinelanalytica.com/bots/duckduckbot) (DuckDuckGo): Crawls pages for DuckDuckGo search.
---
Page: https://www.centinelanalytica.com/bots/baiduspider
# What is Baiduspider?
Baiduspider is Baidu's search crawler. Baidu uses separate user agents for some products, such as Baiduspider-image for image search, so you can set rules per product.
- Operator: Baidu
- Purpose: Crawls pages for Baidu search.
- robots.txt token: `Baiduspider`
- Example user agent: `Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)`
- How to verify: Run a reverse DNS lookup and check that the name ends in baidu.com or baidu.jp, then confirm with a forward lookup.
- Reverse DNS: baidu.com, baidu.jp
- Operator documentation: [https://help.baidu.com/question?prod_id=99&class=476&id=2996](https://help.baidu.com/question?prod_id=99&class=476&id=2996)
- Last verified: 2026-09-30
## How often is Baiduspider impersonated?
For the median organisation, 2% of requests claiming to be Baiduspider failed verification.
Measured between 14 July 2026 and 10 October 2026 across organisations protected by Centinel. We only publish a share when at least five organisations each saw enough of these requests. Pooled across all of their requests, the share was 1%: a few large organisations carry most of the traffic, so the median organisation is the better guide to what yours will see.
This directory measures requests between 14 July 2026 and 10 October 2026. The [fake crawler report](https://www.centinelanalytica.com/blog/fake-crawler-report) covers its own, earlier window (late June to 21 September 2026), so its figures differ from the ones here. Read it for the method.
## Should you block Baiduspider?
- Allow it if: you have visitors who search with Baidu.
- Block it if: you have no audience on Baidu and the crawl adds load.
- What blocking changes: Baidu says your pages can then no longer be found in Baidu search or in the search engines Baidu powers.
To block it, add this group to your robots.txt. Baidu says Baiduspider obeys robots.txt.
```
User-agent: Baiduspider
Disallow: /
```
robots.txt only asks. It does nothing against a client that ignores it or only pretends to be Baiduspider. Stopping those takes verification at your edge.
## Related bots
- [360Spider](https://www.centinelanalytica.com/bots/360spider) (360 Search): Crawls pages for the web search of 360 Search (so.com).
- [Applebot](https://www.centinelanalytica.com/bots/applebot) (Apple): Crawls pages for Spotlight, Siri and Safari search features.
- [Bingbot](https://www.centinelanalytica.com/bots/bingbot) (Microsoft): Crawls pages for Bing search.
- [Diffbot](https://www.centinelanalytica.com/bots/diffbot) (Diffbot): Crawls the web for the Diffbot Knowledge Graph and Diffbot's web search services.
- [DuckDuckBot](https://www.centinelanalytica.com/bots/duckduckbot) (DuckDuckGo): Crawls pages for DuckDuckGo search.
- [Googlebot-Image](https://www.centinelanalytica.com/bots/googlebot-image) (Google): Crawls images for Google Images and image features in Search.
---
Page: https://www.centinelanalytica.com/bots/barkrowler
# What is Barkrowler?
Barkrowler is Babbar's crawler. Babbar says it keeps the links and page metadata it finds, not the crawled content itself.
- Operator: Babbar
- Purpose: Collects links and page metadata for Babbar.
- robots.txt token: `barkrowler`
- How to verify: Check that the reverse DNS name ends in babbar.eu, or check the source IP against Babbar's published ranges.
- Reverse DNS: babbar.eu
- IP ranges: [https://www.babbar.tech/barkrowler-ip-ranges.json](https://www.babbar.tech/barkrowler-ip-ranges.json)
- Operator documentation: [https://www.babbar.tech/crawler](https://www.babbar.tech/crawler)
- Last verified: 2026-09-30
## How often is Barkrowler impersonated?
For the median organisation, under 1% of requests claiming to be Barkrowler failed verification.
Measured between 14 July 2026 and 10 October 2026 across organisations protected by Centinel. We only publish a share when at least five organisations each saw enough of these requests. Pooled across all of their requests, the share was under 1%: a few large organisations carry most of the traffic, so the median organisation is the better guide to what yours will see.
This directory measures requests between 14 July 2026 and 10 October 2026. The [fake crawler report](https://www.centinelanalytica.com/blog/fake-crawler-report) covers its own, earlier window (late June to 21 September 2026), so its figures differ from the ones here. Read it for the method.
## Should you block Barkrowler?
- Allow it if: you or your partners use this tool's backlink or audit data, or the crawl load is not a concern.
- Block it if: the crawl load is noticeable and you have no use for the tool's data. Blocking it does not affect search rankings.
- What blocking changes: Babbar stops crawling your site.
To block it, add this group to your robots.txt. Babbar says Barkrowler respects robots.txt and Crawl-delay, and by default waits 5 seconds between requests to the same host.
```
User-agent: barkrowler
Disallow: /
```
robots.txt only asks. It does nothing against a client that ignores it or only pretends to be Barkrowler. Stopping those takes verification at your edge.
## Related bots
- [AhrefsBot](https://www.centinelanalytica.com/bots/ahrefsbot) (Ahrefs): Builds the link database behind Ahrefs and the Yep search engine.
- [AhrefsSiteAudit](https://www.centinelanalytica.com/bots/ahrefssiteaudit) (Ahrefs): Crawls a site for an Ahrefs Site Audit project.
- [DataForSeoBot](https://www.centinelanalytica.com/bots/dataforseobot) (DataForSEO): Adds links to DataForSEO's backlink database.
- [DotBot](https://www.centinelanalytica.com/bots/dotbot) (Moz): Gathers data for the Moz Link Index.
- [MJ12bot](https://www.centinelanalytica.com/bots/mj12bot) (Majestic): Maps links between websites for Majestic's link index.
- [Rogerbot](https://www.centinelanalytica.com/bots/rogerbot) (Moz): Crawls a site for the site audits of a Moz Pro Campaign.
---
Page: https://www.centinelanalytica.com/bots/bingbot
# What is Bingbot?
Bingbot is Microsoft's standard crawler for Bing and handles most of Bing's crawling. Microsoft notes that the user agent is easy to spoof and offers a verification tool.
- Operator: Microsoft
- Purpose: Crawls pages for Bing search.
- robots.txt token: `bingbot`
- Example user agent: `Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)`
- How to verify: Run a reverse DNS lookup and check that the name ends in search.msn.com, then confirm with a forward lookup. Microsoft also publishes Bingbot's IP ranges.
- Reverse DNS: search.msn.com
- IP ranges: [https://www.bing.com/toolbox/bingbot.json](https://www.bing.com/toolbox/bingbot.json)
- Operator documentation: [https://www.bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0](https://www.bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0)
- Last verified: 2026-09-30
## How often is Bingbot impersonated?
For the median organisation, under 1% of requests claiming to be Bingbot failed verification.
Measured between 14 July 2026 and 10 October 2026 across organisations protected by Centinel. We only publish a share when at least five organisations each saw enough of these requests. Pooled across all of their requests, the share was under 1%: a few large organisations carry most of the traffic, so the median organisation is the better guide to what yours will see.
This directory measures requests between 14 July 2026 and 10 October 2026. The [fake crawler report](https://www.centinelanalytica.com/blog/fake-crawler-report) covers its own, earlier window (late June to 21 September 2026), so its figures differ from the ones here. Read it for the method.
## Should you block Bingbot?
- Allow it if: you want to appear in Bing search. This is nearly every public site.
- Block it if: a section should never appear in Bing. Block the path, not the crawler.
- What blocking changes: Your pages drop out of Bing and the services built on Bing's index.
To block it, add this group to your robots.txt. Microsoft documents robots.txt as the way to control Bing's crawlers.
```
User-agent: bingbot
Disallow: /
```
robots.txt only asks. It does nothing against a client that ignores it or only pretends to be Bingbot. Stopping those takes verification at your edge.
## Related bots
- [360Spider](https://www.centinelanalytica.com/bots/360spider) (360 Search): Crawls pages for the web search of 360 Search (so.com).
- [Applebot](https://www.centinelanalytica.com/bots/applebot) (Apple): Crawls pages for Spotlight, Siri and Safari search features.
- [Baiduspider](https://www.centinelanalytica.com/bots/baiduspider) (Baidu): Crawls pages for Baidu search.
- [Diffbot](https://www.centinelanalytica.com/bots/diffbot) (Diffbot): Crawls the web for the Diffbot Knowledge Graph and Diffbot's web search services.
- [DuckDuckBot](https://www.centinelanalytica.com/bots/duckduckbot) (DuckDuckGo): Crawls pages for DuckDuckGo search.
- [Googlebot-Image](https://www.centinelanalytica.com/bots/googlebot-image) (Google): Crawls images for Google Images and image features in Search.
---
Page: https://www.centinelanalytica.com/bots/bytespider
# What is Bytespider?
Bytespider is a crawler name that third-party catalogs attribute to ByteDance. We found no ByteDance page that documents it, its purpose, its IP ranges or its robots.txt behavior, so a Bytespider claim cannot be checked against the operator.
- Operator: Unconfirmed, attributed to ByteDance
- Purpose: A crawler with no operator documentation.
- robots.txt token: `Bytespider`
- How to verify: No published method. Observe the networks it comes from and its request rate before deciding.
- Operator documentation: None published
- Last verified: 2026-09-30
## How often is Bytespider impersonated?
No published method checks a request that claims to be Bytespider, so we cannot say which claims are false and show no share.
This directory measures requests between 14 July 2026 and 10 October 2026. The [fake crawler report](https://www.centinelanalytica.com/blog/fake-crawler-report) covers its own, earlier window (late June to 21 September 2026), so its figures differ from the ones here. Read it for the method.
## Should you block Bytespider?
- Allow it if: you see no load or abuse from it and prefer to observe first.
- Block it if: its load is noticeable. Since it cannot be verified, enforce the decision at the edge, not only in robots.txt.
- What blocking changes: Without operator documentation there is no stated effect. A robots.txt rule may or may not be honored.
To block it, add this group to your robots.txt. No operator documentation says whether it obeys robots.txt.
```
User-agent: Bytespider
Disallow: /
```
robots.txt only asks. It does nothing against a client that ignores it or only pretends to be Bytespider. Stopping those takes verification at your edge.
## Related bots
---
Page: https://www.centinelanalytica.com/bots/ccbot
# What is CCBot?
CCBot is the crawler of Common Crawl, which publishes its web crawl data as open data for anyone to use. Common Crawl itself warns that other crawlers falsely identify themselves as CCBot.
- Operator: Common Crawl
- Purpose: Builds Common Crawl's open web archive, free for anyone to reuse.
- robots.txt token: `CCBot`
- Example user agent: `CCBot/2.0 (https://commoncrawl.org/faq/)`
- How to verify: Common Crawl says CCBot runs on dedicated IP ranges with reverse DNS (except IPv6). Check the source IP against its published list.
- IP ranges: [https://index.commoncrawl.org/ccbot.json](https://index.commoncrawl.org/ccbot.json)
- Operator documentation: [https://commoncrawl.org/ccbot](https://commoncrawl.org/ccbot)
- Last verified: 2026-09-30
## How often is CCBot impersonated?
For the median organisation, 89% of requests claiming to be CCBot failed verification.
Measured between 14 July 2026 and 10 October 2026 across organisations protected by Centinel. We only publish a share when at least five organisations each saw enough of these requests. Pooled across all of their requests, the share was 43%: a few large organisations carry most of the traffic, so the median organisation is the better guide to what yours will see.
This directory measures requests between 14 July 2026 and 10 October 2026. The [fake crawler report](https://www.centinelanalytica.com/blog/fake-crawler-report) covers its own, earlier window (late June to 21 September 2026), so its figures differ from the ones here. Read it for the method.
## Should you block CCBot?
- Allow it if: you are content for your pages to be in a public dataset others reuse.
- Block it if: you do not want your content in an open dataset that third parties can use for any purpose, including model training.
- What blocking changes: Your pages stop being added to future Common Crawl archives. Copies already in past archives stay there.
To block it, add this group to your robots.txt. Common Crawl says CCBot obeys robots.txt and Crawl-delay.
```
User-agent: CCBot
Disallow: /
```
robots.txt only asks. It does nothing against a client that ignores it or only pretends to be CCBot. Stopping those takes verification at your edge.
## Related bots
- [Amazonbot](https://www.centinelanalytica.com/bots/amazonbot) (Amazon): Crawls to improve Amazon products; may be used to train Amazon AI models.
- [ClaudeBot](https://www.centinelanalytica.com/bots/claudebot) (Anthropic): Collects web content that could contribute to Claude model training.
- [GPTBot](https://www.centinelanalytica.com/bots/gptbot) (OpenAI): Crawls content that may be used to train OpenAI's foundation models.
- [Meta-ExternalAgent](https://www.centinelanalytica.com/bots/meta-externalagent) (Meta): Crawls for training AI models or improving Meta products.
---
Page: https://www.centinelanalytica.com/bots/chatgpt-user
# What is ChatGPT-User?
ChatGPT-User is the user agent OpenAI uses when a person's request in ChatGPT or a Custom GPT needs a web page. OpenAI says it is not used to crawl the web automatically and is not used to decide whether content appears in search.
- Operator: [OpenAI](https://www.centinelanalytica.com/bots/operator/openai)
- Purpose: Visits a page when a ChatGPT or Custom GPT user's request needs it.
- robots.txt token: `ChatGPT-User`
- Example user agent: `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot`
- How to verify: Check that the source IP is in the ranges OpenAI publishes for ChatGPT-User.
- IP ranges: [https://openai.com/chatgpt-user.json](https://openai.com/chatgpt-user.json)
- Operator documentation: [https://platform.openai.com/docs/bots](https://platform.openai.com/docs/bots)
- Last verified: 2026-09-30
## How often is ChatGPT-User impersonated?
For the median organisation, 10% of requests claiming to be ChatGPT-User failed verification.
Measured between 14 July 2026 and 10 October 2026 across organisations protected by Centinel. We only publish a share when at least five organisations each saw enough of these requests. Pooled across all of their requests, the share was 1%: a few large organisations carry most of the traffic, so the median organisation is the better guide to what yours will see.
This directory measures requests between 14 July 2026 and 10 October 2026. The [fake crawler report](https://www.centinelanalytica.com/blog/fake-crawler-report) covers its own, earlier window (late June to 21 September 2026), so its figures differ from the ones here. Read it for the method.
## Should you block ChatGPT-User?
- Allow it if: you want the assistant to be able to read and cite your pages when someone asks about them.
- Block it if: your content is not meant to be fetched on demand by an assistant, and you enforce that at the edge rather than in robots.txt.
- What blocking changes: OpenAI does not promise that a robots.txt rule stops these user-initiated visits. To stop them reliably, enforce the rule at your edge by verified IP.
To block it, add this group to your robots.txt. OpenAI says robots.txt rules may not apply, because a user initiates these requests.
```
User-agent: ChatGPT-User
Disallow: /
```
robots.txt only asks. It does nothing against a client that ignores it or only pretends to be ChatGPT-User. Stopping those takes verification at your edge.
## Related bots
- [GPTBot](https://www.centinelanalytica.com/bots/gptbot) (OpenAI): Crawls content that may be used to train OpenAI's foundation models.
- [OAI-SearchBot](https://www.centinelanalytica.com/bots/oai-searchbot) (OpenAI): Surfaces websites in ChatGPT's search features.
- [Claude-User](https://www.centinelanalytica.com/bots/claude-user) (Anthropic): Fetches a page when a Claude user's question needs it.
- [Meta-ExternalFetcher](https://www.centinelanalytica.com/bots/meta-externalfetcher) (Meta): Fetches individual links on behalf of Meta product users.
- [MistralAI-User](https://www.centinelanalytica.com/bots/mistralai-user) (Mistral AI): Fetches a page when a Mistral user's action needs it.
- [Perplexity-User](https://www.centinelanalytica.com/bots/perplexity-user) (Perplexity): Fetches a page when a Perplexity user's request needs it.
---
Page: https://www.centinelanalytica.com/bots/claude-searchbot
# What is Claude-SearchBot?
Claude-SearchBot is Anthropic's search crawler. Anthropic says it navigates the web to improve the relevance and accuracy of the search responses Claude gives users.
- Operator: [Anthropic](https://www.centinelanalytica.com/bots/operator/anthropic)
- Purpose: Indexes pages to improve Claude's search results.
- robots.txt token: `Claude-SearchBot`
- How to verify: Check that the source IP is in the ranges Anthropic publishes for its crawlers.
- IP ranges: [https://claude.com/crawling/bots.json](https://claude.com/crawling/bots.json)
- Operator documentation: [https://support.claude.com/en/articles/8896518](https://support.claude.com/en/articles/8896518)
- Last verified: 2026-09-30
## How often is Claude-SearchBot impersonated?
For the median organisation, over 99% of requests claiming to be Claude-SearchBot failed verification.
Measured between 14 July 2026 and 10 October 2026 across organisations protected by Centinel. We only publish a share when at least five organisations each saw enough of these requests. Pooled across all of their requests, the share was 2%: a few large organisations carry most of the traffic, so the median organisation is the better guide to what yours will see.
This directory measures requests between 14 July 2026 and 10 October 2026. The [fake crawler report](https://www.centinelanalytica.com/blog/fake-crawler-report) covers its own, earlier window (late June to 21 September 2026), so its figures differ from the ones here. Read it for the method.
## Should you block Claude-SearchBot?
- Allow it if: you want Claude's search to find and cite your pages.
- Block it if: you do not want your pages indexed for Claude's search results.
- What blocking changes: Anthropic says it stops your content being indexed for search, which may reduce your visibility in Claude's search results.
To block it, add this group to your robots.txt. Anthropic says its bots honor robots.txt and support Crawl-delay.
```
User-agent: Claude-SearchBot
Disallow: /
```
robots.txt only asks. It does nothing against a client that ignores it or only pretends to be Claude-SearchBot. Stopping those takes verification at your edge.
## Related bots
- [Claude-User](https://www.centinelanalytica.com/bots/claude-user) (Anthropic): Fetches a page when a Claude user's question needs it.
- [ClaudeBot](https://www.centinelanalytica.com/bots/claudebot) (Anthropic): Collects web content that could contribute to Claude model training.
- [DuckAssistBot](https://www.centinelanalytica.com/bots/duckassistbot) (DuckDuckGo): Crawls in real time to support DuckDuckGo's AI-assisted answers.
- [Meta-WebIndexer](https://www.centinelanalytica.com/bots/meta-webindexer) (Meta): Crawls pages to improve the search results of Meta AI.
- [MistralAI-Index](https://www.centinelanalytica.com/bots/mistralai-index) (Mistral AI): Indexes pages for Mistral search.
- [OAI-SearchBot](https://www.centinelanalytica.com/bots/oai-searchbot) (OpenAI): Surfaces websites in ChatGPT's search features.
---
Page: https://www.centinelanalytica.com/bots/claude-user
# What is Claude-User?
Claude-User is the agent Anthropic uses when a person asks Claude a question that needs a web page. It retrieves content on demand rather than crawling.
- Operator: [Anthropic](https://www.centinelanalytica.com/bots/operator/anthropic)
- Purpose: Fetches a page when a Claude user's question needs it.
- robots.txt token: `Claude-User`
- How to verify: Check that the source IP is in the ranges Anthropic publishes for its crawlers.
- IP ranges: [https://claude.com/crawling/bots.json](https://claude.com/crawling/bots.json)
- Operator documentation: [https://support.claude.com/en/articles/8896518](https://support.claude.com/en/articles/8896518)
- Last verified: 2026-09-30
## How often is Claude-User impersonated?
For the median organisation, 33% of requests claiming to be Claude-User failed verification.
Measured between 14 July 2026 and 10 October 2026 across organisations protected by Centinel. We only publish a share when at least five organisations each saw enough of these requests. Pooled across all of their requests, the share was 29%: a few large organisations carry most of the traffic, so the median organisation is the better guide to what yours will see.
This directory measures requests between 14 July 2026 and 10 October 2026. The [fake crawler report](https://www.centinelanalytica.com/blog/fake-crawler-report) covers its own, earlier window (late June to 21 September 2026), so its figures differ from the ones here. Read it for the method.
## Should you block Claude-User?
- Allow it if: you want the assistant to be able to read and cite your pages when someone asks about them.
- Block it if: you do not want Claude fetching your pages for its users.
- What blocking changes: Anthropic says Claude will not retrieve your content in response to user queries, which may reduce your visibility in user-directed web search.
To block it, add this group to your robots.txt. Anthropic says Claude-User lets site owners control these user-initiated requests through robots.txt.
```
User-agent: Claude-User
Disallow: /
```
robots.txt only asks. It does nothing against a client that ignores it or only pretends to be Claude-User. Stopping those takes verification at your edge.
## Related bots
- [Claude-SearchBot](https://www.centinelanalytica.com/bots/claude-searchbot) (Anthropic): Indexes pages to improve Claude's search results.
- [ClaudeBot](https://www.centinelanalytica.com/bots/claudebot) (Anthropic): Collects web content that could contribute to Claude model training.
- [ChatGPT-User](https://www.centinelanalytica.com/bots/chatgpt-user) (OpenAI): Visits a page when a ChatGPT or Custom GPT user's request needs it.
- [Meta-ExternalFetcher](https://www.centinelanalytica.com/bots/meta-externalfetcher) (Meta): Fetches individual links on behalf of Meta product users.
- [MistralAI-User](https://www.centinelanalytica.com/bots/mistralai-user) (Mistral AI): Fetches a page when a Mistral user's action needs it.
- [Perplexity-User](https://www.centinelanalytica.com/bots/perplexity-user) (Perplexity): Fetches a page when a Perplexity user's request needs it.
---
Page: https://www.centinelanalytica.com/bots/claudebot
# What is ClaudeBot?
ClaudeBot is Anthropic's training crawler. Anthropic says it collects web content that could contribute to training its generative AI models, and that restricting ClaudeBot signals that a site's future material should be excluded from training datasets.
- Operator: [Anthropic](https://www.centinelanalytica.com/bots/operator/anthropic)
- Purpose: Collects web content that could contribute to Claude model training.
- robots.txt token: `ClaudeBot`
- How to verify: Check that the source IP is in the ranges Anthropic publishes for its crawlers.
- IP ranges: [https://claude.com/crawling/bots.json](https://claude.com/crawling/bots.json)
- Operator documentation: [https://support.claude.com/en/articles/8896518](https://support.claude.com/en/articles/8896518)
- Last verified: 2026-09-30
## How often is ClaudeBot impersonated?
For the median organisation, 22% of requests claiming to be ClaudeBot failed verification.
Measured between 14 July 2026 and 10 October 2026 across organisations protected by Centinel. We only publish a share when at least five organisations each saw enough of these requests. Pooled across all of their requests, the share was 4%: a few large organisations carry most of the traffic, so the median organisation is the better guide to what yours will see.
This directory measures requests between 14 July 2026 and 10 October 2026. The [fake crawler report](https://www.centinelanalytica.com/blog/fake-crawler-report) covers its own, earlier window (late June to 21 September 2026), so its figures differ from the ones here. Read it for the method.
## Should you block ClaudeBot?
- Allow it if: you are content for your pages to be used in Claude model training.
- Block it if: you do not want your content used for training. Use Crawl-delay instead if the only issue is load.
- What blocking changes: It signals that your future content should be excluded from Anthropic's training datasets. Claude-SearchBot and Claude-User are controlled separately.
To block it, add this group to your robots.txt. Anthropic says its bots honor robots.txt and support the non-standard Crawl-delay extension.
```
User-agent: ClaudeBot
Disallow: /
```
robots.txt only asks. It does nothing against a client that ignores it or only pretends to be ClaudeBot. Stopping those takes verification at your edge.
## Related bots
- [Claude-SearchBot](https://www.centinelanalytica.com/bots/claude-searchbot) (Anthropic): Indexes pages to improve Claude's search results.
- [Claude-User](https://www.centinelanalytica.com/bots/claude-user) (Anthropic): Fetches a page when a Claude user's question needs it.
- [Amazonbot](https://www.centinelanalytica.com/bots/amazonbot) (Amazon): Crawls to improve Amazon products; may be used to train Amazon AI models.
- [CCBot](https://www.centinelanalytica.com/bots/ccbot) (Common Crawl): Builds Common Crawl's open web archive, free for anyone to reuse.
- [GPTBot](https://www.centinelanalytica.com/bots/gptbot) (OpenAI): Crawls content that may be used to train OpenAI's foundation models.
- [Meta-ExternalAgent](https://www.centinelanalytica.com/bots/meta-externalagent) (Meta): Crawls for training AI models or improving Meta products.
---
Page: https://www.centinelanalytica.com/bots/dataforseobot
# What is DataForSeoBot?
DataForSeoBot is DataForSEO's backlink crawler. It crawls pages and adds the links it finds to the database behind its Backlinks API.
- Operator: DataForSEO
- Purpose: Adds links to DataForSEO's backlink database.
- robots.txt token: `DataForSeoBot`
- Example user agent: `Mozilla/5.0 (compatible; DataForSeoBot; +https://dataforseo.com/dataforseo-bot)`
- How to verify: DataForSEO lists its subnets and a reverse DNS check on its bot page.
- Operator documentation: [https://dataforseo.com/dataforseo-bot](https://dataforseo.com/dataforseo-bot)
- Last verified: 2026-09-30
## How often is DataForSeoBot impersonated?
For the median organisation, under 1% of requests claiming to be DataForSeoBot failed verification.
Measured between 14 July 2026 and 10 October 2026 across organisations protected by Centinel. We only publish a share when at least five organisations each saw enough of these requests. Pooled across all of their requests, the share was under 1%: a few large organisations carry most of the traffic, so the median organisation is the better guide to what yours will see.
This directory measures requests between 14 July 2026 and 10 October 2026. The [fake crawler report](https://www.centinelanalytica.com/blog/fake-crawler-report) covers its own, earlier window (late June to 21 September 2026), so its figures differ from the ones here. Read it for the method.
## Should you block DataForSeoBot?
- Allow it if: you or your partners use this tool's backlink or audit data, or the crawl load is not a concern.
- Block it if: the crawl load is noticeable and you have no use for the tool's data. Blocking it does not affect search rankings.
- What blocking changes: DataForSEO stops adding your links to its backlink database.
To block it, add this group to your robots.txt. DataForSEO says the bot obeys robots.txt and Crawl-delay.
```
User-agent: DataForSeoBot
Disallow: /
```
robots.txt only asks. It does nothing against a client that ignores it or only pretends to be DataForSeoBot. Stopping those takes verification at your edge.
## Related bots
- [AhrefsBot](https://www.centinelanalytica.com/bots/ahrefsbot) (Ahrefs): Builds the link database behind Ahrefs and the Yep search engine.
- [AhrefsSiteAudit](https://www.centinelanalytica.com/bots/ahrefssiteaudit) (Ahrefs): Crawls a site for an Ahrefs Site Audit project.
- [Barkrowler](https://www.centinelanalytica.com/bots/barkrowler) (Babbar): Collects links and page metadata for Babbar.
- [DotBot](https://www.centinelanalytica.com/bots/dotbot) (Moz): Gathers data for the Moz Link Index.
- [MJ12bot](https://www.centinelanalytica.com/bots/mj12bot) (Majestic): Maps links between websites for Majestic's link index.
- [Rogerbot](https://www.centinelanalytica.com/bots/rogerbot) (Moz): Crawls a site for the site audits of a Moz Pro Campaign.
---
Page: https://www.centinelanalytica.com/bots/diffbot
# What is Diffbot?
Diffbot is the crawler of the company with the same name. The company says it crawls the web to build a general search engine, so that websites can be discovered and cited from its Knowledge Graph and web search services, and that the crawl is not used for AI training.
- Operator: Diffbot
- Purpose: Crawls the web for the Diffbot Knowledge Graph and Diffbot's web search services.
- robots.txt token: `Diffbot`
- How to verify: Diffbot publishes no IP list or reverse DNS domain.
- Operator documentation: [https://www.diffbot.com/docs/crawl/faq/robots-txt](https://www.diffbot.com/docs/crawl/faq/robots-txt)
- Last verified: 2026-10-10
## How often is Diffbot impersonated?
No published method checks a request that claims to be Diffbot, so we cannot say which claims are false and show no share.
This directory measures requests between 14 July 2026 and 10 October 2026. The [fake crawler report](https://www.centinelanalytica.com/blog/fake-crawler-report) covers its own, earlier window (late June to 21 September 2026), so its figures differ from the ones here. Read it for the method.
## Should you block Diffbot?
- Allow it if: you want your pages to be found and cited through the Diffbot Knowledge Graph.
- Block it if: you do not want your pages in Diffbot's data products. An IP block alone may not hold, because Diffbot uses proxies for its Extract API.
- What blocking changes: Diffbot does not state the effect of a disallow rule beyond adhering to it. For its Extract API, Diffbot says it applies a proxy pool when a site blocks its default IP addresses.
To block it, add this group to your robots.txt. Diffbot says its web crawls adhere to robots.txt by default, including the disallow and crawl-delay directives. It says the instruction can be overridden in specific cases, typically a partnership or agreement with the site.
```
User-agent: Diffbot
Disallow: /
```
robots.txt only asks. It does nothing against a client that ignores it or only pretends to be Diffbot. Stopping those takes verification at your edge.
## Related bots
- [360Spider](https://www.centinelanalytica.com/bots/360spider) (360 Search): Crawls pages for the web search of 360 Search (so.com).
- [Applebot](https://www.centinelanalytica.com/bots/applebot) (Apple): Crawls pages for Spotlight, Siri and Safari search features.
- [Baiduspider](https://www.centinelanalytica.com/bots/baiduspider) (Baidu): Crawls pages for Baidu search.
- [Bingbot](https://www.centinelanalytica.com/bots/bingbot) (Microsoft): Crawls pages for Bing search.
- [DuckDuckBot](https://www.centinelanalytica.com/bots/duckduckbot) (DuckDuckGo): Crawls pages for DuckDuckGo search.
- [Googlebot-Image](https://www.centinelanalytica.com/bots/googlebot-image) (Google): Crawls images for Google Images and image features in Search.
---
Page: https://www.centinelanalytica.com/bots/discordbot
# What is Discordbot?
Discordbot is the tool Discord uses to visit links and build link previews. Discord says it visits a link only when someone shares it in a message, makes a single GET request and executes no code on the page. Discord caches a preview for about 30 minutes.
- Operator: Discord
- Purpose: Visits a link shared in a Discord message to build its preview.
- robots.txt token: `Discordbot`
- Example user agent: `Mozilla/5.0 (compatible; Discordbot/2.0; +https://discordapp.com)`
- How to verify: Check that the source IP is in the public IP ranges Discord publishes. For the user agent, Discord says to look for the word Discordbot and not to match the whole string, because other parts can change.
- IP ranges: [https://cdn.discordapp.com/ipranges/discord.json](https://cdn.discordapp.com/ipranges/discord.json)
- Operator documentation: [https://support.discord.com/hc/en-us/articles/42500550752919-About-Discord-Link-Previews-and-the-Discordbot](https://support.discord.com/hc/en-us/articles/42500550752919-About-Discord-Link-Previews-and-the-Discordbot)
- Last verified: 2026-10-10
## How often is Discordbot impersonated?
Our verification data for Discordbot was incomplete during the measurement window, so a share would overstate impersonation. We show none until it is complete.
This directory measures requests between 14 July 2026 and 10 October 2026. The [fake crawler report](https://www.centinelanalytica.com/blog/fake-crawler-report) covers its own, earlier window (late June to 21 September 2026), so its figures differ from the ones here. Read it for the method.
## Should you block Discordbot?
- Allow it if: people share your pages on Discord.
- Block it if: you do not want preview cards on Discord.
- What blocking changes: Discord builds a preview only from a 2xx response, so a link to a page that refuses Discordbot has no preview.
To block it, add this group to your robots.txt. Discord's pages about Discordbot do not mention robots.txt.
```
User-agent: Discordbot
Disallow: /
```
robots.txt only asks. It does nothing against a client that ignores it or only pretends to be Discordbot. Stopping those takes verification at your edge.
## Related bots
- [facebookexternalhit](https://www.centinelanalytica.com/bots/facebookexternalhit) (Meta): Builds link previews when content is shared on Meta's apps.
- [Pinterestbot](https://www.centinelanalytica.com/bots/pinterestbot) (Pinterest): Indexes pages behind Pins and keeps Pin details up to date.
- [Slackbot-LinkExpanding](https://www.centinelanalytica.com/bots/slackbot-linkexpanding) (Slack): Fetches a page's meta tags when someone posts its link in Slack.
- [Twitterbot](https://www.centinelanalytica.com/bots/twitterbot) (X): Fetches card markup when a link is shared on X.
---
Page: https://www.centinelanalytica.com/bots/dotbot
# What is DotBot?
DotBot is Moz's crawler. It crawls for the Moz Link Index, which powers the Moz API, Link Explorer and the links data in Moz campaigns.
- Operator: Moz
- Purpose: Gathers data for the Moz Link Index.
- robots.txt token: `dotbot`
- How to verify: Moz publishes no IP list or reverse DNS domain for DotBot.
- Operator documentation: [https://moz.com/help/moz-procedures/crawlers/dotbot](https://moz.com/help/moz-procedures/crawlers/dotbot)
- Last verified: 2026-09-30
## How often is DotBot impersonated?
No published method checks a request that claims to be DotBot, so we cannot say which claims are false and show no share.
This directory measures requests between 14 July 2026 and 10 October 2026. The [fake crawler report](https://www.centinelanalytica.com/blog/fake-crawler-report) covers its own, earlier window (late June to 21 September 2026), so its figures differ from the ones here. Read it for the method.
## Should you block DotBot?
- Allow it if: you or your partners use this tool's backlink or audit data, or the crawl load is not a concern.
- Block it if: the crawl load is noticeable and you have no use for the tool's data. Blocking it does not affect search rankings.
- What blocking changes: Moz stops crawling your site for its link index once it rereads robots.txt.
To block it, add this group to your robots.txt. Moz says it respects robots.txt, but DotBot only rereads the file when it starts a new index crawl of a site.
```
User-agent: dotbot
Disallow: /
```
robots.txt only asks. It does nothing against a client that ignores it or only pretends to be DotBot. Stopping those takes verification at your edge.
## Related bots
- [Rogerbot](https://www.centinelanalytica.com/bots/rogerbot) (Moz): Crawls a site for the site audits of a Moz Pro Campaign.
- [AhrefsBot](https://www.centinelanalytica.com/bots/ahrefsbot) (Ahrefs): Builds the link database behind Ahrefs and the Yep search engine.
- [AhrefsSiteAudit](https://www.centinelanalytica.com/bots/ahrefssiteaudit) (Ahrefs): Crawls a site for an Ahrefs Site Audit project.
- [Barkrowler](https://www.centinelanalytica.com/bots/barkrowler) (Babbar): Collects links and page metadata for Babbar.
- [DataForSeoBot](https://www.centinelanalytica.com/bots/dataforseobot) (DataForSEO): Adds links to DataForSEO's backlink database.
- [MJ12bot](https://www.centinelanalytica.com/bots/mj12bot) (Majestic): Maps links between websites for Majestic's link index.
---
Page: https://www.centinelanalytica.com/bots/duckassistbot
# What is DuckAssistBot?
DuckAssistBot is DuckDuckGo's crawler for AI-assisted answers. DuckDuckGo says it crawls pages in real time to support those answers.
- Operator: DuckDuckGo
- Purpose: Crawls in real time to support DuckDuckGo's AI-assisted answers.
- robots.txt token: `DuckAssistBot`
- Example user agent: `DuckAssistBot/1.2; (+http://duckduckgo.com/duckassistbot.html)`
- How to verify: Check that the source IP is in the ranges DuckDuckGo publishes for DuckAssistBot.
- IP ranges: [https://duckduckgo.com/duckassistbot.json](https://duckduckgo.com/duckassistbot.json)
- Operator documentation: [https://duckduckgo.com/duckduckgo-help-pages/results/duckassistbot](https://duckduckgo.com/duckduckgo-help-pages/results/duckassistbot)
- Last verified: 2026-09-30
## How often is DuckAssistBot impersonated?
For the median organisation, 10% of requests claiming to be DuckAssistBot failed verification.
Measured between 14 July 2026 and 10 October 2026 across organisations protected by Centinel. We only publish a share when at least five organisations each saw enough of these requests. Pooled across all of their requests, the share was 3%: a few large organisations carry most of the traffic, so the median organisation is the better guide to what yours will see.
This directory measures requests between 14 July 2026 and 10 October 2026. The [fake crawler report](https://www.centinelanalytica.com/blog/fake-crawler-report) covers its own, earlier window (late June to 21 September 2026), so its figures differ from the ones here. Read it for the method.
## Should you block DuckAssistBot?
- Allow it if: you want DuckDuckGo's AI answers to cite your pages.
- Block it if: you do not want your pages used in AI-assisted answers.
- What blocking changes: Your pages stop being used for DuckDuckGo's AI-assisted answers. Regular search uses DuckDuckBot.
To block it, add this group to your robots.txt. DuckDuckGo says a robots.txt opt-out takes effect within 72 hours.
```
User-agent: DuckAssistBot
Disallow: /
```
robots.txt only asks. It does nothing against a client that ignores it or only pretends to be DuckAssistBot. Stopping those takes verification at your edge.
## Related bots
- [DuckDuckBot](https://www.centinelanalytica.com/bots/duckduckbot) (DuckDuckGo): Crawls pages for DuckDuckGo search.
- [Claude-SearchBot](https://www.centinelanalytica.com/bots/claude-searchbot) (Anthropic): Indexes pages to improve Claude's search results.
- [Meta-WebIndexer](https://www.centinelanalytica.com/bots/meta-webindexer) (Meta): Crawls pages to improve the search results of Meta AI.
- [MistralAI-Index](https://www.centinelanalytica.com/bots/mistralai-index) (Mistral AI): Indexes pages for Mistral search.
- [OAI-SearchBot](https://www.centinelanalytica.com/bots/oai-searchbot) (OpenAI): Surfaces websites in ChatGPT's search features.
- [PerplexityBot](https://www.centinelanalytica.com/bots/perplexitybot) (Perplexity): Surfaces and links websites in Perplexity's search results.
---
Page: https://www.centinelanalytica.com/bots/duckduckbot
# What is DuckDuckBot?
DuckDuckBot is DuckDuckGo's web crawler for its search results.
- Operator: DuckDuckGo
- Purpose: Crawls pages for DuckDuckGo search.
- robots.txt token: `DuckDuckBot`
- Example user agent: `DuckDuckBot/1.1; (+http://duckduckgo.com/duckduckbot.html)`
- How to verify: Check that the source IP is in the ranges DuckDuckGo publishes for DuckDuckBot.
- IP ranges: [https://duckduckgo.com/duckduckbot.json](https://duckduckgo.com/duckduckbot.json)
- Operator documentation: [https://duckduckgo.com/duckduckgo-help-pages/results/duckduckbot](https://duckduckgo.com/duckduckgo-help-pages/results/duckduckbot)
- Last verified: 2026-09-30
## How often is DuckDuckBot impersonated?
For the median organisation, under 1% of requests claiming to be DuckDuckBot failed verification.
Measured between 14 July 2026 and 10 October 2026 across organisations protected by Centinel. We only publish a share when at least five organisations each saw enough of these requests. Pooled across all of their requests, the share was under 1%: a few large organisations carry most of the traffic, so the median organisation is the better guide to what yours will see.
This directory measures requests between 14 July 2026 and 10 October 2026. The [fake crawler report](https://www.centinelanalytica.com/blog/fake-crawler-report) covers its own, earlier window (late June to 21 September 2026), so its figures differ from the ones here. Read it for the method.
## Should you block DuckDuckBot?
- Allow it if: you want to appear in DuckDuckGo search.
- Block it if: a section should never appear in DuckDuckGo.
- What blocking changes: DuckDuckGo stops crawling your pages for its search results.
To block it, add this group to your robots.txt. DuckDuckGo documents robots.txt control for DuckDuckBot.
```
User-agent: DuckDuckBot
Disallow: /
```
robots.txt only asks. It does nothing against a client that ignores it or only pretends to be DuckDuckBot. Stopping those takes verification at your edge.
## Related bots
- [DuckAssistBot](https://www.centinelanalytica.com/bots/duckassistbot) (DuckDuckGo): Crawls in real time to support DuckDuckGo's AI-assisted answers.
- [360Spider](https://www.centinelanalytica.com/bots/360spider) (360 Search): Crawls pages for the web search of 360 Search (so.com).
- [Applebot](https://www.centinelanalytica.com/bots/applebot) (Apple): Crawls pages for Spotlight, Siri and Safari search features.
- [Baiduspider](https://www.centinelanalytica.com/bots/baiduspider) (Baidu): Crawls pages for Baidu search.
- [Bingbot](https://www.centinelanalytica.com/bots/bingbot) (Microsoft): Crawls pages for Bing search.
- [Diffbot](https://www.centinelanalytica.com/bots/diffbot) (Diffbot): Crawls the web for the Diffbot Knowledge Graph and Diffbot's web search services.
---
Page: https://www.centinelanalytica.com/bots/facebookexternalhit
# What is facebookexternalhit?
facebookexternalhit is Meta's link preview crawler. It fetches a page when someone shares its link in a Meta app, to build the preview.
- Operator: [Meta](https://www.centinelanalytica.com/bots/operator/meta)
- Purpose: Builds link previews when content is shared on Meta's apps.
- robots.txt token: `facebookexternalhit`
- Example user agent: `facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)`
- How to verify: Meta's crawler page gives no IP list and no reverse DNS domain. It says to add the user agent strings or the IP addresses used by the crawler to your allow list, and publishes no list of those addresses.
- Operator documentation: [https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/)
- Last verified: 2026-10-10
## How often is facebookexternalhit impersonated?
Other apps also put "facebookexternalhit" in their user agent, so a request carrying the name is not always a claim to be Meta's bot. We show no share.
This directory measures requests between 14 July 2026 and 10 October 2026. The [fake crawler report](https://www.centinelanalytica.com/blog/fake-crawler-report) covers its own, earlier window (late June to 21 September 2026), so its figures differ from the ones here. Read it for the method.
## Should you block facebookexternalhit?
- Allow it if: people share your pages in Meta's apps.
- Block it if: you never want previews on Meta's apps. Other link preview clients also use this name.
- What blocking changes: Links to your pages shared in Meta's apps lose their preview.
To block it, add this group to your robots.txt. Meta says this crawler may bypass robots.txt when it performs security or integrity checks.
```
User-agent: facebookexternalhit
Disallow: /
```
robots.txt only asks. It does nothing against a client that ignores it or only pretends to be facebookexternalhit. Stopping those takes verification at your edge.
## Related bots
- [Meta-ExternalAgent](https://www.centinelanalytica.com/bots/meta-externalagent) (Meta): Crawls for training AI models or improving Meta products.
- [Meta-ExternalFetcher](https://www.centinelanalytica.com/bots/meta-externalfetcher) (Meta): Fetches individual links on behalf of Meta product users.
- [Meta-WebIndexer](https://www.centinelanalytica.com/bots/meta-webindexer) (Meta): Crawls pages to improve the search results of Meta AI.
- [Discordbot](https://www.centinelanalytica.com/bots/discordbot) (Discord): Visits a link shared in a Discord message to build its preview.
- [Pinterestbot](https://www.centinelanalytica.com/bots/pinterestbot) (Pinterest): Indexes pages behind Pins and keeps Pin details up to date.
- [Slackbot-LinkExpanding](https://www.centinelanalytica.com/bots/slackbot-linkexpanding) (Slack): Fetches a page's meta tags when someone posts its link in Slack.
---
Page: https://www.centinelanalytica.com/bots/google-cloudvertexbot
# What is Google-CloudVertexBot?
Google-CloudVertexBot crawls a site when its owner requests it while building Vertex AI Agents. Google says it has no effect on Google Search or other products.
- Operator: [Google](https://www.centinelanalytica.com/bots/operator/google)
- Purpose: Crawls sites whose owners asked for it when building Vertex AI Agents.
- robots.txt token: `Google-CloudVertexBot`
- How to verify: Run a reverse DNS lookup on the source IP and check that the name ends in googlebot.com (Google uses crawl-….googlebot.com and geo-crawl-….geo.googlebot.com), then run a forward lookup on that name and confirm it returns the same IP. Or check the IP against Google's common crawler ranges. A googleusercontent.com name does not count: any Google Cloud virtual machine can have one.
- Reverse DNS: googlebot.com
- IP ranges: [https://developers.google.com/static/crawling/ipranges/common-crawlers.json](https://developers.google.com/static/crawling/ipranges/common-crawlers.json)
- Operator documentation: [https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers)
- Last verified: 2026-09-30
## How often is Google-CloudVertexBot impersonated?
For the median organisation, 100% of requests claiming to be Google-CloudVertexBot failed verification.
Measured between 14 July 2026 and 10 October 2026 across organisations protected by Centinel. We only publish a share when at least five organisations each saw enough of these requests. Pooled across all of their requests, the share was over 99%: a few large organisations carry most of the traffic, so the median organisation is the better guide to what yours will see.
This directory measures requests between 14 July 2026 and 10 October 2026. The [fake crawler report](https://www.centinelanalytica.com/blog/fake-crawler-report) covers its own, earlier window (late June to 21 September 2026), so its figures differ from the ones here. Read it for the method.
## Should you block Google-CloudVertexBot?
- Allow it if: your organization builds Vertex AI Agents on this site's content.
- Block it if: nobody in your organization requested these crawls. Verify unrequested traffic with this name before you decide.
- What blocking changes: Vertex AI Agents that you or your team set up can no longer crawl the site. Search is unaffected.
To block it, add this group to your robots.txt. Google says its common crawlers always obey robots.txt when crawling automatically.
```
User-agent: Google-CloudVertexBot
Disallow: /
```
robots.txt only asks. It does nothing against a client that ignores it or only pretends to be Google-CloudVertexBot. Stopping those takes verification at your edge.
## Related bots
- [AdsBot-Google](https://www.centinelanalytica.com/bots/adsbot-google) (Google): Checks the quality of ad landing pages for Google Ads.
- [Google-Extended](https://www.centinelanalytica.com/bots/google-extended) (Google): A robots.txt token that controls Gemini training and grounding use.
- [Google-InspectionTool](https://www.centinelanalytica.com/bots/google-inspectiontool) (Google): Fetches pages for Search Console URL inspection and the Rich Results Test.
- [Googlebot-Image](https://www.centinelanalytica.com/bots/googlebot-image) (Google): Crawls images for Google Images and image features in Search.
- [Googlebot-News](https://www.centinelanalytica.com/bots/googlebot-news) (Google): Controls whether your pages appear in Google News.
- [Googlebot-Video](https://www.centinelanalytica.com/bots/googlebot-video) (Google): Crawls video for the video features of Google Search.
---
Page: https://www.centinelanalytica.com/bots/google-extended
# What is Google-Extended?
Google-Extended is a robots.txt control token, not a crawler. Google says it manages whether content Google crawls from your site may be used to train future Gemini models and for grounding in Gemini Apps and Vertex AI. It has no user agent string of its own: the crawling is done by existing Google crawlers.
- Operator: [Google](https://www.centinelanalytica.com/bots/operator/google)
- Purpose: A robots.txt token that controls Gemini training and grounding use.
- robots.txt token: `Google-Extended`
- How to verify: There is nothing to verify: no Google crawler sends Google-Extended as its user agent. A request that does is not from Google.
- Operator documentation: [https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers)
- Last verified: 2026-09-30
## How often is Google-Extended impersonated?
No real crawler sends Google-Extended as its user agent, so every request that claims it is fake; there is nothing to verify.
This directory measures requests between 14 July 2026 and 10 October 2026. The [fake crawler report](https://www.centinelanalytica.com/blog/fake-crawler-report) covers its own, earlier window (late June to 21 September 2026), so its figures differ from the ones here. Read it for the method.
## Should you block Google-Extended?
- Allow it if: you are content for Gemini training and grounding to use your pages.
- Block it if: you want to stay in Google Search but opt out of Gemini training and grounding.
- What blocking changes: Google may not use your content to train future Gemini models or for grounding in Gemini Apps and Vertex AI. Google Search is unaffected.
To block it, add this group to your robots.txt. Google says Google-Extended does not affect inclusion in Google Search and is not a ranking signal.
```
User-agent: Google-Extended
Disallow: /
```
robots.txt only asks. It does nothing against a client that ignores it or only pretends to be Google-Extended. Stopping those takes verification at your edge.
## Related bots
- [AdsBot-Google](https://www.centinelanalytica.com/bots/adsbot-google) (Google): Checks the quality of ad landing pages for Google Ads.
- [Google-CloudVertexBot](https://www.centinelanalytica.com/bots/google-cloudvertexbot) (Google): Crawls sites whose owners asked for it when building Vertex AI Agents.
- [Google-InspectionTool](https://www.centinelanalytica.com/bots/google-inspectiontool) (Google): Fetches pages for Search Console URL inspection and the Rich Results Test.
- [Googlebot-Image](https://www.centinelanalytica.com/bots/googlebot-image) (Google): Crawls images for Google Images and image features in Search.
- [Googlebot-News](https://www.centinelanalytica.com/bots/googlebot-news) (Google): Controls whether your pages appear in Google News.
- [Googlebot-Video](https://www.centinelanalytica.com/bots/googlebot-video) (Google): Crawls video for the video features of Google Search.
---
Page: https://www.centinelanalytica.com/bots/google-inspectiontool
# What is Google-InspectionTool?
Google-InspectionTool is the crawler behind Google's Search testing tools, such as URL inspection in Search Console and the Rich Results Test. Google says it has no effect on Google Search or other products.
- Operator: [Google](https://www.centinelanalytica.com/bots/operator/google)
- Purpose: Fetches pages for Search Console URL inspection and the Rich Results Test.
- robots.txt token: `Google-InspectionTool`
- Example user agent: `Mozilla/5.0 (compatible; Google-InspectionTool/1.0)`
- How to verify: Run a reverse DNS lookup on the source IP and check that the name ends in googlebot.com (Google uses crawl-….googlebot.com and geo-crawl-….geo.googlebot.com), then run a forward lookup on that name and confirm it returns the same IP. Or check the IP against Google's common crawler ranges. A googleusercontent.com name does not count: any Google Cloud virtual machine can have one.
- Reverse DNS: googlebot.com
- IP ranges: [https://developers.google.com/static/crawling/ipranges/common-crawlers.json](https://developers.google.com/static/crawling/ipranges/common-crawlers.json)
- Operator documentation: [https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers)
- Last verified: 2026-09-30
## How often is Google-InspectionTool impersonated?
For the median organisation, 99% of requests claiming to be Google-InspectionTool failed verification.
Measured between 14 July 2026 and 10 October 2026 across organisations protected by Centinel. We only publish a share when at least five organisations each saw enough of these requests. Pooled across all of their requests, the share was 99%: a few large organisations carry most of the traffic, so the median organisation is the better guide to what yours will see.
This directory measures requests between 14 July 2026 and 10 October 2026. The [fake crawler report](https://www.centinelanalytica.com/blog/fake-crawler-report) covers its own, earlier window (late June to 21 September 2026), so its figures differ from the ones here. Read it for the method.
## Should you block Google-InspectionTool?
- Allow it if: your team uses Search Console or Google's testing tools.
- Block it if: you never use Google's testing tools. In our data most requests with this name fail verification, so verify before you decide.
- What blocking changes: Search Console's URL inspection and the Rich Results Test can no longer fetch your pages. Search itself is unaffected.
To block it, add this group to your robots.txt. Google says its common crawlers always obey robots.txt when crawling automatically.
```
User-agent: Google-InspectionTool
Disallow: /
```
robots.txt only asks. It does nothing against a client that ignores it or only pretends to be Google-InspectionTool. Stopping those takes verification at your edge.
## Related bots
- [AdsBot-Google](https://www.centinelanalytica.com/bots/adsbot-google) (Google): Checks the quality of ad landing pages for Google Ads.
- [Google-CloudVertexBot](https://www.centinelanalytica.com/bots/google-cloudvertexbot) (Google): Crawls sites whose owners asked for it when building Vertex AI Agents.
- [Google-Extended](https://www.centinelanalytica.com/bots/google-extended) (Google): A robots.txt token that controls Gemini training and grounding use.
- [Googlebot-Image](https://www.centinelanalytica.com/bots/googlebot-image) (Google): Crawls images for Google Images and image features in Search.
- [Googlebot-News](https://www.centinelanalytica.com/bots/googlebot-news) (Google): Controls whether your pages appear in Google News.
- [Googlebot-Video](https://www.centinelanalytica.com/bots/googlebot-video) (Google): Crawls video for the video features of Google Search.
---
Page: https://www.centinelanalytica.com/bots/googlebot-image
# What is Googlebot-Image?
Googlebot-Image is Google's image crawler. Google says rules addressed to it affect Google Images, Discover, Google Video and every Search feature that shows images, logos or favicons. If you have no Googlebot-Image group in robots.txt, it follows your Googlebot rules.
- Operator: [Google](https://www.centinelanalytica.com/bots/operator/google)
- Purpose: Crawls images for Google Images and image features in Search.
- robots.txt token: `Googlebot-Image`
- Example user agent: `Googlebot-Image/1.0`
- How to verify: Run a reverse DNS lookup on the source IP and check that the name ends in googlebot.com (Google uses crawl-….googlebot.com and geo-crawl-….geo.googlebot.com), then run a forward lookup on that name and confirm it returns the same IP. Or check the IP against Google's common crawler ranges. A googleusercontent.com name does not count: any Google Cloud virtual machine can have one.
- Reverse DNS: googlebot.com
- IP ranges: [https://developers.google.com/static/crawling/ipranges/common-crawlers.json](https://developers.google.com/static/crawling/ipranges/common-crawlers.json)
- Operator documentation: [https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers)
- Last verified: 2026-09-30
## How often is Googlebot-Image impersonated?
For the median organisation, 1% of requests claiming to be Googlebot-Image failed verification.
Measured between 14 July 2026 and 10 October 2026 across organisations protected by Centinel. We only publish a share when at least five organisations each saw enough of these requests. Pooled across all of their requests, the share was under 1%: a few large organisations carry most of the traffic, so the median organisation is the better guide to what yours will see.
This directory measures requests between 14 July 2026 and 10 October 2026. The [fake crawler report](https://www.centinelanalytica.com/blog/fake-crawler-report) covers its own, earlier window (late June to 21 September 2026), so its figures differ from the ones here. Read it for the method.
## Should you block Googlebot-Image?
- Allow it if: image search sends you visitors or your images carry your brand.
- Block it if: specific image directories should not appear in Google Images.
- What blocking changes: Your images stop being crawled for Google Images, Discover and the Search features that show images, logos and favicons. Your pages stay in web search.
To block it, add this group to your robots.txt. Google says its common crawlers always obey robots.txt when crawling automatically.
```
User-agent: Googlebot-Image
Disallow: /
```
robots.txt only asks. It does nothing against a client that ignores it or only pretends to be Googlebot-Image. Stopping those takes verification at your edge.
## Related bots
- [AdsBot-Google](https://www.centinelanalytica.com/bots/adsbot-google) (Google): Checks the quality of ad landing pages for Google Ads.
- [Google-CloudVertexBot](https://www.centinelanalytica.com/bots/google-cloudvertexbot) (Google): Crawls sites whose owners asked for it when building Vertex AI Agents.
- [Google-Extended](https://www.centinelanalytica.com/bots/google-extended) (Google): A robots.txt token that controls Gemini training and grounding use.
- [Google-InspectionTool](https://www.centinelanalytica.com/bots/google-inspectiontool) (Google): Fetches pages for Search Console URL inspection and the Rich Results Test.
- [Googlebot-News](https://www.centinelanalytica.com/bots/googlebot-news) (Google): Controls whether your pages appear in Google News.
- [Googlebot-Video](https://www.centinelanalytica.com/bots/googlebot-video) (Google): Crawls video for the video features of Google Search.
---
Page: https://www.centinelanalytica.com/bots/googlebot-news
# What is Googlebot-News?
Googlebot-News is the robots.txt token for Google News. It has no user-agent string of its own: Google crawls for News with its Googlebot user-agent strings. Google says rules addressed to Googlebot-News affect the Google News product, including news.google.com and the Google News app.
- Operator: [Google](https://www.centinelanalytica.com/bots/operator/google)
- Purpose: Controls whether your pages appear in Google News.
- robots.txt token: `Googlebot-News`
- How to verify: No request carries Googlebot-News as its user agent. The crawl comes from Googlebot, so verify it as Googlebot: run a reverse DNS lookup on the source IP, check that the name ends in googlebot.com, then run a forward lookup on that name and confirm it returns the same IP. Or check the IP against Google's common crawler ranges.
- Reverse DNS: googlebot.com
- IP ranges: [https://developers.google.com/static/crawling/ipranges/common-crawlers.json](https://developers.google.com/static/crawling/ipranges/common-crawlers.json)
- Operator documentation: [https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers)
- Last verified: 2026-10-10
## How often is Googlebot-News impersonated?
No real crawler sends Googlebot-News as its user agent, so every request that claims it is fake; there is nothing to verify.
This directory measures requests between 14 July 2026 and 10 October 2026. The [fake crawler report](https://www.centinelanalytica.com/blog/fake-crawler-report) covers its own, earlier window (late June to 21 September 2026), so its figures differ from the ones here. Read it for the method.
## Should you block Googlebot-News?
- Allow it if: you want your pages in Google News.
- Block it if: you want to stay out of Google News. Address the rule to Googlebot-News, not to Googlebot, to keep it apart from Google Search.
- What blocking changes: Google says that blocking Googlebot-News in robots.txt prevents your site from appearing in Google News.
To block it, add this group to your robots.txt. Google says its common crawlers always obey robots.txt when crawling automatically.
```
User-agent: Googlebot-News
Disallow: /
```
robots.txt only asks. It does nothing against a client that ignores it or only pretends to be Googlebot-News. Stopping those takes verification at your edge.
## Related bots
- [AdsBot-Google](https://www.centinelanalytica.com/bots/adsbot-google) (Google): Checks the quality of ad landing pages for Google Ads.
- [Google-CloudVertexBot](https://www.centinelanalytica.com/bots/google-cloudvertexbot) (Google): Crawls sites whose owners asked for it when building Vertex AI Agents.
- [Google-Extended](https://www.centinelanalytica.com/bots/google-extended) (Google): A robots.txt token that controls Gemini training and grounding use.
- [Google-InspectionTool](https://www.centinelanalytica.com/bots/google-inspectiontool) (Google): Fetches pages for Search Console URL inspection and the Rich Results Test.
- [Googlebot-Image](https://www.centinelanalytica.com/bots/googlebot-image) (Google): Crawls images for Google Images and image features in Search.
- [Googlebot-Video](https://www.centinelanalytica.com/bots/googlebot-video) (Google): Crawls video for the video features of Google Search.
---
Page: https://www.centinelanalytica.com/bots/googlebot-video
# What is Googlebot-Video?
Googlebot-Video is Google's crawler for video. Google says rules addressed to Googlebot-Video affect video-related Google Search features and other products that depend on videos.
- Operator: [Google](https://www.centinelanalytica.com/bots/operator/google)
- Purpose: Crawls video for the video features of Google Search.
- robots.txt token: `Googlebot-Video`
- Example user agent: `Googlebot-Video/1.0`
- How to verify: Run a reverse DNS lookup on the source IP and check that the name ends in googlebot.com (Google uses crawl-….googlebot.com and geo-crawl-….geo.googlebot.com), then run a forward lookup on that name and confirm it returns the same IP. Or check the IP against Google's common crawler ranges. A googleusercontent.com name does not count: any Google Cloud virtual machine can have one.
- Reverse DNS: googlebot.com
- IP ranges: [https://developers.google.com/static/crawling/ipranges/common-crawlers.json](https://developers.google.com/static/crawling/ipranges/common-crawlers.json)
- Operator documentation: [https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers)
- Last verified: 2026-10-10
## How often is Googlebot-Video impersonated?
Too few organisations saw requests claiming to be Googlebot-Video in the measurement window to publish a share.
This directory measures requests between 14 July 2026 and 10 October 2026. The [fake crawler report](https://www.centinelanalytica.com/blog/fake-crawler-report) covers its own, earlier window (late June to 21 September 2026), so its figures differ from the ones here. Read it for the method.
## Should you block Googlebot-Video?
- Allow it if: you publish video and want it in Google's video results.
- Block it if: you do not want your videos in Google's video features.
- What blocking changes: Your videos can drop out of the video features of Google Search and of other Google products that depend on videos.
To block it, add this group to your robots.txt. Google says its common crawlers always obey robots.txt when crawling automatically.
```
User-agent: Googlebot-Video
Disallow: /
```
robots.txt only asks. It does nothing against a client that ignores it or only pretends to be Googlebot-Video. Stopping those takes verification at your edge.
## Related bots
- [AdsBot-Google](https://www.centinelanalytica.com/bots/adsbot-google) (Google): Checks the quality of ad landing pages for Google Ads.
- [Google-CloudVertexBot](https://www.centinelanalytica.com/bots/google-cloudvertexbot) (Google): Crawls sites whose owners asked for it when building Vertex AI Agents.
- [Google-Extended](https://www.centinelanalytica.com/bots/google-extended) (Google): A robots.txt token that controls Gemini training and grounding use.
- [Google-InspectionTool](https://www.centinelanalytica.com/bots/google-inspectiontool) (Google): Fetches pages for Search Console URL inspection and the Rich Results Test.
- [Googlebot-Image](https://www.centinelanalytica.com/bots/googlebot-image) (Google): Crawls images for Google Images and image features in Search.
- [Googlebot-News](https://www.centinelanalytica.com/bots/googlebot-news) (Google): Controls whether your pages appear in Google News.
---
Page: https://www.centinelanalytica.com/bots/googlebot
# What is Googlebot?
Googlebot is Google's main search crawler. Google says rules addressed to Googlebot affect Google Search, including Discover and all Search features, and other products such as Google Images, Google Video and Google News.
- Operator: [Google](https://www.centinelanalytica.com/bots/operator/google)
- Purpose: Crawls pages for Google Search, Images, Video, News and Discover.
- robots.txt token: `Googlebot`
- Example user agent: `Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)`
- How to verify: Run a reverse DNS lookup on the source IP and check that the name ends in googlebot.com (Google uses crawl-….googlebot.com and geo-crawl-….geo.googlebot.com), then run a forward lookup on that name and confirm it returns the same IP. Or check the IP against Google's common crawler ranges. A googleusercontent.com name does not count: any Google Cloud virtual machine can have one.
- Reverse DNS: googlebot.com
- IP ranges: [https://developers.google.com/static/crawling/ipranges/common-crawlers.json](https://developers.google.com/static/crawling/ipranges/common-crawlers.json)
- Operator documentation: [https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers)
- Last verified: 2026-09-30
## How often is Googlebot impersonated?
For the median organisation, 10% of requests claiming to be Googlebot failed verification.
Measured between 14 July 2026 and 10 October 2026 across organisations protected by Centinel. We only publish a share when at least five organisations each saw enough of these requests. Pooled across all of their requests, the share was 5%: a few large organisations carry most of the traffic, so the median organisation is the better guide to what yours will see.
This directory measures requests between 14 July 2026 and 10 October 2026. The [fake crawler report](https://www.centinelanalytica.com/blog/fake-crawler-report) covers its own, earlier window (late June to 21 September 2026), so its figures differ from the ones here. Read it for the method.
## Should you block Googlebot?
- Allow it if: you want to appear in Google Search. This is nearly every public site.
- Block it if: a section should never appear in Google, such as internal search results. Block the path, not the crawler.
- What blocking changes: Your pages drop out of Google Search and the other products Googlebot feeds. For almost every public site that is not the goal.
To block it, add this group to your robots.txt. Google says its common crawlers always obey robots.txt when crawling automatically.
```
User-agent: Googlebot
Disallow: /
```
robots.txt only asks. It does nothing against a client that ignores it or only pretends to be Googlebot. Stopping those takes verification at your edge.
## Related bots
- [AdsBot-Google](https://www.centinelanalytica.com/bots/adsbot-google) (Google): Checks the quality of ad landing pages for Google Ads.
- [Google-CloudVertexBot](https://www.centinelanalytica.com/bots/google-cloudvertexbot) (Google): Crawls sites whose owners asked for it when building Vertex AI Agents.
- [Google-Extended](https://www.centinelanalytica.com/bots/google-extended) (Google): A robots.txt token that controls Gemini training and grounding use.
- [Google-InspectionTool](https://www.centinelanalytica.com/bots/google-inspectiontool) (Google): Fetches pages for Search Console URL inspection and the Rich Results Test.
- [Googlebot-Image](https://www.centinelanalytica.com/bots/googlebot-image) (Google): Crawls images for Google Images and image features in Search.
- [Googlebot-News](https://www.centinelanalytica.com/bots/googlebot-news) (Google): Controls whether your pages appear in Google News.
---
Page: https://www.centinelanalytica.com/bots/googleother
# What is GoogleOther?
GoogleOther is a generic Google crawler. Google says its product teams may use it to fetch publicly accessible content from sites, for example in one-off crawls for internal research and development. Rules addressed to GoogleOther do not affect any specific Google product.
- Operator: [Google](https://www.centinelanalytica.com/bots/operator/google)
- Purpose: Fetches public pages for Google product teams, for example for research crawls.
- robots.txt token: `GoogleOther`
- Example user agent: `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GoogleOther) Chrome/W.X.Y.Z Safari/537.36`
- How to verify: Run a reverse DNS lookup on the source IP and check that the name ends in googlebot.com (Google uses crawl-….googlebot.com and geo-crawl-….geo.googlebot.com), then run a forward lookup on that name and confirm it returns the same IP. Or check the IP against Google's common crawler ranges. A googleusercontent.com name does not count: any Google Cloud virtual machine can have one.
- Reverse DNS: googlebot.com
- IP ranges: [https://developers.google.com/static/crawling/ipranges/common-crawlers.json](https://developers.google.com/static/crawling/ipranges/common-crawlers.json)
- Operator documentation: [https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers)
- Last verified: 2026-10-10
## How often is GoogleOther impersonated?
For the median organisation, 11% of requests claiming to be GoogleOther failed verification.
Measured between 14 July 2026 and 10 October 2026 across organisations protected by Centinel. We only publish a share when at least five organisations each saw enough of these requests. Pooled across all of their requests, the share was under 1%: a few large organisations carry most of the traffic, so the median organisation is the better guide to what yours will see.
This directory measures requests between 14 July 2026 and 10 October 2026. The [fake crawler report](https://www.centinelanalytica.com/blog/fake-crawler-report) covers its own, earlier window (late June to 21 September 2026), so its figures differ from the ones here. Read it for the method.
## Should you block GoogleOther?
- Allow it if: the crawl load is not a concern.
- Block it if: you do not want Google's research and development crawls on your site. A rule for GoogleOther does not change how Googlebot crawls.
- What blocking changes: GoogleOther stops fetching your pages. Google says rules addressed to GoogleOther do not affect any specific product.
To block it, add this group to your robots.txt. Google says its common crawlers always obey robots.txt when crawling automatically.
```
User-agent: GoogleOther
Disallow: /
```
robots.txt only asks. It does nothing against a client that ignores it or only pretends to be GoogleOther. Stopping those takes verification at your edge.
## Related bots
- [AdsBot-Google](https://www.centinelanalytica.com/bots/adsbot-google) (Google): Checks the quality of ad landing pages for Google Ads.
- [Google-CloudVertexBot](https://www.centinelanalytica.com/bots/google-cloudvertexbot) (Google): Crawls sites whose owners asked for it when building Vertex AI Agents.
- [Google-Extended](https://www.centinelanalytica.com/bots/google-extended) (Google): A robots.txt token that controls Gemini training and grounding use.
- [Google-InspectionTool](https://www.centinelanalytica.com/bots/google-inspectiontool) (Google): Fetches pages for Search Console URL inspection and the Rich Results Test.
- [Googlebot-Image](https://www.centinelanalytica.com/bots/googlebot-image) (Google): Crawls images for Google Images and image features in Search.
- [Googlebot-News](https://www.centinelanalytica.com/bots/googlebot-news) (Google): Controls whether your pages appear in Google News.
---
Page: https://www.centinelanalytica.com/bots/gptbot
# What is GPTBot?
GPTBot is OpenAI's web crawler for model training. OpenAI says it crawls content that may be used in training its generative AI foundation models, and that disallowing GPTBot indicates a site's content should not be used for that training.
- Operator: [OpenAI](https://www.centinelanalytica.com/bots/operator/openai)
- Purpose: Crawls content that may be used to train OpenAI's foundation models.
- robots.txt token: `GPTBot`
- Example user agent: `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot`
- How to verify: Check that the source IP is in the ranges OpenAI publishes for GPTBot.
- IP ranges: [https://openai.com/gptbot.json](https://openai.com/gptbot.json)
- Operator documentation: [https://platform.openai.com/docs/bots](https://platform.openai.com/docs/bots)
- Last verified: 2026-09-30
## How often is GPTBot impersonated?
For the median organisation, 94% of requests claiming to be GPTBot failed verification.
Measured between 14 July 2026 and 10 October 2026 across organisations protected by Centinel. We only publish a share when at least five organisations each saw enough of these requests. Pooled across all of their requests, the share was 3%: a few large organisations carry most of the traffic, so the median organisation is the better guide to what yours will see.
This directory measures requests between 14 July 2026 and 10 October 2026. The [fake crawler report](https://www.centinelanalytica.com/blog/fake-crawler-report) covers its own, earlier window (late June to 21 September 2026), so its figures differ from the ones here. Read it for the method.
## Should you block GPTBot?
- Allow it if: you are content for your pages to be used in OpenAI model training.
- Block it if: you do not want your content used for training, while still allowing OAI-SearchBot so ChatGPT search can cite you.
- What blocking changes: It tells OpenAI not to use your content for training. OpenAI documents GPTBot, OAI-SearchBot and ChatGPT-User as independent settings, so blocking GPTBot does not remove you from ChatGPT search.
To block it, add this group to your robots.txt. OpenAI uses the GPTBot robots.txt token to control training use.
```
User-agent: GPTBot
Disallow: /
```
robots.txt only asks. It does nothing against a client that ignores it or only pretends to be GPTBot. Stopping those takes verification at your edge.
## Related bots
- [ChatGPT-User](https://www.centinelanalytica.com/bots/chatgpt-user) (OpenAI): Visits a page when a ChatGPT or Custom GPT user's request needs it.
- [OAI-SearchBot](https://www.centinelanalytica.com/bots/oai-searchbot) (OpenAI): Surfaces websites in ChatGPT's search features.
- [Amazonbot](https://www.centinelanalytica.com/bots/amazonbot) (Amazon): Crawls to improve Amazon products; may be used to train Amazon AI models.
- [CCBot](https://www.centinelanalytica.com/bots/ccbot) (Common Crawl): Builds Common Crawl's open web archive, free for anyone to reuse.
- [ClaudeBot](https://www.centinelanalytica.com/bots/claudebot) (Anthropic): Collects web content that could contribute to Claude model training.
- [Meta-ExternalAgent](https://www.centinelanalytica.com/bots/meta-externalagent) (Meta): Crawls for training AI models or improving Meta products.
---
Page: https://www.centinelanalytica.com/bots/kagibot
# What is Kagibot?
Kagibot is the web crawler for the Kagi search engine. When a robots.txt has no rule that targets Kagibot but has one that targets Googlebot, Kagibot follows the Googlebot directives.
- Operator: Kagi
- Purpose: Crawls pages for the Kagi search engine.
- robots.txt token: `Kagibot`
- Example user agent: `Mozilla/5.0 (compatible; Kagibot/1.0; +https://kagi.com/bot)`
- How to verify: Kagi lists the 4 IP addresses its requests come from on its bot page, each with a kagibot.org host name. Compare the source IP with that list. Kagi documents no reverse DNS check.
- Operator documentation: [https://kagi.com/bot](https://kagi.com/bot)
- Last verified: 2026-10-10
## How often is Kagibot impersonated?
Too few organisations saw requests claiming to be Kagibot in the measurement window to publish a share.
This directory measures requests between 14 July 2026 and 10 October 2026. The [fake crawler report](https://www.centinelanalytica.com/blog/fake-crawler-report) covers its own, earlier window (late June to 21 September 2026), so its figures differ from the ones here. Read it for the method.
## Should you block Kagibot?
- Allow it if: you want Kagi to crawl your pages for its search engine.
- Block it if: you do not want Kagi to crawl your pages. A rule for Googlebot also applies to Kagibot unless Kagibot has a rule of its own.
- What blocking changes: Kagi does not state what a block changes. It says only that it respects rules that target Kagibot.
To block it, add this group to your robots.txt. Kagi says standard robots.txt directives that target Kagibot are respected. With no rule for Kagibot, it follows a rule that targets Googlebot.
```
User-agent: Kagibot
Disallow: /
```
robots.txt only asks. It does nothing against a client that ignores it or only pretends to be Kagibot. Stopping those takes verification at your edge.
## Related bots
- [360Spider](https://www.centinelanalytica.com/bots/360spider) (360 Search): Crawls pages for the web search of 360 Search (so.com).
- [Applebot](https://www.centinelanalytica.com/bots/applebot) (Apple): Crawls pages for Spotlight, Siri and Safari search features.
- [Baiduspider](https://www.centinelanalytica.com/bots/baiduspider) (Baidu): Crawls pages for Baidu search.
- [Bingbot](https://www.centinelanalytica.com/bots/bingbot) (Microsoft): Crawls pages for Bing search.
- [Diffbot](https://www.centinelanalytica.com/bots/diffbot) (Diffbot): Crawls the web for the Diffbot Knowledge Graph and Diffbot's web search services.
- [DuckDuckBot](https://www.centinelanalytica.com/bots/duckduckbot) (DuckDuckGo): Crawls pages for DuckDuckGo search.
---
Page: https://www.centinelanalytica.com/bots/mediapartners-google
# What is Mediapartners-Google?
Mediapartners-Google is the Google AdSense crawler. Google says it visits participating sites in order to provide them with relevant ads. Rules addressed to it affect ad products such as Google AdSense and Google Ad Manager.
- Operator: [Google](https://www.centinelanalytica.com/bots/operator/google)
- Purpose: Reads the pages of sites that show Google ads, so the ads match the content.
- robots.txt token: `Mediapartners-Google`
- Example user agent: `Mediapartners-Google`
- How to verify: Run a reverse DNS lookup on the source IP and check that the name ends in google.com (Google uses rate-limited-proxy-….google.com for special-case crawlers), then run a forward lookup on that name and confirm it returns the same IP. Or check the IP against Google's special-case crawler ranges.
- Reverse DNS: google.com
- IP ranges: [https://developers.google.com/static/crawling/ipranges/special-crawlers.json](https://developers.google.com/static/crawling/ipranges/special-crawlers.json)
- Operator documentation: [https://developers.google.com/crawling/docs/crawlers-fetchers/google-special-case-crawlers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-special-case-crawlers)
- Last verified: 2026-10-10
## How often is Mediapartners-Google impersonated?
Our verification data for Mediapartners-Google was incomplete during the measurement window, so a share would overstate impersonation. We show none until it is complete.
This directory measures requests between 14 July 2026 and 10 October 2026. The [fake crawler report](https://www.centinelanalytica.com/blog/fake-crawler-report) covers its own, earlier window (late June to 21 September 2026), so its figures differ from the ones here. Read it for the method.
## Should you block Mediapartners-Google?
- Allow it if: you show Google AdSense or Google Ad Manager ads on this site.
- Block it if: you show no Google ads. Remember that User-agent: * does not cover it.
- What blocking changes: Google says that if robots.txt disallows the AdSense crawler, it cannot serve Google ads on the site.
To block it, add this group to your robots.txt. Google says Mediapartners-Google ignores the global User-agent: * group. A rule only applies if it names Mediapartners-Google.
```
User-agent: Mediapartners-Google
Disallow: /
```
robots.txt only asks. It does nothing against a client that ignores it or only pretends to be Mediapartners-Google. Stopping those takes verification at your edge.
## Related bots
- [AdsBot-Google](https://www.centinelanalytica.com/bots/adsbot-google) (Google): Checks the quality of ad landing pages for Google Ads.
- [Google-CloudVertexBot](https://www.centinelanalytica.com/bots/google-cloudvertexbot) (Google): Crawls sites whose owners asked for it when building Vertex AI Agents.
- [Google-Extended](https://www.centinelanalytica.com/bots/google-extended) (Google): A robots.txt token that controls Gemini training and grounding use.
- [Google-InspectionTool](https://www.centinelanalytica.com/bots/google-inspectiontool) (Google): Fetches pages for Search Console URL inspection and the Rich Results Test.
- [Googlebot-Image](https://www.centinelanalytica.com/bots/googlebot-image) (Google): Crawls images for Google Images and image features in Search.
- [Googlebot-News](https://www.centinelanalytica.com/bots/googlebot-news) (Google): Controls whether your pages appear in Google News.
---
Page: https://www.centinelanalytica.com/bots/meta-externalagent
# What is Meta-ExternalAgent?
Meta-ExternalAgent is a Meta crawler. Meta says it crawls the web for use cases such as training AI models or improving products by indexing content directly.
- Operator: [Meta](https://www.centinelanalytica.com/bots/operator/meta)
- Purpose: Crawls for training AI models or improving Meta products.
- robots.txt token: `meta-externalagent`
- Example user agent: `meta-externalagent/1.1`
- How to verify: Meta's crawler page gives no IP list and no reverse DNS domain for Meta-ExternalAgent.
- Operator documentation: [https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/)
- Last verified: 2026-10-10
## How often is Meta-ExternalAgent impersonated?
No published method checks a request that claims to be Meta-ExternalAgent, so we cannot say which claims are false and show no share.
This directory measures requests between 14 July 2026 and 10 October 2026. The [fake crawler report](https://www.centinelanalytica.com/blog/fake-crawler-report) covers its own, earlier window (late June to 21 September 2026), so its figures differ from the ones here. Read it for the method.
## Should you block Meta-ExternalAgent?
- Allow it if: you are content for Meta to use your pages this way.
- Block it if: you do not want your content used for Meta's AI training.
- What blocking changes: Meta's crawler stops collecting your pages for AI training and product indexing. Link previews use a different crawler.
To block it, add this group to your robots.txt. Meta documents robots.txt control for this crawler.
```
User-agent: meta-externalagent
Disallow: /
```
robots.txt only asks. It does nothing against a client that ignores it or only pretends to be Meta-ExternalAgent. Stopping those takes verification at your edge.
## Related bots
- [facebookexternalhit](https://www.centinelanalytica.com/bots/facebookexternalhit) (Meta): Builds link previews when content is shared on Meta's apps.
- [Meta-ExternalFetcher](https://www.centinelanalytica.com/bots/meta-externalfetcher) (Meta): Fetches individual links on behalf of Meta product users.
- [Meta-WebIndexer](https://www.centinelanalytica.com/bots/meta-webindexer) (Meta): Crawls pages to improve the search results of Meta AI.
- [Amazonbot](https://www.centinelanalytica.com/bots/amazonbot) (Amazon): Crawls to improve Amazon products; may be used to train Amazon AI models.
- [CCBot](https://www.centinelanalytica.com/bots/ccbot) (Common Crawl): Builds Common Crawl's open web archive, free for anyone to reuse.
- [ClaudeBot](https://www.centinelanalytica.com/bots/claudebot) (Anthropic): Collects web content that could contribute to Claude model training.
---
Page: https://www.centinelanalytica.com/bots/meta-externalfetcher
# What is Meta-ExternalFetcher?
Meta-ExternalFetcher is the Meta crawler that fetches individual links at a user's request. Meta says it supports product functions such as evaluating and improving agentic AI capabilities, including helping AI navigate websites to complete tasks for users.
- Operator: [Meta](https://www.centinelanalytica.com/bots/operator/meta)
- Purpose: Fetches individual links on behalf of Meta product users.
- robots.txt token: `meta-externalfetcher`
- Example user agent: `meta-externalfetcher/1.1`
- How to verify: Meta's crawler page gives no IP list and no reverse DNS domain for Meta-ExternalFetcher.
- Operator documentation: [https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/)
- Last verified: 2026-10-10
## How often is Meta-ExternalFetcher impersonated?
No published method checks a request that claims to be Meta-ExternalFetcher, so we cannot say which claims are false and show no share.
This directory measures requests between 14 July 2026 and 10 October 2026. The [fake crawler report](https://www.centinelanalytica.com/blog/fake-crawler-report) covers its own, earlier window (late June to 21 September 2026), so its figures differ from the ones here. Read it for the method.
## Should you block Meta-ExternalFetcher?
- Allow it if: you want Meta's AI products to be able to open your pages when a user asks.
- Block it if: your content is not meant to be fetched on demand, and you enforce that at the edge.
- What blocking changes: Meta says this fetcher may bypass robots.txt, so a Disallow line may not stop it.
To block it, add this group to your robots.txt. Meta says that because a user requested the fetch, this crawler may bypass robots.txt.
```
User-agent: meta-externalfetcher
Disallow: /
```
robots.txt only asks. It does nothing against a client that ignores it or only pretends to be Meta-ExternalFetcher. Stopping those takes verification at your edge.
## Related bots
- [facebookexternalhit](https://www.centinelanalytica.com/bots/facebookexternalhit) (Meta): Builds link previews when content is shared on Meta's apps.
- [Meta-ExternalAgent](https://www.centinelanalytica.com/bots/meta-externalagent) (Meta): Crawls for training AI models or improving Meta products.
- [Meta-WebIndexer](https://www.centinelanalytica.com/bots/meta-webindexer) (Meta): Crawls pages to improve the search results of Meta AI.
- [ChatGPT-User](https://www.centinelanalytica.com/bots/chatgpt-user) (OpenAI): Visits a page when a ChatGPT or Custom GPT user's request needs it.
- [Claude-User](https://www.centinelanalytica.com/bots/claude-user) (Anthropic): Fetches a page when a Claude user's question needs it.
- [MistralAI-User](https://www.centinelanalytica.com/bots/mistralai-user) (Mistral AI): Fetches a page when a Mistral user's action needs it.
---
Page: https://www.centinelanalytica.com/bots/meta-webindexer
# What is Meta-WebIndexer?
Meta-WebIndexer is the Meta crawler for Meta AI search. Meta says it navigates the web to improve Meta AI search result quality, and that allowing it in robots.txt helps Meta cite and link to your content in Meta AI's responses.
- Operator: [Meta](https://www.centinelanalytica.com/bots/operator/meta)
- Purpose: Crawls pages to improve the search results of Meta AI.
- robots.txt token: `meta-webindexer`
- Example user agent: `meta-webindexer/1.1`
- How to verify: Meta's crawler page gives no IP list and no reverse DNS domain for Meta-WebIndexer.
- Operator documentation: [https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/)
- Last verified: 2026-10-10
## How often is Meta-WebIndexer impersonated?
No published method checks a request that claims to be Meta-WebIndexer, so we cannot say which claims are false and show no share.
This directory measures requests between 14 July 2026 and 10 October 2026. The [fake crawler report](https://www.centinelanalytica.com/blog/fake-crawler-report) covers its own, earlier window (late June to 21 September 2026), so its figures differ from the ones here. Read it for the method.
## Should you block Meta-WebIndexer?
- Allow it if: you want Meta AI to cite and link to your pages.
- Block it if: you do not want your pages indexed for Meta AI's answers.
- What blocking changes: Meta states only the allow side: allowing Meta-WebIndexer helps Meta cite and link to your content in Meta AI's responses.
To block it, add this group to your robots.txt. Meta says to add a disallow for the crawler to robots.txt to block it. Its crawlers may cache robots.txt for up to 24 hours.
```
User-agent: meta-webindexer
Disallow: /
```
robots.txt only asks. It does nothing against a client that ignores it or only pretends to be Meta-WebIndexer. Stopping those takes verification at your edge.
## Related bots
- [facebookexternalhit](https://www.centinelanalytica.com/bots/facebookexternalhit) (Meta): Builds link previews when content is shared on Meta's apps.
- [Meta-ExternalAgent](https://www.centinelanalytica.com/bots/meta-externalagent) (Meta): Crawls for training AI models or improving Meta products.
- [Meta-ExternalFetcher](https://www.centinelanalytica.com/bots/meta-externalfetcher) (Meta): Fetches individual links on behalf of Meta product users.
- [Claude-SearchBot](https://www.centinelanalytica.com/bots/claude-searchbot) (Anthropic): Indexes pages to improve Claude's search results.
- [DuckAssistBot](https://www.centinelanalytica.com/bots/duckassistbot) (DuckDuckGo): Crawls in real time to support DuckDuckGo's AI-assisted answers.
- [MistralAI-Index](https://www.centinelanalytica.com/bots/mistralai-index) (Mistral AI): Indexes pages for Mistral search.
---
Page: https://www.centinelanalytica.com/bots/mistralai-index
# What is MistralAI-Index?
MistralAI-Index is Mistral's crawler for indexing. Mistral says it crawls the web automatically for indexing purposes only, to index content for Mistral search, which helps answer user questions in Vibe. Mistral says the content it crawls is not used for generative AI training of any kind.
- Operator: Mistral AI
- Purpose: Indexes pages for Mistral search.
- robots.txt token: `MistralAI-Index`
- Example user agent: `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-Index/1.0; +https://docs.mistral.ai/robots)`
- How to verify: Check that the source IP is in the ranges Mistral publishes for MistralAI-Index.
- IP ranges: [https://mistral.ai/mistralai-index-ips.json](https://mistral.ai/mistralai-index-ips.json)
- Operator documentation: [https://docs.mistral.ai/robots](https://docs.mistral.ai/robots)
- Last verified: 2026-10-10
## How often is MistralAI-Index impersonated?
Too few organisations saw requests claiming to be MistralAI-Index in the measurement window to publish a share.
This directory measures requests between 14 July 2026 and 10 October 2026. The [fake crawler report](https://www.centinelanalytica.com/blog/fake-crawler-report) covers its own, earlier window (late June to 21 September 2026), so its figures differ from the ones here. Read it for the method.
## Should you block MistralAI-Index?
- Allow it if: you want Mistral's search to find your pages when it answers a user's question.
- Block it if: you do not want your pages in Mistral's search index.
- What blocking changes: Mistral does not state what a block changes. The crawler's stated job is to index content for Mistral search.
To block it, add this group to your robots.txt. Mistral lists MistralAI-Index among the robots.txt tags a site can use to manage how its content interacts with Mistral's AI.
```
User-agent: MistralAI-Index
Disallow: /
```
robots.txt only asks. It does nothing against a client that ignores it or only pretends to be MistralAI-Index. Stopping those takes verification at your edge.
## Related bots
- [MistralAI-User](https://www.centinelanalytica.com/bots/mistralai-user) (Mistral AI): Fetches a page when a Mistral user's action needs it.
- [Claude-SearchBot](https://www.centinelanalytica.com/bots/claude-searchbot) (Anthropic): Indexes pages to improve Claude's search results.
- [DuckAssistBot](https://www.centinelanalytica.com/bots/duckassistbot) (DuckDuckGo): Crawls in real time to support DuckDuckGo's AI-assisted answers.
- [Meta-WebIndexer](https://www.centinelanalytica.com/bots/meta-webindexer) (Meta): Crawls pages to improve the search results of Meta AI.
- [OAI-SearchBot](https://www.centinelanalytica.com/bots/oai-searchbot) (OpenAI): Surfaces websites in ChatGPT's search features.
- [PerplexityBot](https://www.centinelanalytica.com/bots/perplexitybot) (Perplexity): Surfaces and links websites in Perplexity's search results.
---
Page: https://www.centinelanalytica.com/bots/mistralai-user
# What is MistralAI-User?
MistralAI-User is the agent Mistral AI uses to fetch a page when a user's action in its assistant needs it. Mistral says it is not used for automatic crawling or to train its models.
- Operator: Mistral AI
- Purpose: Fetches a page when a Mistral user's action needs it.
- robots.txt token: `MistralAI-User`
- Example user agent: `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; MistralAI-User/1.0; +https://docs.mistral.ai/robots)`
- How to verify: Check that the source IP is in the ranges Mistral publishes for MistralAI-User.
- IP ranges: [https://mistral.ai/mistralai-user-ips.json](https://mistral.ai/mistralai-user-ips.json)
- Operator documentation: [https://docs.mistral.ai/robots](https://docs.mistral.ai/robots)
- Last verified: 2026-09-30
## How often is MistralAI-User impersonated?
Our verification data for MistralAI-User was incomplete during the measurement window, so a share would overstate impersonation. We show none until it is complete.
This directory measures requests between 14 July 2026 and 10 October 2026. The [fake crawler report](https://www.centinelanalytica.com/blog/fake-crawler-report) covers its own, earlier window (late June to 21 September 2026), so its figures differ from the ones here. Read it for the method.
## Should you block MistralAI-User?
- Allow it if: you want the assistant to be able to read and cite your pages when someone asks about them.
- Block it if: you do not want Mistral's assistant fetching your pages.
- What blocking changes: Mistral's assistant stops fetching your pages for its users.
To block it, add this group to your robots.txt. Mistral documents robots.txt control for its agents.
```
User-agent: MistralAI-User
Disallow: /
```
robots.txt only asks. It does nothing against a client that ignores it or only pretends to be MistralAI-User. Stopping those takes verification at your edge.
## Related bots
- [MistralAI-Index](https://www.centinelanalytica.com/bots/mistralai-index) (Mistral AI): Indexes pages for Mistral search.
- [ChatGPT-User](https://www.centinelanalytica.com/bots/chatgpt-user) (OpenAI): Visits a page when a ChatGPT or Custom GPT user's request needs it.
- [Claude-User](https://www.centinelanalytica.com/bots/claude-user) (Anthropic): Fetches a page when a Claude user's question needs it.
- [Meta-ExternalFetcher](https://www.centinelanalytica.com/bots/meta-externalfetcher) (Meta): Fetches individual links on behalf of Meta product users.
- [Perplexity-User](https://www.centinelanalytica.com/bots/perplexity-user) (Perplexity): Fetches a page when a Perplexity user's request needs it.
---
Page: https://www.centinelanalytica.com/bots/mj12bot
# What is MJ12bot?
MJ12bot is Majestic's crawler. It maps the link relationships between websites for the data served at Majestic.com, using a distributed, community-run crawler.
- Operator: Majestic
- Purpose: Maps links between websites for Majestic's link index.
- robots.txt token: `MJ12bot`
- How to verify: Majestic's crawler is distributed and uses no fixed IP blocks. Majestic offers a site-specific identification string instead.
- Operator documentation: [https://mj12bot.com/](https://mj12bot.com/)
- Last verified: 2026-09-30
## How often is MJ12bot impersonated?
No published method checks a request that claims to be MJ12bot, so we cannot say which claims are false and show no share.
This directory measures requests between 14 July 2026 and 10 October 2026. The [fake crawler report](https://www.centinelanalytica.com/blog/fake-crawler-report) covers its own, earlier window (late June to 21 September 2026), so its figures differ from the ones here. Read it for the method.
## Should you block MJ12bot?
- Allow it if: you or your partners use this tool's backlink or audit data, or the crawl load is not a concern.
- Block it if: the crawl load is noticeable and you have no use for the tool's data. Blocking it does not affect search rankings.
- What blocking changes: Majestic stops crawling your site for its link index.
To block it, add this group to your robots.txt. Majestic says MJ12bot obeys robots.txt and Crawl-delay up to 20 seconds.
```
User-agent: MJ12bot
Disallow: /
```
robots.txt only asks. It does nothing against a client that ignores it or only pretends to be MJ12bot. Stopping those takes verification at your edge.
## Related bots
- [AhrefsBot](https://www.centinelanalytica.com/bots/ahrefsbot) (Ahrefs): Builds the link database behind Ahrefs and the Yep search engine.
- [AhrefsSiteAudit](https://www.centinelanalytica.com/bots/ahrefssiteaudit) (Ahrefs): Crawls a site for an Ahrefs Site Audit project.
- [Barkrowler](https://www.centinelanalytica.com/bots/barkrowler) (Babbar): Collects links and page metadata for Babbar.
- [DataForSeoBot](https://www.centinelanalytica.com/bots/dataforseobot) (DataForSEO): Adds links to DataForSEO's backlink database.
- [DotBot](https://www.centinelanalytica.com/bots/dotbot) (Moz): Gathers data for the Moz Link Index.
- [Rogerbot](https://www.centinelanalytica.com/bots/rogerbot) (Moz): Crawls a site for the site audits of a Moz Pro Campaign.
---
Page: https://www.centinelanalytica.com/bots/oai-searchbot
# What is OAI-SearchBot?
OAI-SearchBot is OpenAI's search crawler. It is used to surface websites in search results in ChatGPT's search features. It is separate from GPTBot, which crawls for model training.
- Operator: [OpenAI](https://www.centinelanalytica.com/bots/operator/openai)
- Purpose: Surfaces websites in ChatGPT's search features.
- robots.txt token: `OAI-SearchBot`
- Example user agent: `Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot`
- How to verify: Check that the source IP is in the ranges OpenAI publishes for OAI-SearchBot.
- IP ranges: [https://openai.com/searchbot.json](https://openai.com/searchbot.json)
- Operator documentation: [https://platform.openai.com/docs/bots](https://platform.openai.com/docs/bots)
- Last verified: 2026-09-30
## How often is OAI-SearchBot impersonated?
For the median organisation, 11% of requests claiming to be OAI-SearchBot failed verification.
Measured between 14 July 2026 and 10 October 2026 across organisations protected by Centinel. We only publish a share when at least five organisations each saw enough of these requests. Pooled across all of their requests, the share was 4%: a few large organisations carry most of the traffic, so the median organisation is the better guide to what yours will see.
This directory measures requests between 14 July 2026 and 10 October 2026. The [fake crawler report](https://www.centinelanalytica.com/blog/fake-crawler-report) covers its own, earlier window (late June to 21 September 2026), so its figures differ from the ones here. Read it for the method.
## Should you block OAI-SearchBot?
- Allow it if: you want ChatGPT search to find and link your pages.
- Block it if: you do not want your pages shown in ChatGPT search answers at all.
- What blocking changes: OpenAI says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links.
To block it, add this group to your robots.txt. OpenAI uses the OAI-SearchBot token for search opt-outs.
```
User-agent: OAI-SearchBot
Disallow: /
```
robots.txt only asks. It does nothing against a client that ignores it or only pretends to be OAI-SearchBot. Stopping those takes verification at your edge.
## Related bots
- [ChatGPT-User](https://www.centinelanalytica.com/bots/chatgpt-user) (OpenAI): Visits a page when a ChatGPT or Custom GPT user's request needs it.
- [GPTBot](https://www.centinelanalytica.com/bots/gptbot) (OpenAI): Crawls content that may be used to train OpenAI's foundation models.
- [Claude-SearchBot](https://www.centinelanalytica.com/bots/claude-searchbot) (Anthropic): Indexes pages to improve Claude's search results.
- [DuckAssistBot](https://www.centinelanalytica.com/bots/duckassistbot) (DuckDuckGo): Crawls in real time to support DuckDuckGo's AI-assisted answers.
- [Meta-WebIndexer](https://www.centinelanalytica.com/bots/meta-webindexer) (Meta): Crawls pages to improve the search results of Meta AI.
- [MistralAI-Index](https://www.centinelanalytica.com/bots/mistralai-index) (Mistral AI): Indexes pages for Mistral search.
---
Page: https://www.centinelanalytica.com/bots/perplexity-user
# What is Perplexity-User?
Perplexity-User is the agent Perplexity uses to fetch a page when a user's question needs it. Perplexity says it is not used for automatic web crawling or for AI training.
- Operator: Perplexity
- Purpose: Fetches a page when a Perplexity user's request needs it.
- robots.txt token: `Perplexity-User`
- Example user agent: `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)`
- How to verify: Check that the source IP is in the ranges Perplexity publishes for Perplexity-User.
- IP ranges: [https://www.perplexity.ai/perplexity-user.json](https://www.perplexity.ai/perplexity-user.json)
- Operator documentation: [https://docs.perplexity.ai/guides/bots](https://docs.perplexity.ai/guides/bots)
- Last verified: 2026-09-30
## How often is Perplexity-User impersonated?
Our verification data for Perplexity-User was incomplete during the measurement window, so a share would overstate impersonation. We show none until it is complete.
This directory measures requests between 14 July 2026 and 10 October 2026. The [fake crawler report](https://www.centinelanalytica.com/blog/fake-crawler-report) covers its own, earlier window (late June to 21 September 2026), so its figures differ from the ones here. Read it for the method.
## Should you block Perplexity-User?
- Allow it if: you want the assistant to be able to read and cite your pages when someone asks about them.
- Block it if: your content is not meant to be fetched on demand by an assistant, and you enforce that at the edge.
- What blocking changes: Perplexity says Perplexity-User generally ignores robots.txt, so a Disallow line is unlikely to stop it. Enforce the rule at your edge by verified IP.
To block it, add this group to your robots.txt. Perplexity says that because a user requested the fetch, Perplexity-User generally ignores robots.txt.
```
User-agent: Perplexity-User
Disallow: /
```
robots.txt only asks. It does nothing against a client that ignores it or only pretends to be Perplexity-User. Stopping those takes verification at your edge.
## Related bots
- [PerplexityBot](https://www.centinelanalytica.com/bots/perplexitybot) (Perplexity): Surfaces and links websites in Perplexity's search results.
- [ChatGPT-User](https://www.centinelanalytica.com/bots/chatgpt-user) (OpenAI): Visits a page when a ChatGPT or Custom GPT user's request needs it.
- [Claude-User](https://www.centinelanalytica.com/bots/claude-user) (Anthropic): Fetches a page when a Claude user's question needs it.
- [Meta-ExternalFetcher](https://www.centinelanalytica.com/bots/meta-externalfetcher) (Meta): Fetches individual links on behalf of Meta product users.
- [MistralAI-User](https://www.centinelanalytica.com/bots/mistralai-user) (Mistral AI): Fetches a page when a Mistral user's action needs it.
---
Page: https://www.centinelanalytica.com/bots/perplexitybot
# What is PerplexityBot?
PerplexityBot is Perplexity's search crawler. Perplexity says it is designed to surface and link websites in its search results, and is not used to crawl content for AI foundation models.
- Operator: Perplexity
- Purpose: Surfaces and links websites in Perplexity's search results.
- robots.txt token: `PerplexityBot`
- Example user agent: `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)`
- How to verify: Check that the source IP is in the ranges Perplexity publishes for PerplexityBot.
- IP ranges: [https://www.perplexity.ai/perplexitybot.json](https://www.perplexity.ai/perplexitybot.json)
- Operator documentation: [https://docs.perplexity.ai/guides/bots](https://docs.perplexity.ai/guides/bots)
- Last verified: 2026-09-30
## How often is PerplexityBot impersonated?
For the median organisation, 22% of requests claiming to be PerplexityBot failed verification.
Measured between 14 July 2026 and 10 October 2026 across organisations protected by Centinel. We only publish a share when at least five organisations each saw enough of these requests. Pooled across all of their requests, the share was 9%: a few large organisations carry most of the traffic, so the median organisation is the better guide to what yours will see.
This directory measures requests between 14 July 2026 and 10 October 2026. The [fake crawler report](https://www.centinelanalytica.com/blog/fake-crawler-report) covers its own, earlier window (late June to 21 September 2026), so its figures differ from the ones here. Read it for the method.
## Should you block PerplexityBot?
- Allow it if: you want Perplexity's answers to find and link your pages.
- Block it if: you do not want your pages in Perplexity's search index.
- What blocking changes: Your pages stop being crawled for Perplexity's search index, so they are less likely to be surfaced and linked in its answers.
To block it, add this group to your robots.txt. Perplexity documents robots.txt as the control for PerplexityBot.
```
User-agent: PerplexityBot
Disallow: /
```
robots.txt only asks. It does nothing against a client that ignores it or only pretends to be PerplexityBot. Stopping those takes verification at your edge.
## Related bots
- [Perplexity-User](https://www.centinelanalytica.com/bots/perplexity-user) (Perplexity): Fetches a page when a Perplexity user's request needs it.
- [Claude-SearchBot](https://www.centinelanalytica.com/bots/claude-searchbot) (Anthropic): Indexes pages to improve Claude's search results.
- [DuckAssistBot](https://www.centinelanalytica.com/bots/duckassistbot) (DuckDuckGo): Crawls in real time to support DuckDuckGo's AI-assisted answers.
- [Meta-WebIndexer](https://www.centinelanalytica.com/bots/meta-webindexer) (Meta): Crawls pages to improve the search results of Meta AI.
- [MistralAI-Index](https://www.centinelanalytica.com/bots/mistralai-index) (Mistral AI): Indexes pages for Mistral search.
- [OAI-SearchBot](https://www.centinelanalytica.com/bots/oai-searchbot) (OpenAI): Surfaces websites in ChatGPT's search features.
---
Page: https://www.centinelanalytica.com/bots/petalbot
# What is PetalBot?
PetalBot is the crawler of Petal Search. It indexes PC and mobile sites for Petal Search and for content recommendations in Huawei Assistant and AI Search.
- Operator: Huawei
- Purpose: Crawls pages for Petal Search and Huawei Assistant.
- robots.txt token: `PetalBot`
- Example user agent: `Mozilla/5.0 (compatible;PetalBot;+https://webmaster.petalsearch.com/site/petalbot)`
- How to verify: Run a reverse DNS lookup and check that the name ends in petalsearch.com, then run a forward lookup on that name and confirm it returns the same IP.
- Reverse DNS: petalsearch.com
- Operator documentation: [https://webmaster.petalsearch.com/site/petalbot](https://webmaster.petalsearch.com/site/petalbot)
- Last verified: 2026-09-30
## How often is PetalBot impersonated?
For the median organisation, under 1% of requests claiming to be PetalBot failed verification.
Measured between 14 July 2026 and 10 October 2026 across organisations protected by Centinel. We only publish a share when at least five organisations each saw enough of these requests. Pooled across all of their requests, the share was under 1%: a few large organisations carry most of the traffic, so the median organisation is the better guide to what yours will see.
This directory measures requests between 14 July 2026 and 10 October 2026. The [fake crawler report](https://www.centinelanalytica.com/blog/fake-crawler-report) covers its own, earlier window (late June to 21 September 2026), so its figures differ from the ones here. Read it for the method.
## Should you block PetalBot?
- Allow it if: you have visitors on Huawei devices or Petal Search.
- Block it if: you have no audience there and the crawl adds load.
- What blocking changes: Your pages drop out of Petal Search and Huawei Assistant.
To block it, add this group to your robots.txt. The operator says PetalBot complies with robots.txt.
```
User-agent: PetalBot
Disallow: /
```
robots.txt only asks. It does nothing against a client that ignores it or only pretends to be PetalBot. Stopping those takes verification at your edge.
## Related bots
- [360Spider](https://www.centinelanalytica.com/bots/360spider) (360 Search): Crawls pages for the web search of 360 Search (so.com).
- [Applebot](https://www.centinelanalytica.com/bots/applebot) (Apple): Crawls pages for Spotlight, Siri and Safari search features.
- [Baiduspider](https://www.centinelanalytica.com/bots/baiduspider) (Baidu): Crawls pages for Baidu search.
- [Bingbot](https://www.centinelanalytica.com/bots/bingbot) (Microsoft): Crawls pages for Bing search.
- [Diffbot](https://www.centinelanalytica.com/bots/diffbot) (Diffbot): Crawls the web for the Diffbot Knowledge Graph and Diffbot's web search services.
- [DuckDuckBot](https://www.centinelanalytica.com/bots/duckduckbot) (DuckDuckGo): Crawls pages for DuckDuckGo search.
---
Page: https://www.centinelanalytica.com/bots/pinterestbot
# What is Pinterestbot?
Pinterestbot is Pinterest's crawler. It indexes public pages to drive traffic back to them, keeps Pin details such as price and title up to date, and detects broken links behind Pins.
- Operator: Pinterest
- Purpose: Indexes pages behind Pins and keeps Pin details up to date.
- robots.txt token: `Pinterestbot`
- Example user agent: `Mozilla/5.0 (compatible; Pinterestbot/1.0; +https://www.pinterest.com/bot.html)`
- How to verify: Run a reverse DNS lookup and check that the name ends in pinterest.com or pinterestcrawler.com, then confirm with a forward lookup.
- Reverse DNS: pinterest.com, pinterestcrawler.com
- Operator documentation: [https://help.pinterest.com/en/business/article/pinterest-crawler](https://help.pinterest.com/en/business/article/pinterest-crawler)
- Last verified: 2026-09-30
## How often is Pinterestbot impersonated?
For the median organisation, under 1% of requests claiming to be Pinterestbot failed verification.
Measured between 14 July 2026 and 10 October 2026 across organisations protected by Centinel. We only publish a share when at least five organisations each saw enough of these requests. Pooled across all of their requests, the share was under 1%: a few large organisations carry most of the traffic, so the median organisation is the better guide to what yours will see.
This directory measures requests between 14 July 2026 and 10 October 2026. The [fake crawler report](https://www.centinelanalytica.com/blog/fake-crawler-report) covers its own, earlier window (late June to 21 September 2026), so its figures differ from the ones here. Read it for the method.
## Should you block Pinterestbot?
- Allow it if: people save your pages to Pinterest.
- Block it if: Pinterest sends you no visitors and the crawl adds load.
- What blocking changes: Pins that link to your pages stop being refreshed and checked.
To block it, add this group to your robots.txt. Pinterest says Pinterestbot respects robots.txt and honors Crawl-delay up to 1 second.
```
User-agent: Pinterestbot
Disallow: /
```
robots.txt only asks. It does nothing against a client that ignores it or only pretends to be Pinterestbot. Stopping those takes verification at your edge.
## Related bots
- [Discordbot](https://www.centinelanalytica.com/bots/discordbot) (Discord): Visits a link shared in a Discord message to build its preview.
- [facebookexternalhit](https://www.centinelanalytica.com/bots/facebookexternalhit) (Meta): Builds link previews when content is shared on Meta's apps.
- [Slackbot-LinkExpanding](https://www.centinelanalytica.com/bots/slackbot-linkexpanding) (Slack): Fetches a page's meta tags when someone posts its link in Slack.
- [Twitterbot](https://www.centinelanalytica.com/bots/twitterbot) (X): Fetches card markup when a link is shared on X.
---
Page: https://www.centinelanalytica.com/bots/rogerbot
# What is Rogerbot?
Rogerbot is the Moz crawler for Moz Pro Campaign site audits. Moz says it is different from Dotbot, the web crawler that powers the Moz Links index.
- Operator: Moz
- Purpose: Crawls a site for the site audits of a Moz Pro Campaign.
- robots.txt token: `rogerbot`
- How to verify: Moz says it has no IP range for Rogerbot, because it does not use a static IP address or range. Moz says the best option is to identify the crawler by its user agent, rogerbot.
- Operator documentation: [https://moz.com/help/moz-procedures/crawlers/rogerbot](https://moz.com/help/moz-procedures/crawlers/rogerbot)
- Last verified: 2026-10-10
## How often is Rogerbot impersonated?
No published method checks a request that claims to be Rogerbot, so we cannot say which claims are false and show no share.
This directory measures requests between 14 July 2026 and 10 October 2026. The [fake crawler report](https://www.centinelanalytica.com/blog/fake-crawler-report) covers its own, earlier window (late June to 21 September 2026), so its figures differ from the ones here. Read it for the method.
## Should you block Rogerbot?
- Allow it if: you or your agency track this site in a Moz Pro Campaign.
- Block it if: nobody who works on this site uses Moz Pro.
- What blocking changes: Moz says Rogerbot cannot continue a crawl when it is blocked, so a Moz Pro site audit of your site stops.
To block it, add this group to your robots.txt. Moz says Rogerbot is built to obey robots.txt. It supports Crawl-delay, and Moz advises a delay of no more than 30 seconds.
```
User-agent: rogerbot
Disallow: /
```
robots.txt only asks. It does nothing against a client that ignores it or only pretends to be Rogerbot. Stopping those takes verification at your edge.
## Related bots
- [DotBot](https://www.centinelanalytica.com/bots/dotbot) (Moz): Gathers data for the Moz Link Index.
- [AhrefsBot](https://www.centinelanalytica.com/bots/ahrefsbot) (Ahrefs): Builds the link database behind Ahrefs and the Yep search engine.
- [AhrefsSiteAudit](https://www.centinelanalytica.com/bots/ahrefssiteaudit) (Ahrefs): Crawls a site for an Ahrefs Site Audit project.
- [Barkrowler](https://www.centinelanalytica.com/bots/barkrowler) (Babbar): Collects links and page metadata for Babbar.
- [DataForSeoBot](https://www.centinelanalytica.com/bots/dataforseobot) (DataForSEO): Adds links to DataForSEO's backlink database.
- [MJ12bot](https://www.centinelanalytica.com/bots/mj12bot) (Majestic): Maps links between websites for Majestic's link index.
---
Page: https://www.centinelanalytica.com/bots/screaming-frog-seo-spider
# What is Screaming Frog SEO Spider?
Screaming Frog SEO Spider is a website crawler for technical SEO site audits. It runs from the machine it is installed on, so its requests come from the IP address of that machine or network. In the paid version, the person who runs it can change the user agent and tell it to ignore robots.txt.
- Operator: Screaming Frog
- Purpose: Audits a site's technical SEO from the machine that runs it.
- robots.txt token: `Screaming Frog SEO Spider`
- How to verify: Screaming Frog publishes no IP list or reverse DNS domain. The SEO Spider runs from the user's own machine, so the source IP belongs to that machine or its network.
- Operator documentation: [https://www.screamingfrog.co.uk/seo-spider/user-guide/general/](https://www.screamingfrog.co.uk/seo-spider/user-guide/general/)
- Last verified: 2026-10-10
## How often is Screaming Frog SEO Spider impersonated?
No published method checks a request that claims to be Screaming Frog SEO Spider, so we cannot say which claims are false and show no share.
This directory measures requests between 14 July 2026 and 10 October 2026. The [fake crawler report](https://www.centinelanalytica.com/blog/fake-crawler-report) covers its own, earlier window (late June to 21 September 2026), so its figures differ from the ones here. Read it for the method.
## Should you block Screaming Frog SEO Spider?
- Allow it if: you or your agency audit this site with the tool.
- Block it if: audit crawls by other people add load you do not want. A robots.txt rule stops only the users who keep the default settings.
- What blocking changes: Screaming Frog says the SEO Spider cannot crawl a site that disallows it in robots.txt. The paid version has an option to ignore robots.txt, and the operator says its use is the responsibility of the user entirely.
To block it, add this group to your robots.txt. Screaming Frog says the SEO Spider obeys robots.txt by default. When no group names it, it follows the rules for Googlebot, then the rules for all robots. A user of the paid version can set it to ignore robots.txt.
```
User-agent: Screaming Frog SEO Spider
Disallow: /
```
robots.txt only asks. It does nothing against a client that ignores it or only pretends to be Screaming Frog SEO Spider. Stopping those takes verification at your edge.
## Related bots
- [AhrefsBot](https://www.centinelanalytica.com/bots/ahrefsbot) (Ahrefs): Builds the link database behind Ahrefs and the Yep search engine.
- [AhrefsSiteAudit](https://www.centinelanalytica.com/bots/ahrefssiteaudit) (Ahrefs): Crawls a site for an Ahrefs Site Audit project.
- [Barkrowler](https://www.centinelanalytica.com/bots/barkrowler) (Babbar): Collects links and page metadata for Babbar.
- [DataForSeoBot](https://www.centinelanalytica.com/bots/dataforseobot) (DataForSEO): Adds links to DataForSEO's backlink database.
- [DotBot](https://www.centinelanalytica.com/bots/dotbot) (Moz): Gathers data for the Moz Link Index.
- [MJ12bot](https://www.centinelanalytica.com/bots/mj12bot) (Majestic): Maps links between websites for Majestic's link index.
---
Page: https://www.centinelanalytica.com/bots/semrushbot
# What is SemrushBot?
SemrushBot is Semrush's crawler. It discovers and collects new and updated web data for Semrush's tools, including its backlink index. Semrush runs several variants with their own robots.txt tokens, such as SemrushBot-BA and SiteAuditBot.
- Operator: Semrush
- Purpose: Collects web data for Semrush's SEO tools.
- robots.txt token: `SemrushBot`
- How to verify: Semrush publishes no IP list and says it does not use consecutive IP blocks, so it asks you not to block it by IP.
- Operator documentation: [https://www.semrush.com/bot/](https://www.semrush.com/bot/)
- Last verified: 2026-09-30
## How often is SemrushBot impersonated?
Our verification data for SemrushBot was incomplete during the measurement window, so a share would overstate impersonation. We show none until it is complete.
This directory measures requests between 14 July 2026 and 10 October 2026. The [fake crawler report](https://www.centinelanalytica.com/blog/fake-crawler-report) covers its own, earlier window (late June to 21 September 2026), so its figures differ from the ones here. Read it for the method.
## Should you block SemrushBot?
- Allow it if: you or your partners use this tool's backlink or audit data, or the crawl load is not a concern.
- Block it if: the crawl load is noticeable and you have no use for the tool's data. Blocking it does not affect search rankings.
- What blocking changes: Semrush stops collecting data about your site for its tools. Each variant needs its own robots.txt line.
To block it, add this group to your robots.txt. Semrush says SemrushBot obeys robots.txt and honors Crawl-delay up to 10 seconds.
```
User-agent: SemrushBot
Disallow: /
```
robots.txt only asks. It does nothing against a client that ignores it or only pretends to be SemrushBot. Stopping those takes verification at your edge.
## Related bots
- [AhrefsBot](https://www.centinelanalytica.com/bots/ahrefsbot) (Ahrefs): Builds the link database behind Ahrefs and the Yep search engine.
- [AhrefsSiteAudit](https://www.centinelanalytica.com/bots/ahrefssiteaudit) (Ahrefs): Crawls a site for an Ahrefs Site Audit project.
- [Barkrowler](https://www.centinelanalytica.com/bots/barkrowler) (Babbar): Collects links and page metadata for Babbar.
- [DataForSeoBot](https://www.centinelanalytica.com/bots/dataforseobot) (DataForSEO): Adds links to DataForSEO's backlink database.
- [DotBot](https://www.centinelanalytica.com/bots/dotbot) (Moz): Gathers data for the Moz Link Index.
- [MJ12bot](https://www.centinelanalytica.com/bots/mj12bot) (Majestic): Maps links between websites for Majestic's link index.
---
Page: https://www.centinelanalytica.com/bots/seznambot
# What is SeznamBot?
SeznamBot is the crawler behind Seznam.cz's search engine. Seznam also runs FreshBot, a fast crawler for news pages and RSS feeds.
- Operator: Seznam.cz
- Purpose: Crawls pages for Seznam, a Czech search engine.
- robots.txt token: `SeznamBot`
- Example user agent: `Mozilla/5.0 (compatible; SeznamBot/4.0; +https://o-seznam.cz/napoveda/vyhledavani/en/seznambot-crawler/)`
- How to verify: Check that the reverse DNS name ends in seznam.cz and confirm with a forward lookup, or check the source IP against Seznam's published list.
- Reverse DNS: seznam.cz
- IP ranges: [https://search.seznam.cz/ipranges/seznambot.json](https://search.seznam.cz/ipranges/seznambot.json)
- Operator documentation: [https://napoveda.seznam.cz/en/full-text-search/seznambot-crawler/](https://napoveda.seznam.cz/en/full-text-search/seznambot-crawler/)
- Last verified: 2026-09-30
## How often is SeznamBot impersonated?
Our verification data for SeznamBot was incomplete during the measurement window, so a share would overstate impersonation. We show none until it is complete.
This directory measures requests between 14 July 2026 and 10 October 2026. The [fake crawler report](https://www.centinelanalytica.com/blog/fake-crawler-report) covers its own, earlier window (late June to 21 September 2026), so its figures differ from the ones here. Read it for the method.
## Should you block SeznamBot?
- Allow it if: you have Czech-speaking visitors.
- Block it if: you have no audience in the Czech Republic and the crawl adds load.
- What blocking changes: Your pages drop out of Seznam search.
To block it, add this group to your robots.txt. Seznam documents robots.txt control for SeznamBot.
```
User-agent: SeznamBot
Disallow: /
```
robots.txt only asks. It does nothing against a client that ignores it or only pretends to be SeznamBot. Stopping those takes verification at your edge.
## Related bots
- [360Spider](https://www.centinelanalytica.com/bots/360spider) (360 Search): Crawls pages for the web search of 360 Search (so.com).
- [Applebot](https://www.centinelanalytica.com/bots/applebot) (Apple): Crawls pages for Spotlight, Siri and Safari search features.
- [Baiduspider](https://www.centinelanalytica.com/bots/baiduspider) (Baidu): Crawls pages for Baidu search.
- [Bingbot](https://www.centinelanalytica.com/bots/bingbot) (Microsoft): Crawls pages for Bing search.
- [Diffbot](https://www.centinelanalytica.com/bots/diffbot) (Diffbot): Crawls the web for the Diffbot Knowledge Graph and Diffbot's web search services.
- [DuckDuckBot](https://www.centinelanalytica.com/bots/duckduckbot) (DuckDuckGo): Crawls pages for DuckDuckGo search.
---
Page: https://www.centinelanalytica.com/bots/slackbot-linkexpanding
# What is Slackbot-LinkExpanding?
Slackbot-LinkExpanding is the robot Slack sends when a user posts a link into a channel. Slack says it fetches as little of the page as it can, using HTTP Range headers, to read oEmbed, Twitter Card and Open Graph tags. Slack caches the response for around 30 minutes.
- Operator: Slack
- Purpose: Fetches a page's meta tags when someone posts its link in Slack.
- robots.txt token: `Slackbot-LinkExpanding`
- Example user agent: `Slackbot-LinkExpanding 1.0 (+https://api.slack.com/robots)`
- How to verify: Slack publishes no IP list or reverse DNS domain for its robots.
- Operator documentation: [https://api.slack.com/robots](https://api.slack.com/robots)
- Last verified: 2026-10-10
## How often is Slackbot-LinkExpanding impersonated?
No published method checks a request that claims to be Slackbot-LinkExpanding, so we cannot say which claims are false and show no share.
This directory measures requests between 14 July 2026 and 10 October 2026. The [fake crawler report](https://www.centinelanalytica.com/blog/fake-crawler-report) covers its own, earlier window (late June to 21 September 2026), so its figures differ from the ones here. Read it for the method.
## Should you block Slackbot-LinkExpanding?
- Allow it if: people share your pages in Slack.
- Block it if: you do not want preview cards in Slack. Ask Slack to add your site to its blocklist, because a robots.txt rule has no effect.
- What blocking changes: A robots.txt rule does not stop it. Slack says to contact it if you want your site blocked from Slack embeds.
To block it, add this group to your robots.txt. Slack says it does not currently honor robots.txt files, because the robot does not follow links and acts on behalf of a person.
```
User-agent: Slackbot-LinkExpanding
Disallow: /
```
robots.txt only asks. It does nothing against a client that ignores it or only pretends to be Slackbot-LinkExpanding. Stopping those takes verification at your edge.
## Related bots
- [Discordbot](https://www.centinelanalytica.com/bots/discordbot) (Discord): Visits a link shared in a Discord message to build its preview.
- [facebookexternalhit](https://www.centinelanalytica.com/bots/facebookexternalhit) (Meta): Builds link previews when content is shared on Meta's apps.
- [Pinterestbot](https://www.centinelanalytica.com/bots/pinterestbot) (Pinterest): Indexes pages behind Pins and keeps Pin details up to date.
- [Twitterbot](https://www.centinelanalytica.com/bots/twitterbot) (X): Fetches card markup when a link is shared on X.
---
Page: https://www.centinelanalytica.com/bots/storebot-google
# What is Storebot-Google?
Storebot-Google is Google's shopping crawler. Google lists it among its common crawlers, and crawling preferences addressed to it affect Google Shopping.
- Operator: [Google](https://www.centinelanalytica.com/bots/operator/google)
- Purpose: Crawls product and store pages for Google Shopping.
- robots.txt token: `Storebot-Google`
- Example user agent: `Mozilla/5.0 (X11; Linux x86_64; Storebot-Google/1.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Safari/537.36`
- How to verify: Run a reverse DNS lookup on the source IP and check that the name ends in googlebot.com (Google uses crawl-….googlebot.com and geo-crawl-….geo.googlebot.com), then run a forward lookup on that name and confirm it returns the same IP. Or check the IP against Google's common crawler ranges. A googleusercontent.com name does not count: any Google Cloud virtual machine can have one.
- Reverse DNS: googlebot.com
- IP ranges: [https://developers.google.com/static/crawling/ipranges/common-crawlers.json](https://developers.google.com/static/crawling/ipranges/common-crawlers.json)
- Operator documentation: [https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers)
- Last verified: 2026-09-30
## How often is Storebot-Google impersonated?
Our verification data for Storebot-Google was incomplete during the measurement window, so a share would overstate impersonation. We show none until it is complete.
This directory measures requests between 14 July 2026 and 10 October 2026. The [fake crawler report](https://www.centinelanalytica.com/blog/fake-crawler-report) covers its own, earlier window (late June to 21 September 2026), so its figures differ from the ones here. Read it for the method.
## Should you block Storebot-Google?
- Allow it if: you list products on Google Shopping.
- Block it if: you sell nothing through Google and the crawl adds load.
- What blocking changes: Google Shopping loses its view of your product pages, which can affect product listings.
To block it, add this group to your robots.txt. Google says its common crawlers always obey robots.txt when crawling automatically.
```
User-agent: Storebot-Google
Disallow: /
```
robots.txt only asks. It does nothing against a client that ignores it or only pretends to be Storebot-Google. Stopping those takes verification at your edge.
## Related bots
- [AdsBot-Google](https://www.centinelanalytica.com/bots/adsbot-google) (Google): Checks the quality of ad landing pages for Google Ads.
- [Google-CloudVertexBot](https://www.centinelanalytica.com/bots/google-cloudvertexbot) (Google): Crawls sites whose owners asked for it when building Vertex AI Agents.
- [Google-Extended](https://www.centinelanalytica.com/bots/google-extended) (Google): A robots.txt token that controls Gemini training and grounding use.
- [Google-InspectionTool](https://www.centinelanalytica.com/bots/google-inspectiontool) (Google): Fetches pages for Search Console URL inspection and the Rich Results Test.
- [Googlebot-Image](https://www.centinelanalytica.com/bots/googlebot-image) (Google): Crawls images for Google Images and image features in Search.
- [Googlebot-News](https://www.centinelanalytica.com/bots/googlebot-news) (Google): Controls whether your pages appear in Google News.
---
Page: https://www.centinelanalytica.com/bots/twitterbot
# What is Twitterbot?
Twitterbot is the crawler X uses to read card markup when a link is shared, so the post can show a title, description and image. X caches the result for 7 days.
- Operator: X
- Purpose: Fetches card markup when a link is shared on X.
- robots.txt token: `Twitterbot`
- How to verify: X publishes no IP list or reverse DNS domain for Twitterbot.
- Operator documentation: [https://developer.x.com/en/docs/x-for-websites/cards/guides/getting-started](https://developer.x.com/en/docs/x-for-websites/cards/guides/getting-started)
- Last verified: 2026-09-30
## How often is Twitterbot impersonated?
Other apps also put "Twitterbot" in their user agent, so a request carrying the name is not always a claim to be X's bot. We show no share.
This directory measures requests between 14 July 2026 and 10 October 2026. The [fake crawler report](https://www.centinelanalytica.com/blog/fake-crawler-report) covers its own, earlier window (late June to 21 September 2026), so its figures differ from the ones here. Read it for the method.
## Should you block Twitterbot?
- Allow it if: people share your pages on X.
- Block it if: you do not want preview cards on X.
- What blocking changes: Links to your pages shared on X lose their preview card.
To block it, add this group to your robots.txt. X says Twitterbot respects robots.txt. If a page or its image is blocked, no card or thumbnail is shown.
```
User-agent: Twitterbot
Disallow: /
```
robots.txt only asks. It does nothing against a client that ignores it or only pretends to be Twitterbot. Stopping those takes verification at your edge.
## Related bots
- [Discordbot](https://www.centinelanalytica.com/bots/discordbot) (Discord): Visits a link shared in a Discord message to build its preview.
- [facebookexternalhit](https://www.centinelanalytica.com/bots/facebookexternalhit) (Meta): Builds link previews when content is shared on Meta's apps.
- [Pinterestbot](https://www.centinelanalytica.com/bots/pinterestbot) (Pinterest): Indexes pages behind Pins and keeps Pin details up to date.
- [Slackbot-LinkExpanding](https://www.centinelanalytica.com/bots/slackbot-linkexpanding) (Slack): Fetches a page's meta tags when someone posts its link in Slack.
---
Page: https://www.centinelanalytica.com/bots/yandexbot
# What is YandexBot?
YandexBot is the main indexing robot of Yandex search. Yandex runs many other robots; some of them ignore the general User-agent: * rules because they do not index pages.
- Operator: Yandex
- Purpose: Yandex's main search indexing crawler.
- robots.txt token: `YandexBot`
- Example user agent: `Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)`
- How to verify: Run a reverse DNS lookup and check that the name ends in yandex.ru, yandex.net or yandex.com, then confirm with a forward lookup. Yandex does not publish an IP list.
- Reverse DNS: yandex.ru, yandex.net, yandex.com
- Operator documentation: [https://yandex.com/support/webmaster/en/robot-workings/check-yandex-robots](https://yandex.com/support/webmaster/en/robot-workings/check-yandex-robots)
- Last verified: 2026-09-30
## How often is YandexBot impersonated?
For the median organisation, under 1% of requests claiming to be YandexBot failed verification.
Measured between 14 July 2026 and 10 October 2026 across organisations protected by Centinel. We only publish a share when at least five organisations each saw enough of these requests. Pooled across all of their requests, the share was under 1%: a few large organisations carry most of the traffic, so the median organisation is the better guide to what yours will see.
This directory measures requests between 14 July 2026 and 10 October 2026. The [fake crawler report](https://www.centinelanalytica.com/blog/fake-crawler-report) covers its own, earlier window (late June to 21 September 2026), so its figures differ from the ones here. Read it for the method.
## Should you block YandexBot?
- Allow it if: you have visitors who search with Yandex.
- Block it if: you have no audience on Yandex and the crawl adds load.
- What blocking changes: Your pages drop out of Yandex search.
To block it, add this group to your robots.txt. Yandex says YandexBot follows the general rules in robots.txt.
```
User-agent: YandexBot
Disallow: /
```
robots.txt only asks. It does nothing against a client that ignores it or only pretends to be YandexBot. Stopping those takes verification at your edge.
## Related bots
- [360Spider](https://www.centinelanalytica.com/bots/360spider) (360 Search): Crawls pages for the web search of 360 Search (so.com).
- [Applebot](https://www.centinelanalytica.com/bots/applebot) (Apple): Crawls pages for Spotlight, Siri and Safari search features.
- [Baiduspider](https://www.centinelanalytica.com/bots/baiduspider) (Baidu): Crawls pages for Baidu search.
- [Bingbot](https://www.centinelanalytica.com/bots/bingbot) (Microsoft): Crawls pages for Bing search.
- [Diffbot](https://www.centinelanalytica.com/bots/diffbot) (Diffbot): Crawls the web for the Diffbot Knowledge Graph and Diffbot's web search services.
- [DuckDuckBot](https://www.centinelanalytica.com/bots/duckduckbot) (DuckDuckGo): Crawls pages for DuckDuckGo search.
---
Page: https://www.centinelanalytica.com/bots/yeti
# What is Yeti?
Yeti is the crawler Naver uses for Naver search. Naver documents a reverse DNS check so you can confirm a request is really Yeti.
- Operator: Naver
- Purpose: Crawls pages for Naver search in South Korea.
- robots.txt token: `Yeti`
- Example user agent: `Mozilla/5.0 (compatible; Yeti/1.1; +https://naver.me/spd)`
- How to verify: Run a reverse DNS lookup and check that the name ends in naver.com, then confirm with a forward lookup.
- Reverse DNS: naver.com
- Operator documentation: [https://searchadvisor.naver.com/guide/seo-basic-firewall](https://searchadvisor.naver.com/guide/seo-basic-firewall)
- Last verified: 2026-09-30
## How often is Yeti impersonated?
For the median organisation, under 1% of requests claiming to be Yeti failed verification.
Measured between 14 July 2026 and 10 October 2026 across organisations protected by Centinel. We only publish a share when at least five organisations each saw enough of these requests. Pooled across all of their requests, the share was under 1%: a few large organisations carry most of the traffic, so the median organisation is the better guide to what yours will see.
This directory measures requests between 14 July 2026 and 10 October 2026. The [fake crawler report](https://www.centinelanalytica.com/blog/fake-crawler-report) covers its own, earlier window (late June to 21 September 2026), so its figures differ from the ones here. Read it for the method.
## Should you block Yeti?
- Allow it if: you have Korean-speaking visitors.
- Block it if: you have no audience in South Korea and the crawl adds load.
- What blocking changes: Your pages drop out of Naver search.
To block it, add this group to your robots.txt. Naver documents robots.txt control for Yeti.
```
User-agent: Yeti
Disallow: /
```
robots.txt only asks. It does nothing against a client that ignores it or only pretends to be Yeti. Stopping those takes verification at your edge.
## Related bots
- [360Spider](https://www.centinelanalytica.com/bots/360spider) (360 Search): Crawls pages for the web search of 360 Search (so.com).
- [Applebot](https://www.centinelanalytica.com/bots/applebot) (Apple): Crawls pages for Spotlight, Siri and Safari search features.
- [Baiduspider](https://www.centinelanalytica.com/bots/baiduspider) (Baidu): Crawls pages for Baidu search.
- [Bingbot](https://www.centinelanalytica.com/bots/bingbot) (Microsoft): Crawls pages for Bing search.
- [Diffbot](https://www.centinelanalytica.com/bots/diffbot) (Diffbot): Crawls the web for the Diffbot Knowledge Graph and Diffbot's web search services.
- [DuckDuckBot](https://www.centinelanalytica.com/bots/duckduckbot) (DuckDuckGo): Crawls pages for DuckDuckGo search.
---
Page: https://www.centinelanalytica.com/bots/youbot
# What is YouBot?
YouBot is the web crawler that powers the You.com search engine. You.com says it automatically discovers and indexes web pages to provide search results for You.com users.
- Operator: You.com
- Purpose: Crawls and indexes pages for the You.com search engine.
- robots.txt token: `YouBot`
- Example user agent: `Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; YouBot/1.0; +https://docs.you.com/youbot; env:prod) Chrome/X.X.X.X Safari/537.36`
- How to verify: You.com documents three checks: a Web Bot Auth signature, a reverse DNS name of the form youbot-….search.you.com, and a source IP in the range 68.67.112.0/24. It recommends the signature.
- Reverse DNS: search.you.com
- Operator documentation: [https://you.com/docs/youbot](https://you.com/docs/youbot)
- Last verified: 2026-10-10
## How often is YouBot impersonated?
Our verification data for YouBot was incomplete during the measurement window, so a share would overstate impersonation. We show none until it is complete.
This directory measures requests between 14 July 2026 and 10 October 2026. The [fake crawler report](https://www.centinelanalytica.com/blog/fake-crawler-report) covers its own, earlier window (late June to 21 September 2026), so its figures differ from the ones here. Read it for the method.
## Should you block YouBot?
- Allow it if: you want your pages in You.com search results.
- Block it if: you do not want your pages in the You.com index.
- What blocking changes: You.com says content that is public and not blocked by robots.txt may be crawled and indexed. A Disallow rule keeps YouBot from crawling your pages.
To block it, add this group to your robots.txt. You.com says YouBot fully respects robots.txt directives, including user-agent specific rules and crawl-delay settings. YouBot caches robots.txt for 30 minutes.
```
User-agent: YouBot
Disallow: /
```
robots.txt only asks. It does nothing against a client that ignores it or only pretends to be YouBot. Stopping those takes verification at your edge.
## Related bots
- [360Spider](https://www.centinelanalytica.com/bots/360spider) (360 Search): Crawls pages for the web search of 360 Search (so.com).
- [Applebot](https://www.centinelanalytica.com/bots/applebot) (Apple): Crawls pages for Spotlight, Siri and Safari search features.
- [Baiduspider](https://www.centinelanalytica.com/bots/baiduspider) (Baidu): Crawls pages for Baidu search.
- [Bingbot](https://www.centinelanalytica.com/bots/bingbot) (Microsoft): Crawls pages for Bing search.
- [Diffbot](https://www.centinelanalytica.com/bots/diffbot) (Diffbot): Crawls the web for the Diffbot Knowledge Graph and Diffbot's web search services.
- [DuckDuckBot](https://www.centinelanalytica.com/bots/duckduckbot) (DuckDuckGo): Crawls pages for DuckDuckGo search.
---
Page: https://www.centinelanalytica.com/blog/bot-mitigation-for-publishers-manage-unknown-crawlers-at-edge
# Bot Mitigation for Publishers: Manage Unknown Crawlers at Edge
Publisher bot mitigation cannot treat unknown crawlers like known threats. Learn how to govern gray-zone traffic at the edge with clear rules and expiry dates.
By Centinel Analytica. Published 2026-10-16.
A high-traffic news site publishes an exclusive report, and within forty seconds, edge logs show forty thousand requests hitting article endpoints from an unfamiliar user agent. Origin CPU climbs toward ninety percent. The network engineering team reaches for a blanket IP block. The audience development director worries the traffic belongs to an unannounced syndication partner or a new search engine pilot.
This standoff happens weekly in digital publishing. Content-heavy sites get huge volumes of traffic that sit between clearly identified partners and confirmed attacks. Most security advice covers known entities: you verify Googlebot through reverse DNS, and you drop obvious vulnerability scanners running mass exploit payloads. The real operational drain sits in the gray zone, where requests fail standard verification or match no editorial policy.
Good bot mitigation for publishers means handling that ambiguity at the CDN layer. You don't need certainty before you act. You do need governance. Every action at the edge needs a named owner, an evidence threshold, an expiry date, an audit trail, and a rollback path.
## What counts as an unknown crawler (and why it is not the same as a fake one)
An unknown crawler is not an impersonator. A fake crawler lies about its identity, typically by using a legitimate user agent string from Google, Bing, or Apple while coming from an unrelated hosting network. Centinel Analytica research found that on a typical news site, 84% of requests claiming to be GPTBot were fake. Catching a fake is easy: run reverse DNS, match the domain suffix against published IP lists, and drop the connection when the proof fails. You can see how this works in practice with [fake Googlebot detection](https://www.centinelanalytica.com/blog/fake-googlebot).
An unknown crawler plays by different rules. It might declare an honest but unfamiliar name in its header, such as `ResearchBot/1.0` or a generic `python-requests/2.31.0` signature. It might come from an Amazon Web Services, Hetzner, or DigitalOcean subnet with no reverse DNS record. It breaks no WAF rules, trips no credential-stuffing heuristics, and requests public articles one after another.
The problem for publishers is that intent stays opaque. The visitor could be an academic project scraping archives for linguistic analysis, an undocumented aggregator delivering regional traffic, a commercial AI lab harvesting training data without attribution, or a competitor watching paywalled pricing changes. Treat every unverified crawler as hostile and you break partner relationships and search discovery. Treat every one as benign and you get unmetered scraping that inflates cloud bills and devalues original journalism.
## Triage evidence: what to check before any edge action
Never touch an edge routing rule based on a raw user agent string alone. Header strings are arbitrary text, and any client script can change one in a single line of code. Before you take any edge action, inspect four pieces of network evidence to set a baseline.
First, check the autonomous system number (ASN) and network category. Traffic from consumer ISPs like Comcast or Deutsche Telekom points to residential connections. Traffic from OVH, Leaseweb, or AWS points to automated data center infrastructure. Commercial scrapers often hide behind residential proxy pools to get around simple IP reputation lists.
Second, inspect the TLS and HTTP fingerprints. Standard browsers negotiate connections with specific cipher suites, extensions, and HTTP/2 settings frames. Curl, Python Requests, and Go each have their own default handshake signatures. If an unknown client presents a desktop Chrome user agent but negotiates TLS with default Python parameters, it's automation, whatever the user agent says.
Third, map the request topology. Humans don't request thirty articles per second in alphabetical order across three thousand URLs. Automated clients read pagination lists, extract sitemaps, and poll RSS feeds on precise intervals.
Fourth, review origin cache impact. Check whether the traffic produces edge cache hits or origin cache misses caused by appended cache-busting query parameters. If an unverified bot systematically bypasses your cache layer, it's an immediate threat to availability. Before changing anything, log the incident context using structured protocols like those in [scraper incident response](https://www.centinelanalytica.com/blog/scraper-incident-response-preserve-evidence-before-blocking).
## Policy owners and escalation: who can decide what
Disputes over bot traffic almost always come from conflicting incentives. The infrastructure engineer cares about server stability and CDN egress costs. The audience development editor cares about referral traffic, syndication, and search visibility. The ad operations manager cares about programmatic viewability and analytics contamination. When an unknown crawler arrives, these priorities collide.
Publishers need a clear decision framework before traffic spikes occur:
Security and DevOps teams own technical risk thresholds. If an unverified crawler pushes origin CPU past seventy percent or error rates above two percent, technical teams can act immediately without editorial sign-off.
Audience and syndication leads own identity verification. When an unknown bot matches traffic from commercial partners, editorial leads check contracts and secure static IP ranges or authentication tokens. They have forty-eight hours to validate a crawler before edge controls tighten.
Content operations and legal teams own monetization policy. They decide whether an unverified commercial AI crawler gets full-text editorial copy for free or hits a paywall. Set this ownership down using a template like an [AI crawler policy for publishers](https://www.centinelanalytica.com/blog/ai-crawler-policy-for-publishers-a-newsroom-access-template).
When a gray-zone crawler shows up, the triage engineer tags the audience lead. If no owner claims the source within twelve hours, the edge policy moves from monitoring to active rate containment.
## Temporary edge decisions: challenge, throttle, serve-limited, block, with expiry dates
Permanent edge blocks are where publisher bot policies go to rot. An engineer drops an unverified IP range into a WAF block list on a Friday afternoon, forgets about it, and nine months later finds that the rule broke a new syndication feed from an international partner. Edge actions against gray-zone crawlers should be temporary by default.
Apply four progressive interventions based on how strong the evidence is:
1. JavaScript or cryptographic challenge. Present a client-side challenge to see whether the visitor runs a real interactive browser. Simple scripts and headless scrapers fail or disconnect because of the compute overhead. Real readers pass through.
2. Rate throttling. Restrict the client to a sustainable rate, such as one request every two seconds. Legitimate aggregators can still fetch breaking headlines, but large-scale archive extraction becomes too slow to be worth it.
3. Serve-limited responses. Route the unknown crawler to stale edge cache copies, stripped HTML without heavy assets, or summary metadata instead of origin-rendered full text. This protects origin capacity and keeps the content indexable.
4. Temporary edge block. Return a standard HTTP 403 or 429 status code directly from edge nodes, so the request never reaches your origin server.
Every rule you create needs an explicit time-to-live: twenty-four, forty-eight, or seventy-two hours. Centinel Analytica applies allow, challenge, or block policy through your existing CDN or edge, so teams can manage this lifecycle cleanly. If an edge rule has no expiration date, your deployment pipeline should reject it.
## Audit trail: what to log so decisions can be defended later
When an executive asks why a content partner got HTTP 403 errors over the weekend, an undocumented WAF rule is a liability. Every edge mitigation decision needs an immutable audit record tied to a specific rule identifier.
Capture eight data points for every gray-zone edge intervention:
- Unique rule identifier and operator handle
- Exact deployment timestamp and pre-set expiration timestamp
- Matched indicators: client IP, ASN, TLS signature, and user agent
- Observed request volume and target URI patterns
- Specific mitigation applied: challenge, throttle, serve-limited, or block
- Origin impact metric: cache miss ratio, CPU delta, or egress cost spike
- Rationale: short justification citing the specific triage evidence
- Notification status: confirmation that audience or engineering leads were tagged
Send these records straight to your central SIEM or structured logging platform. When engineering reviews CDN performance, the audit trail turns emotional debates into objective evaluations of network evidence. If an unknown entity files a support ticket, the operations team can pull the exact session data that triggered the edge policy in seconds.
## Rollback and review: expiring rules, handling complaints, promoting to permanent policy
A governed mitigation policy has to handle rule lifecycles automatically. When a temporary edge rule hits its expiration timestamp, the system removes the restriction by default and keeps telemetry logging running.
Review expired rules every Tuesday and Thursday. The team looks at what happened during the mitigation window, and the outcome usually falls into one of three scenarios.
The unknown crawler stopped sending traffic. The temporary throttle or challenge raised compute costs enough that the operator abandoned the target. The rule expires and nothing more is needed.
The operator reached out through designated contact channels. High-value search engines and legitimate research organizations keep contact emails or check your `security.txt` and `robots.txt` directives. When a legitimate operator makes contact, your team moves them to an allowlist with verified mutual TLS, API keys, or designated IP ranges.
The client escalated. If the unknown crawler answered an edge block by rotating to residential proxies, altering user agent headers, or increasing hit volume, the evidence of malicious intent is conclusive. The rule graduates from a temporary gray-zone rule to a permanent infrastructure ban.
If editorial complaints arrive before a rule expires, rollback must take under two minutes. Platform engineers can deactivate a targeted edge rule across all edge locations without redeploying the entire CDN configuration.
## Where this fits with your existing crawler policy, incident response and proof-of-concept work
Edge bot mitigation doesn't replace your security perimeter. It fills the gap between static rule configurations and ongoing threat discovery. Your web application firewall keeps blocking SQL injections and known exploits, and your `robots.txt` file keeps defining baseline crawling terms for compliant platforms.
Many media publishers hesitate to update bot controls because they fear vendor lock-in or routing disruptions. Replacing your CDN is off the table for publishers running complex caching topologies. The answer is modular defense. Centinel Analytica was founded by Frederick Jahn and Simeon Räthel, who lead a Berlin team of former bot builders and reverse engineers.
When you evaluate edge defense tooling, test against real scraping stacks, not basic curl commands. Modern scrapers use automation frameworks running patched browser runtimes that simulate realistic human interaction. Run a two-week proof-of-concept against your live staging or passive production traffic. Compare how many gray-zone crawlers your current rules misclassify against what low-level TLS and behavioral inspection actually reveals.
## Conclusion
Publishers can't run bot mitigation on guesswork or permanent, unmonitored IP bans. When unknown crawlers hit your content at scale, edge policies need fast defensive action and clear governance: explicit thresholds, defined owners, strict expiration dates, and auditable evidence logs.
Gray-zone scrapers degrade origin availability and dilute editorial value, so stop tolerating them. Centinel Analytica plugs into your existing Cloudflare, Fastly, Akamai, or CloudFront edge to identify automated traffic and enforce challenge, allow, or block policies without changing your core routing stack. Contact the Centinel Analytica team to benchmark your edge traffic and take back governance over your publication.
## Frequently asked questions
### What is the difference between an unknown crawler and a fake crawler?
A fake crawler actively impersonates a known search engine or AI bot by forging its user agent header, which fails reverse DNS or cryptographic verification. An unknown crawler uses an unverified or generic identity, operates from commercial hosting providers, and lacks public documentation or editorial agreements.
### How does edge bot mitigation prevent false positives for news publishers?
By using progressive edge responses like JavaScript challenges or rate throttling rather than permanent IP blocks. Rules include mandatory expiration timers, giving operations and audience teams time to verify whether the source is a legitimate syndication partner before making any permanent routing decision.
### Can I run bot mitigation without replacing my current CDN?
Yes. Centinel Analytica works alongside Cloudflare, CloudFront, Akamai, and Fastly, keeping your existing CDN and WAF layers in place while inspecting traffic and applying allow, challenge, or block policies directly at the edge.
### Why does robots.txt fail to stop unverified scrapers?
The robots.txt file is an advisory protocol, not a technical control. Compliant search engines honor it, but unauthorized scrapers, proprietary AI scrapers, and competitors routinely ignore robots exclusion directives, requiring edge-level enforcement to protect origin resources.
---
Page: https://www.centinelanalytica.com/blog/form-bot-protection-reduce-fake-leads-without-blocking-buyers
# Form Bot Protection: Reduce Fake Leads Without Blocking Buyers
Stop fake form submissions without breaking buyer conversions. Learn how server-side form bot protection blocks bots while preserving sales leads.
By Centinel Analytica. Published 2026-10-16.
Every sales team knows the Monday morning sinking feeling: thirty new demo requests arrived over the weekend, and every single one is garbage. Some contain random strings like "asdf asdf", others pitch search engine optimization services, and a few contain stolen corporate domain names that bounce the moment a sales representative sends an introductory email.
The reaction is predictable. A developer drops a front-end CAPTCHA widget onto the contact form, checks the box in production, and calls the problem solved. Within forty-eight hours, the sales team notices that lead volume dropped by twenty percent, yet the spam never completely stopped.
Client-side widgets don't solve automated form abuse. Lead-form exploitation is an architectural issue, not a missing puzzle piece. Effective form bot protection needs server-side validation, behavioral evidence, and velocity controls that stop automated scripts while keeping the path clear for legitimate buyers.
## Fake leads are two different problems: spam content and automated submissions
Engineering teams usually treat lead form pollution as a single nuisance. It isn't. Fake leads come from two distinct mechanisms, and each needs its own defense.
The first category is spam content submitted through real browser sessions. That means low-wage manual data entry workers, affiliate marketers, and automated browser agents carrying text payloads built to sell backlinks or services. The connection looks identical to a human visitor because a human or a full browser engine sits behind the keyboard. The problem is the payload, not the network protocol. You can't solve content spam by analyzing TCP handshakes. You handle it with schema validation, disposable domain blocklists, and post-submission text classification.
The second category is automated form abuse. Attackers use headless runtimes, command-line scripts, or distributed botnets to blast thousands of requests directly at your form endpoint. They rarely want to sell you marketing services. They use your contact forms for credit card testing, mass phishing relays through your transactional email provider, or endpoint discovery. In a card-testing attack, a bot injects stolen card numbers into a checkout or donation form to verify which credentials still work (Stripe, 2025).
Try to stop automated scripts with text analysis tools like Akismet, and your server still burns memory on hundreds of bogus requests per minute. Try to stop content spam with network rate limits, and legitimate prospects on shared corporate VPNs get blocked. Identify the submission engine before you inspect the text.
## Validate on the server: token checks, field rules and what a client-side widget cannot do
Any security mechanism that runs only inside the visitor's browser is under the attacker's control. Front-end validation scripts, client-side token generators, and simple honeypot fields give site owners a false sense of security while barely slowing down basic automation.
Think about how an attacker targets a form. They don't visit your landing page, render the CSS, and gently click each input field. They inspect your network traffic, find the destination POST route (such as `/api/v1/lead-capture` or `/wp-admin/admin-post.php`), and send raw HTTP payloads straight to that URL. If your application server doesn't verify that a submission carries a cryptographically signed token issued by your own backend within the last five minutes, the attacker bypasses your entire front-end interface in a single line of Python.
Honeypot fields show this weakness clearly. For years, developers hid an input field with CSS (`display: none;` or `opacity: 0;`), assuming only blind bots would fill it. Modern scrapers and browser agents don't fall for it. If a field has zero height or invisible opacity, an automated script ignores it just as a human does.
Move every validation check to the server. Your backend must enforce strict field length limits, reject suspicious character encodings, and verify cryptographic tokens against a server-side secret. When a request arrives without a valid token signed by your infrastructure, drop the connection before it triggers internal webhooks or CRM sync routines.
## Request and session evidence: what a legitimate submission should carry
A genuine buyer leaves a distinct trail as they move through a website. They load static assets, negotiate TLS parameters that match their reported user agent, and spend measurable time reading copy before clicking submit. Automated scripts almost always leave gaps in that chain.
Start with timing. A real human filling out a five-field form needs between eight and forty-five seconds to type a name, pick a dropdown option, and hit submit. When an HTTP POST request lands on your server six hundred milliseconds after the initial page fetch, that submission is synthetic. Record the gap between the page delivery timestamp and the submission timestamp, and your application can flag or discard impossibly fast submissions.
Next, look at network signatures. Automated tools built on default HTTP libraries like Python Requests, Go HTTP, or cURL emit distinctive TLS client fingerprints (JA4) and HTTP/2 settings frames that clash with real desktop browsers. A request that claims to be Safari on macOS but presents the TLS cipher suite order of an unconfigured Node.js script is fraudulent.
Session continuity is the final layer. A legitimate visitor requests CSS files, downloads web fonts, and keeps persistent cookies across pages. Automated injectors skip asset loading entirely to save bandwidth. Spotting these low-level traffic anomalies takes deep edge inspection.
Centinel Analytica, founded in Berlin by former bot builders Frederick Jahn and Simeon Räthel, focuses on exactly this problem. The platform evaluates traffic signatures and browser anomalies at the edge alongside providers like Cloudflare, Fastly, CloudFront, and Akamai, separating automated tools from real buyers before they reach your form handlers.
## Replay and velocity controls: single-use tokens, per-identity limits and WordPress caching pitfalls
Even if your form requires a cryptographic token, an attacker can record one valid token from a real session and replay it thousands of times across distributed proxy nodes. Without strict replay defenses, token verification collapses under distributed attacks.
To stop replay attacks, use single-use tokens stored in an ephemeral, in-memory store such as Redis. When your server renders the form, it generates a cryptographically random token tied to that visitor session with a short time-to-live (typically ten to fifteen minutes). The instant a submission arrives, the server checks the token, marks it as consumed in the cache, and invalidates it. Any later request presenting that same token must fail immediately.
This pattern exposes a big pitfall on content management systems like WordPress. To maximize performance, plugins like WP Rocket or edge caching at the CDN level cache the entire rendered HTML of public pages. If your WordPress site caches HTML containing a static CSRF token or WordPress nonce, every visitor gets the exact same token for hours or days. The nonce stops being a number used once. Bots can harvest that cached nonce one time and fire endless submissions at your admin endpoints.
Avoid this by decoupling token generation from the static HTML cache. Load the form structure from cache, but inject the verification token dynamically with an asynchronous background fetch to an uncached endpoint, or generate it at the edge with lightweight serverless functions.
Pair single-use tokens with multi-dimensional rate limiting. Don't limit only by client IP address, because residential proxy pools let attackers cycle through millions of clean IP addresses. Enforce velocity limits across IP subnets, target corporate domains, and session identifiers instead.
## Accessibility and buyer friction: challenges, alternatives and fail-open decisions
A security control that solves your engineering problem while ruining buyer conversion is an organizational failure. When an enterprise executive about to spend fifty thousand dollars hits an unreadable grid of distorted crosswalks or bicycles, they don't persevere. They close the tab and visit your competitor.
Traditional visual CAPTCHAs create severe accessibility hurdles, and failing to provide alternative options across different sensory modes can lead to non-compliance with Web Content Accessibility Guidelines (WCAG 2.1 AA). Puzzle widgets are also miserable on mobile, where prospects fill out forms over spotty cellular connections.
Modern form bot protection should default to invisible behavioral scoring. Instead of making visitors perform manual tasks, run passive proof-of-work challenges in background threads or inspect network-level metadata at the edge. The user experience should match an unprotected form: fill in the fields, click submit, get confirmation.
You also need explicit failure policies for edge verification services. If your verification provider has an outage or elevated latency, does your form fail closed or fail open? In high-security banking portals, failing closed is standard. In high-intent business-to-business lead capture, failing closed burns inbound sales pipeline. A resilient architecture fails open, routing suspicious submissions during provider timeouts to a quarantined CRM review queue instead of showing a raw error page to a prospective client.
## Measure the result and review false positives (linking the CAPTCHA-loop and false-positive guides)
Zero false positives doesn't exist in bot mitigation. Any vendor or security engineer promising one hundred percent accuracy without ever interrupting a legitimate customer is misrepresenting how edge heuristics work. The goal of form bot protection isn't perfection. It's cutting abusive noise by orders of magnitude while keeping clear paths open for real buyers.
To measure protection health, watch two metrics together: invalid submission volume and overall form completion rate. If bot submissions drop by ninety-nine percent but your total qualified lead count falls by fifteen percent, your detection thresholds are overtuned. Legitimate buyers are probably getting caught in aggressive challenge loops.
Read our guide on [bot false positive testing](https://www.centinelanalytica.com/blog/bot-false-positive-testing-verify-visitors-before-blocking) to learn how to test your rules against clean reference traffic before turning challenges into hard blocks. When configuring CDN-level rules, read [Cloudflare Bot Fight Mode vs. Bot Management](https://www.centinelanalytica.com/blog/cloudflare-bot-fight-mode-vs-bot-management) to see why basic heuristic firewalls often trap corporate visitors in endless verification cycles.
Keep a quarantine workflow you can actually review. When a submission scores on the boundary between bot and human, don't drop the payload into a void. Store it in an isolated database collection with its network evidence, such as user agent, header ordering, and TLS fingerprint. Review that queue weekly to find corporate networks, privacy-focused browsers, or enterprise security proxies that tripped your alarms by mistake.
## Next step: protect your forms with Centinel Analytica
Securing web forms takes continuous visibility into who is hitting your endpoints. If you manage forms across multiple marketing subdomains, custom web applications, or content platforms, manual script inspection only gets you so far.
Take stock of your defenses today. Run a simple cURL test against your form handler to see if it accepts submissions with no browser context. Check your caching configuration to make sure nonces and session secrets aren't leaking from static edge caches. Then inspect your edge traffic logs to find how many automated requests bypass your presentation layer entirely.
Centinel Analytica helps engineering and security teams find these blind spots. It runs alongside your existing CDN or WAF (including Cloudflare, AWS CloudFront, Fastly, and Akamai), inspects incoming requests at the edge, classifies automation with precision, and enforces custom allow, challenge, or block policies without forcing you to replace your infrastructure.
Deploy Centinel Analytica to inspect your actual form traffic, uncover automated bot activity, and protect your sales pipeline.
## Conclusion
Form bot protection isn't about making your website harder to use. It's about demanding verifiable cryptographic and behavioral proof before your backend processes an untrusted request. Front-end visual puzzles push genuine buyers away while automated scripts quietly bypass them in the background. Move your validation to the server, bind every submission to real-time session telemetry, and keep your revenue pipeline wide open for real humans.
## Frequently asked questions
### Why do traditional CAPTCHAs fail to stop form spam?
Traditional visual CAPTCHAs only protect the browser interface, not the underlying server endpoint. Attackers easily bypass the interface by submitting HTTP requests directly to the form handler. Furthermore, automated scripts solve modern puzzles using inexpensive AI vision models or automated solving services, while legitimate human buyers abandon difficult visual tests.
### Can scrapers and form bots bypass honeypot fields?
Yes. Modern automated bots and browser tools like Playwright and Puppeteer read computed DOM styles. If a honeypot input field uses CSS rules like display: none or visibility: hidden, the automation identifies the field as non-interactive and skips it, rendering basic honeypots ineffective against current scraping frameworks.
### How does edge caching break form security tokens?
When a CDN or caching plugin caches a form page HTML, it caches the embedded anti-CSRF token or nonce for every visitor. Attackers harvest this single, public token from the cached response and reuse it across thousands of automated form submissions until the cache expires.
### What is the best way to handle borderline suspicious submissions?
Instead of immediately blocking borderline requests or presenting invasive puzzles, use a fail-open quarantine pattern. Accept the form submission, but tag it as unverified in your database or CRM. Send an automated email verification link to the submitter before dispatching the lead to your sales team.
---
Page: https://www.centinelanalytica.com/blog/camoufox-paywall-audit-test-publisher-access-controls
# Camoufox Paywall Audit: Test Publisher Access Controls
Conduct a Camoufox paywall audit on your media property. Test subscriber access controls, separate bot detection from authorization, and capture evidence.
By Centinel Analytica. Published 2026-10-16.
A news publisher's paywall fails in two distinct ways. Either your application grants access to someone who shouldn't have it, or your infrastructure fails to notice that an automated visitor is posing as a human reader. Engineering teams that conflate the two end up playing whack-a-mole, patching client-side scripts while the core content API stays wide open.
A Camoufox paywall audit lets technical leaders test their properties against modern browser-based automation without guessing. Camoufox is an anti-detect browser for web scraping that modifies Firefox at the C++ implementation level to address automation-detection signals like navigator.webdriver. Run it as an authorized internal audit and you get answers to two separate questions: does your authorization hold, and does your perimeter recognize automation? Here is how to scope, run, and document that audit with defensible evidence.
## Why a real-browser test belongs in a paywall audit
Curl scripts and basic headless drivers like vanilla Puppeteer no longer reflect how unauthorized actors consume metered content. Simple HTTP requests show whether your paywall relies on crude client-side tricks, such as CSS overlays or JavaScript blur filters. But modern commercial scrapers and aggregator pipelines run modified browser binaries that execute JavaScript, build complete DOM trees, and mimic legitimate user canvas renders.
Camoufox is a good example of this generation of extraction tooling. It's a customized Firefox build that handles the native browser fingerprints, WebGL variables, and runtime markers that conventional headless browsers give away. When an engineering team runs a Camoufox paywall audit, they're testing the same environment scraping networks use to parse dynamic client applications. If your paywall logic evaluates browser state, token storage, or client-rendered hydration bundles, curl will miss the flaws entirely.
A patched browser also shows whether your security boundary rests entirely on bot mitigation instead of sound authorization. A solid edge architecture has to manage both. Centinel Analytica provides bot and AI crawler protection that works alongside existing CDNs like Cloudflare, CloudFront, Akamai, and Fastly, applying allow, challenge, or block policy at the edge. A real-browser test confirms whether those edge policies interact correctly with your downstream origin permissions.
## Set scope, authorization and stop conditions before any test
Auditing production systems takes precise boundaries. Before launching any test runner, get formal written authorization from both the digital publishing director and your infrastructure lead. The document must define exact URLs, targeted user tiers, IP pools, and explicit time windows. Testing access controls without those limits risks polluting analytics pipelines, corrupting reader metering databases, or triggering automated billing operations.
Set quantitative stop conditions before sending a single request. If the automated test agent hits an HTTP 5xx error rate above 1% over any five-minute window, the test halts immediately. If origin CPU utilization rises more than 15 percentage points above baseline, execution stops. And if your test script records a single charge against an active subscriber credit card, pull the plug.
Isolate the testing traffic so production reporting stays clean. Inject a custom header on all audit requests, such as X-Audit-Session-ID, which your logging layer can record and filter out of ad impression counts and audience analytics. Security teams writing these protocols should collect evidence in a structured way, like the playbooks in [scraper incident response](https://www.centinelanalytica.com/blog/scraper-incident-response-preserve-evidence-before-blocking). Documented constraints turn an aggressive probe into an audit your legal team can defend.
## Build a bounded test matrix: previews, subscriber content, sessions, origin vs rendered
A complete paywall audit can't rely on spot checks of the homepage. You need a structured matrix that inspects four distinct layers across your publishing stack.
First, test public preview endpoints against full subscriber assets. Publishers often serve truncated text to anonymous visitors while exposing full article text inside hidden JSON-LD metadata for search engine indexing. If an automated browser can pull the full copy from an unauthenticated script tag or dynamic hydration state, your paywall is broken no matter what the screen displays.
Second, test metered state handling across sessions. Media publishers typically give readers three to five free articles each month before asking for authentication. Configure your test runner to cycle through ephemeral cookie jars, local storage contexts, and incognito profiles. Then check whether your metering service enforces counters on the server side using persistent browser identifiers, or whether wiping local cookies resets the meter indefinitely.
Third, compare the origin HTML payload directly against the fully rendered client DOM. If the initial HTTP 200 payload from your edge CDN already contains the full article body, relying on a JavaScript framework to strip that text when a subscription cookie is missing is an anti-pattern. Record the exact byte size of the origin response next to the rendered DOM text length.
Fourth, audit authenticated sessions to catch data harvesting behind active logins. Scrapers routinely buy legitimate low-tier subscriptions and run browser agents to siphon entire archives. Defending against this pattern means auditing how sessions behave under high request volumes, an approach covered in our guide on [how to prevent authenticated scraping](https://www.centinelanalytica.com/blog/prevent-authenticated-scraping). Your matrix must record how rate limits and token revocations behave when an active session visits 50 articles in 60 seconds.
## Separate access-control flaws from automation detection
Say a Camoufox paywall audit shows that an automated browser captured an entire subscriber-only investigation. Teams often make the wrong fix here. The security team rushes to deploy tighter CAPTCHA challenges or tweak perimeter browser fingerprint rules. That's a mistake. Broken access control and automation detection are different problems.
Broken access control means the client obtained data it was never authorized to receive. If an unauthenticated GET request can retrieve subscriber copy just because the client rendered the page without running your React paywall component, you have an authorization vulnerability. No amount of bot management will permanently fix a backend that hands private data to anyone who asks. Authorization belongs on the server: verify session tokens, validate entitlements, and strip restricted text before the payload leaves your origin or edge worker.
Automation detection asks a different question: is the client talking to your application a human reader or an autonomous script? This is where perimeter intelligence comes in. In research published by Centinel Analytica, 84% of requests claiming to be GPTBot on a typical news site were fake. Attackers spoof identities and use tools like Camoufox to get past simplistic header checks. Your defense needs strict server-side access control for content boundaries, paired with edge-level verification that tells real readers from automated tools.
## Test legitimate subscriber workflows alongside the automated ones
A paywall that stops scrapers but locks out paying customers is a bad trade. Any configuration change applied during an audit has to be benchmarked against standard subscriber traffic. When perimeter security teams overtune bot mitigation to beat stealth browsers, legitimate readers pay for it with broken single-sign-on redirects, blocked cookie refreshes, and disruptive verification challenges.
Run dual control groups during your audit. While your automated Camoufox instances attempt access across varying network profiles, run parallel test scripts using standard, unmodified desktop and mobile browsers across residential IPs. Log every interstitial challenge, redirect delay, and session failure the control group hits. If a newly configured perimeter rule challenges more than 0.1% of authenticated subscriber requests, that rule is too aggressive.
Pay particular attention to mobile in-app browsers and privacy-focused user environments. Subscribers clicking links from newsletter clients or social apps operate inside webviews that strip standard headers or restrict canvas rendering. If your bot defense mistakes these privacy configurations for an automated extraction framework, customer support tickets will spike immediately. Validate these boundaries deliberately, following processes like those in our guide on [bot false positive testing](https://www.centinelanalytica.com/blog/bot-false-positive-testing-verify-visitors-before-blocking).
## Record evidence and define rollback conditions
An audit finding without reproducible technical evidence is just an opinion. For every URI in the test matrix, your audit framework must capture a complete evidence bundle. Don't rely on manual browser screenshots. Build automation that captures raw HTTP archive (HAR) files, full request and response headers, TLS handshake details, and timestamped console logs.
Capture DOM snapshots at three points: immediately on the DOMContentLoaded event, after full window load, and five seconds post-hydration. This timeline shows whether unauthorized content flashed briefly before client-side paywall scripts ran, which gives automated readers an extraction window. Hash the harvested HTML strings with SHA-256 and log the hash next to the session identifier. The result is an immutable audit record your engineering and compliance teams can inspect.
Tie your audit runs to automated rollback conditions. If you're testing edge rules in staging or canary production environments, keep a direct kill-switch in your deployment pipeline. If the error rate on standard reader traffic drifts from historical baselines during test runs, trigger an automated rollback of the edge configuration within 60 seconds. Store your baseline telemetry in Prometheus or Datadog, and make sure test execution scripts monitor those metrics via API.
## Turn findings into fixes and a repeatable regression check
Once the audit wraps up, turn the evidence bundle into architectural fixes. Start with authorization failures. If origin servers deliver complete articles to unauthenticated visitors, refactor server-side rendering pipelines to withhold subscriber content until an entitlement token validates. Move payload sanitization to the edge: use Cloudflare Workers, Fastly Compute, or AWS CloudFront functions to trim unauthenticated responses before they cross the public network.
Next, close the automation detection gaps. Review how your existing CDN handles incoming traffic. Legacy bot mitigation often depends on obsolete IP blocklists or intrusive CAPTCHAs, and modern scrapers get around both with residential proxies and patched browser runtimes. Centinel Analytica was founded by a small Berlin team of former bot builders and reverse engineers, Frederick Jahn and Simeon Räthel. The platform integrates directly with Cloudflare, CloudFront, Akamai, and Fastly, so publishers can tell real visitors from scrapers and enforce granular challenge, allow, or block policies without changing core infrastructure.
Finally, put your Camoufox test suite into your continuous delivery pipeline. An access audit run once a year guarantees regression as product teams ship new frontend components and dynamic page templates. Schedule the audit matrix against your pre-production staging environment with every major release. If a developer accidentally exposes premium article bodies in dynamic JSON bundles, the regression check catches it before the code reaches production.
## Conclusion
A disciplined Camoufox paywall audit gives media companies the ground truth about their revenue perimeter. Separate server-side authorization failures from perimeter bot evasion, and your engineering team can fix content leaks at the origin while sharpening edge detection against scraping tools. If your newsroom or digital media property needs to protect subscriber content from automated extraction while keeping its existing infrastructure, deploy Centinel Analytica. Our edge protection runs alongside Cloudflare, Fastly, Akamai, and CloudFront to tell legitimate readers from automated browser scrapers, so your access controls stay defensible.
## Frequently asked questions
### What is a Camoufox paywall audit?
A Camoufox paywall audit is an authorized technical assessment using Camoufox, a customized, fingerprint-resistant Firefox browser. It tests whether a publisher's paywall enforces server-side authorization or inadvertently leaks content to automated scraping frameworks that bypass basic bot detection.
### How does Camoufox bypass basic paywall detection?
Camoufox modifies Firefox fingerprint data at the C++ implementation level to spoof browser and hardware-related properties. Scrapers use it to mimic human navigation, which exposes paywalls that rely on client-side hiding instead of backend authorization.
### Why should publishers test origin payloads instead of rendered pages?
Many publishers deliver full article text in the initial HTTP response and use client-side JavaScript to overlay a subscription banner. Testing the raw origin payload proves whether automated tools can scrape restricted content without executing browser rendering scripts.
### How do you prevent paywall audits from disrupting real readers?
Set strict stop conditions, run audits through canary environments, tag audit traffic with distinct tracking headers, and benchmark test results against legitimate subscriber control groups to avoid false positives and analytics contamination.
---
Page: https://www.centinelanalytica.com/blog/bot-false-positive-testing-verify-visitors-before-blocking
# Bot False Positive Testing: Verify Visitors Before Blocking
Run structured bot false positive testing before tightening edge WAF rules. Protect real readers, search bots, and subscribers with this publisher test matrix.
By Centinel Analytica. Published 2026-10-15.
A platform engineer at a digital newsroom flips an enforcement toggle in their edge firewall at ten in the morning to kill an aggressive scraper. By ten fifteen, the scraper has stopped, but the newsroom Slack channel is flooded with alerts. Paid subscribers accessing articles through corporate VPNs cannot get past interstitial verification pages. Readers using assistive screen readers get stuck in silent JavaScript loops. Syndication feeds that supply partners with hourly content report HTTP 403 Forbidden errors.
Blocking automated scripts is easy if you ignore collateral damage. The hard part is proving that legitimate traffic survives the security filter untouched. When engineering teams evaluate bot detection layers on top of an existing CDN, the biggest risk isn't a single Python script scraping an article. It's locking out paying subscribers, corporate networks, and search engine indexers.
Running systematic bot false positive testing before you tighten edge rules prevents self-inflicted outages. You need an explicit test matrix, quantitative baselines, and a deterministic rollback protocol before pushing any new rule to production.
## Why false positives are the real cost of bot enforcement
False positives directly destroy publisher unit economics. When an unauthorized scraper hits your site, you pay minor compute and egress costs. When an overzealous security rule blocks a human subscriber, you lose subscription lifetime value, ad viewability impressions, and customer trust. A reader hit with an unexpected CAPTCHA or an outright block page rarely files a polite ticket with customer support. They close the tab and visit a competing publisher.
The search engine penalty is worse. If an aggressive rate limit or JavaScript challenge misclassifies Googlebot as an evasive crawler, the search engine drops those pages from the index. Search traffic that took five years to build can fall by thirty percent in a weekend because an edge rule returned 403 responses to legitimate web crawlers. Newsrooms run on tight editorial deadlines and fast discovery. Losing indexing status on breaking news breaks the business model.
Engineering teams often treat bot filtering as a classification game where a higher catch rate automatically wins. In publishing, that logic fails. A policy that blocks ninety-nine percent of scrapers but misclassifies 0.5 percent of legitimate readers costs far more than the infrastructure overhead of the scrapers it stopped. Precision beats raw aggression.
## Set baselines and acceptance criteria before you change anything
Don't touch an edge configuration until you have logged two full weeks of clean baseline metrics. Traffic patterns vary between weekday mornings, weekend evenings, and breaking news spikes. You need an unpolluted sample across all of these states.
Record four metrics during your baseline period: overall HTTP error rates (specifically 403, 429, and 503 status codes), organic search crawl frequency by user agent and IP block, challenge completion rates if your CDN already issues challenges, and reader session duration across specific network types. Segment these by subscriber state, separating anonymous readers from authenticated accounts.
Turn these baselines into written acceptance criteria signed off by both platform engineering and the audience growth team. Set hard thresholds. Challenge triggers on logged-in subscribers must stay at 0.00 percent. Error responses returned to verified search crawlers must be zero. The overall challenge rate across corporate ASNs must not drift more than 0.1 percent from baseline. If a vendor or internal configuration fails these criteria during evaluation, the rollout halts immediately. Vague standards produce compromised deployments.
## The legitimate-user test matrix: browsing, accessibility tools, approved automation, and verified crawlers
To evaluate detection safety, run synthetic and manual tests against a structured matrix of legitimate visitor types. Testing with a clean desktop browser on a residential fiber line tells you nothing about edge cases.
First, test corporate and institutional networks. Readers in finance, law, and government reach publisher sites through secure web gateways like Zscaler, shared Squid proxies, or strict egress NATs. These environments put hundreds of users behind a single IP address and often strip or standardize browser headers. Basic IP reputation filters frequently mistake these gateways for botnets. If your platform uses Centinel Analytica to separate real visitors from automated scrapers, make sure your test plan sends high-volume traffic from these multi-tenant enterprise gateways.
Second, test accessibility tooling. Screen readers such as NVDA, JAWS, and VoiceOver interact with pages differently than sighted users. They don't produce standard mouse movements, their focus rings trigger programmatic DOM events, and they can't solve visual image puzzles. If your mitigation pipeline serves challenges, verify that blind users can get past the edge without landing in infinite redirects.
Third, verify search indexers and syndication pipelines. Scrapers routinely impersonate search engines, but blocking real search engines ruins your organic traffic. Set a clear [AI crawler policy for publishers](https://www.centinelanalytica.com/blog/ai-crawler-policy-for-publishers-a-newsroom-access-template) and test whether your system handles genuine search traffic differently from impersonators. Always identify genuine crawlers with reverse DNS checks instead of raw user-agent strings. The difference between legitimate search infrastructure and spoofed scrapers is covered in our breakdown of [fake Googlebot](https://www.centinelanalytica.com/blog/fake-googlebot).
Finally, add your own internal tooling to the matrix. Newsroom RSS readers, content distribution APIs, and uptime monitors must be tested explicitly. Catalog every approved automated pipeline before testing begins.
## Test challenge completion and latency, not just block rates
Bot detection vendors love dashboards full of high block counts. A high block count means nothing if half of those blocks are abandoned user challenges. Every interstitial verification charges the visitor a latency tax.
Measure the full execution pipeline of any challenge mechanism you test. How many milliseconds does the client spend downloading, parsing, and executing the challenge payload? A mobile reader on a high-latency cellular connection will leave if an edge script adds two seconds of blocking JavaScript to the critical rendering path. Mobile bounce rates climb with every extra second of verification time. Test time-to-first-byte (TTFB) and first contentful paint (FCP) on mobile device profiles while the detection engine runs.
Track the completion rate of the challenge itself. If your edge rules route one thousand suspicious requests to an interactive verification, watch the exact percentage that complete. When a puzzle challenge shows a twenty percent completion rate among users with human browsing profiles, it isn't stopping bots. It's turning real humans away. A transparent, non-interactive verification beats an intrusive puzzle. If you need an interactive challenge, treat every abandoned one as a potential false positive.
## Hunt for unexpected blocks: logs, support tickets and segment comparisons
False positives rarely arrive with clear labels. You have to dig for them in edge telemetry. Query your CDN or edge logs for every request that ended in an HTTP 403, HTTP 429, or challenge action. Skip the global totals. Slice the data by autonomous system number (ASN), operating system, and geographic region.
Compare your block distributions against traffic norms. If an ASN belonging to a regional mobile carrier like T-Mobile or Vodafone suddenly accounts for fifteen percent of all blocked requests, your edge rules are flagging carrier-grade NAT traffic as suspicious automation. If requests with older browser headers or uncommon viewport resolutions get blocked disproportionately, your heuristics are penalizing users on older hardware. Centinel Analytica applies allow, challenge, or block policy directly through your existing CDN or edge.
Cross-reference your log analysis with customer operations. Monitor help desk tickets in Zendesk or Freshdesk for keywords like 'blocked', 'access denied', 'infinite loop', or 'captcha'. Watch social media mentions of your brand. Human users who hit broken pages complain in public long before security teams notice a dip in traffic telemetry. If complaints line up with edge rule deployments, you have a false positive problem.
## Staged enforcement and rollback: how to tighten controls safely
Never flip an edge bot rule from off to active blocking across your whole infrastructure. Safe mitigation takes a phased, four-stage rollout that lets you catch misclassifications before they hurt readers.
Stage one is silent telemetry. Route all traffic through the detection engine, but enforce an allow-only policy. The engine inspects requests, generates classifications, and emits log headers, but takes no blocking action. Run stage one for at least seven business days. Audit the classifications against your legitimate user matrix to calculate the expected error rate.
Stage two is scoped challenges on non-critical endpoints. Apply verification challenges only to high-abuse paths, such as content export APIs, login forms, and search query endpoints. Leave standard article pages untouched. This isolates your core publication while you test whether the detection logic holds up under real load.
Stage three expands enforcement to standard article templates, using passive verification instead of hard blocks. Stage four applies targeted blocks only to high-confidence headless scrapers that fail all verification steps.
Every stage needs an automated rollback plan. Write the kill-switch code before you deploy the first rule. If your error budget spikes, or if the support desk logs more than three verified reader lockouts in an hour, roll back immediately. A rollback is an engineering safety valve, not an admission of defeat.
## Turn the results into a go/no-go decision and next steps
Once your evaluation has run through the full test matrix and staged rollout, compile the findings into a clear decision document. Compare the measured data against the written acceptance criteria you set at the start. Don't accept excuses for boundary violations.
Did the detection engine flag legitimate corporate proxy users? Did assistive technology users finish their sessions without friction? Did search engine crawl velocity stay completely unchanged? If the system violated any critical threshold, the decision is no-go until the vendor or engineering team refines the underlying signatures.
If the deployment meets your acceptance criteria, document the validated configuration and schedule ongoing audits. Bot defense is never static. Scraping frameworks evolve, browser fingerprints shift, and upstream CDN rules get updates. Rerun your legitimate user matrix every quarter to confirm your false positive rate hasn't drifted. For teams building a structured validation process from scratch, our guide to running a [bot protection proof of concept](https://www.centinelanalytica.com/blog/bot-protection-proof-of-concept) lays out an operational blueprint for comprehensive bake-offs.
## Conclusion
Tightening edge security without testing false positives is a gamble newsrooms and media publishers can't afford. Blocking unwanted automation means nothing if paying subscribers and search engine crawlers get caught in the dragnet. Founded in Berlin by former bot builders and reverse engineers Frederick Jahn and Simeon Räthel, Centinel Analytica provides bot and AI crawler protection that works alongside Cloudflare, CloudFront, Akamai, and Fastly. You keep your existing CDN and WAF in place while applying precise allow, challenge, or block policies that protect real readers. Contact the Centinel Analytica team to run a controlled false positive assessment on your actual reader traffic.
## Frequently asked questions
### What is bot false positive testing?
Bot false positive testing is the practice of evaluating bot detection and WAF rules against legitimate traffic before activating automated blocks. Teams test real readers, corporate proxies, accessibility tools, and search crawlers to ensure security policies do not disrupt paying subscribers or SEO indexing.
### How do you detect false positives in bot protection logs?
Analyze edge logs for HTTP 403, 429, or challenge actions segmented by autonomous system number, user agent, and device type. Spikes in blocks on residential mobile carriers, corporate VPN gateways, or accessibility user agents indicate that the detection rules are misclassifying legitimate human visitors.
### Why do bot firewalls block corporate VPN and proxy users?
Corporate gateways route hundreds of employees through a shared IP address. Simple rate-limiting rules and static IP reputation lists frequently mistake this concentrated, high-frequency traffic for coordinated bot activity, triggering challenges or blocks for verified employees.
### Can I test bot detection without replacing my existing CDN?
Yes. Platforms like Centinel Analytica integrate directly alongside existing CDNs, including Cloudflare, CloudFront, Akamai, and Fastly. This architecture lets security teams run detection policies and verify false positive rates at the edge without replacing their current CDN or routing infrastructure.
---
Page: https://www.centinelanalytica.com/blog/scraper-incident-response-preserve-evidence-before-blocking
# Scraper Incident Response: Preserve Evidence Before Blocking
Execute a scraper incident response that preserves logs and forensic signals before blocking bots. Learn how to verify crawlers and stop bot IP rotation.
By Centinel Analytica. Published 2026-10-15.
At 2:15 AM an alert trips in your on-call channel. Product page latency has jumped by 400 milliseconds and origin CPU is at 92 percent. Every engineering team reacts the same way: find the offending IP in the CDN dashboard and hit block.
That instinct feels decisive, but it's a trap. The moment you drop an IP ban or a broad geo-block, the attacker knows you spotted them. They rotate to residential proxies, change their TLS parameters, alter their request paths, and scatter across hundreds of clean subnets. You lose the forensic trail, you never isolate what data they pulled, and you go blind to their next wave.
Good scraper incident response treats aggressive web scraping as a security incident, not a firewall tweak. You need a runbook that locks down evidence, confirms identity, and assesses financial damage before you enforce anything.
## Why blocking first destroys your best evidence
Knee-jerk mitigation creates an adversary who adapts faster than your monitoring can follow. When an operator runs an automated harvesting script against your product catalog or media archive, they watch telemetry on their end: error rates, response latencies, captcha triggers. If an immediate 403 Forbidden cuts off one IP block, the script pivots at once.
Modern scraping tools treat proxy rotation as default error handling. Commercial proxy providers maintain vast pools of exit nodes. When your edge returns a block, tools like Scrapy or custom Puppeteer scripts can quickly fall back to another egress IP. Ban them immediately and you lose the chance to observe their full footprint.
Blocking first also cuts short your behavioral observation. You can't easily learn which endpoints they mapped, how deep they crawled, or whether they took proprietary pricing data or just public sitemaps. In 2026, teams that keep the connection open during the initial discovery window can pull the operator's request sequencing, HTTP/2 multiplexing signatures, and custom client headers. Let the request run into a monitored trap, or mirror the traffic to a forensic logging queue, while you build a full profile.
## Triage in the first hour: confirm it is a scraper and not Googlebot or a partner
Before you touch any rate-limiting rules, check whether the spike is an adversary, a search engine crawler, or a contracted commercial partner. Blocking search crawlers degrades organic indexing within 48 hours. Breaking an affiliated aggregator's API access can trigger commercial penalties.
Never trust the User-Agent header. Anyone running curl, Python requests, or Playwright can send a Googlebot or Bingbot string. Centinel Analytica research found that on a typical news site, 84% of requests claiming to be GPTBot were fake. Verify search crawlers with DNS-based validation. Run a forward-confirmed reverse DNS (FCrDNS) lookup on the inbound IP: confirm the PTR record resolves to a recognized Google or Bing domain, then resolve that hostname back to the original IP. If the reverse lookup returns an unverified residential hostname or fails the forward match, you're dealing with a [fake Googlebot](https://www.centinelanalytica.com/blog/fake-googlebot).
Next, check your internal partner registry. Did marketing onboard an ad verification tool? Did enterprise sales grant a data aggregator temporary scraping rights? Then look at the access paths. Legitimate indexers respect disallow rules in robots.txt and pace requests across endpoints. Scrapers blast unindexed staging routes, query internal search APIs, or iterate sequential database IDs.
## Freeze the evidence: origin, CDN, and WAF logs, robots.txt and terms versions, and cost impact
Once you've identified non-legitimate automation, freeze the state of your infrastructure. On high-throughput platforms, ephemeral CDN logs rotate out within hours unless you export them to immutable storage. Stream raw edge access logs, Web Application Firewall (WAF) event streams, and origin ingress logs into an isolated storage bucket with Object Lock enabled.
Preserve the legal environment that was live during the crawl. Capture a timestamped snapshot of your live robots.txt file and your published Terms of Service. If your legal team files a cease-and-desist or pursues claims under computer abuse law, you'll need to prove the scraper ignored the machine-readable directives and contractual terms in place at the time.
Then calculate the infrastructure bill. Excessive bot crawls cost real money through CDN data transfer out, database read spikes, and third-party API consumption. Document server compute use, the cache hit ratio drop during the event, and origin egress spikes. Our guide to calculating [AI crawler costs](https://www.centinelanalytica.com/blog/ai-crawler-costs-build-a-business-case-from-your-own-traffic-logs) walks through the full footprint. Exact dollar figures turn a vague technical complaint into a documented business disruption.
## Capture identity signals: IPs, ASNs, TLS and browser fingerprints, session patterns, and crawler-claim verification results
Advanced scrapers hide behind distributed proxy swarms. To build a signature you can enforce, capture identity signals at the network, transport, and application layers. Start with routing data: log the Autonomous System Number (ASN), the hosting provider classification, and the reverse DNS hostname for every suspect request. Scrapers routing through hosting providers like DigitalOcean or Hetzner look very different from those using peer-to-peer residential proxy pools.
Next, extract the transport-layer fingerprint. Record the JA4 or JA3 fingerprint from the client's TLS Client Hello message. Automation tools built on standard Python libraries or Go crypto packages show cipher suites and extension orders that differ from stock Chrome or Safari. For headless browsers that mimic desktop stacks, look into [residential proxy scraper detection](https://www.centinelanalytica.com/blog/residential-proxy-scraper-detection-how-to-test-scraping-services) to spot proxy latency anomalies and TCP window size mismatches.
Then analyze the application layer. Collect the client's HTTP/2 frames, header casing, header ordering, and pseudo-header sequences. While Chrome’s HTTP/2 pseudo-header order has often been observed as :method, :authority, :scheme, and :path, pseudo-header sequences vary across different client implementations. Record session patterns too. Do requests arrive at a steady 10 per second across 500 IP addresses, or do they mimic human dwell times? Bundle the network metrics, TLS fingerprints, and session topologies into one incident profile.
## Apply the narrowest block and log the rule version so you can measure what changed
Broad blocks hurt real users and hide your tactical wins. Don't drop blanket country-level blocks or rate-limit entire API prefixes to two requests per minute. A good mitigation rule isolates the scraper's unique combination of traits and leaves valid traffic alone.
Use a layered compound rule. Instead of blocking an entire /24 subnet, write a WAF rule that matches the suspect JA4 TLS fingerprint, the specific catalog route, and the absence of legitimate session cookies. If your security platform supports progressive defense, challenge instead of dropping. A silent JavaScript or cryptographic challenge forces headless runtimes to burn CPU, which raises the operator's cost without tipping them off to an IP blacklist.
Centinel Analytica works alongside platforms like Cloudflare, CloudFront, Akamai, and Fastly. Teams use it to tell real visitors from scrapers and enforce allow, challenge, or block policies at their existing CDN edge. Whatever enforces your rule, version it in code. Store your edge routing configuration or Terraform templates in Git, tag the commit with the incident tracking ID, and record the change timestamp. After you deploy, watch origin error rates and conversion funnels to confirm the traffic drop is only unwanted bots.
## Handle and retain the evidence: access limits, redaction, retention, and when chain of custody matters (NIST SP 800-61)
Forensic artifacts need handling that meets computer security standards. The National Institute of Standards and Technology lays out evidence handling protocols in NIST SP 800-61 (Rev. 2, Computer Security Incident Handling Guide). If your executives plan to seek injunctive relief, file statutory damages claims, or submit insurance claims, informal log dumps won't survive scrutiny.
Isolate your evidence repository. Restrict read and write access to incident commanders and security leads with role-based access control. Hash your raw forensic files with SHA-256 the moment you export them, and record the hashes in a signed ledger. Any change to a log file after the incident destroys its value in legal proceedings.
Redact sensitive end-user data before long-term archiving. Scraper incident logs often capture query strings, session headers, and customer identifiers. To comply with GDPR and CCPA, redact or mask personal data from uncompromised sessions before archiving. Keep the sanitized, hashed evidence archive for at least 180 to 365 days. Experienced scraping operators often pause campaigns for weeks after hitting resistance, then resume once security teams have flushed temporary log storage.
## After the block: watch for rotation, monitor false positives, and write the post-incident review
Mitigation is never the last step of scraper incident response. Once your rule deploys, start a 72-hour period of heightened monitoring. Operators running persistent commercial extraction don't walk away when their first connection pool fails. They rewrite request headers, update user-agent strings, and switch egress routes to cleaner residential networks.
Watch error rates and telemetry for displacement signals. If catalog traffic drops by 30 percent after the block, confirm that legitimate search indexers and mobile app users didn't hit 403 or 429 status codes. Also watch downstream API routes for spikes. Scrapers that hit a wall on HTML pages often point their scripts at unauthenticated mobile JSON endpoints or internal search queries.
Close out the incident with a structured post-incident review. Quantify the event: total requests blocked, egress bandwidth consumed, origin compute spend, and unique IP addresses identified. Name the root cause, such as an exposed catalog sitemap or an unprotected pricing endpoint. Update your edge filtering policies, share the forensic indicators with your platform teams, and add detection rules so your infrastructure rejects similar harvesting campaigns before they trip a latency alarm.
## Conclusion
A good scraper incident response plan rejects the urge to panic-block. Dropping an IP address burns forensic evidence, tips off the operator, and invites fast residential proxy rotation. Capture TLS signatures, freeze CDN access logs, and verify crawler identities first, and your team holds the upper hand.
Centinel Analytica was founded by Frederick Jahn and Simeon Räthel, a small Berlin team of former bot builders and reverse engineers who know how automated data extraction works. It tells real visitors from scrapers and enforces granular allow, challenge, or block policies through your existing CDN or edge, so you keep your infrastructure while shutting down scraping campaigns cleanly. Review your edge telemetry today and build a runbook that captures evidence before the next extraction spike hits your origin.
## Frequently asked questions
### What is the first step in scraper incident response?
Confirm the traffic source before altering edge firewall rules. Run forward-confirmed reverse DNS lookups to verify if the client is a search engine crawler like Googlebot, check internal partner whitelists, and export raw CDN and origin logs to immutable storage. Preserving identity signals and traffic logs guarantees you can evaluate the scrape scope without triggering automated proxy rotation.
### How do scrapers bypass simple IP blocking?
Some scraping systems support proxy rotation after 403 or 429 responses, though standard client behavior calls for honoring rate limits and not automatically retrying forbidden requests. By routing requests through residential proxy networks containing millions of IP addresses, scrapers distribute requests across unique subnets. Stopping them requires multi-layered identification, including TLS fingerprinting, HTTP/2 frame analysis, and behavioral inspection, rather than static IP bans.
### How does Centinel Analytica stop scrapers without replacing my CDN?
Centinel Analytica works alongside Cloudflare, CloudFront, Akamai, and Fastly, keeping your CDN or WAF in place. It tells real visitors from scrapers and applies allow, challenge, or block policies at your edge. This integration provides protection against automated extractors while avoiding costly changes to your existing network architecture.
### When does NIST SP 800-61 evidence retention matter for web scraping?
While NIST SP 800-61 provides incident-response guidance, requirements for pursuing legal remedies, reporting data breaches, or filing cyber insurance claims depend on applicable laws, contracts, and insurer policies. Evidence-handling controls like maintaining chain of custody, hashing logs, and restricting access can support data integrity and admissibility, though specific requirements depend on the investigation and relevant legal standards.
---
Page: https://www.centinelanalytica.com/blog/checkout-fraud-detection-separate-bot-traffic-from-payment-risk
# Checkout Fraud Detection: Separate Bot Traffic from Payment Risk
Checkout fraud detection fails when you conflate bots with payment risk. Learn how to separate edge automation defense from transaction scoring.
By Centinel Analytica. Published 2026-10-13.
A payment gateway engineer watching an authorization spike sees one problem: a stolen credit card run through an API. An infrastructure engineer watching the exact same burst sees another: ten thousand HTTP POST requests hitting a checkout endpoint across four thousand residential proxy IPs in sixty seconds.
Both views are accurate, but neither tool sees the full picture. Payment fraud platforms evaluate cardholder behavior, credit histories, address verification matches, and issuing bank responses. Web application firewalls look at IP reputations, TLS handshakes, and token syntax. When automated scripts hit your store and you treat checkout fraud detection as a single problem, one of two things happens. Either your payment provider racks up massive authorization fees, or your edge WAF drops legitimate buyers during flash sales.
Effective checkout protection means treating checkout abuse as two distinct operational problems on the same URL path. One is an automation problem. The other is a transaction problem.
## Two problems on one checkout endpoint: automation and payment risk
Your payment risk engine answers a specific financial question: is this cardholder authorized to spend this money, and will this transaction result in a chargeback? It looks at device hashes, billing zip codes, velocity across the payment processor's global merchant network, and issuing bank fraud scores. It assumes an HTTP client represents a human who can be held financially accountable.
Automated checkout abuse breaks that assumption before the payment engine even runs. When a headless browser script fires fifteen checkout attempts per second, the identity of the card matters less than the mechanism sending the request. If you rely only on your payment processor for checkout fraud detection, every bot request still triggers an authorization check. Depending on the pricing arrangement, processors like Adyen may charge per authorization attempt, including failed ones, whereas Stripe's standard processing fees apply to successful transactions. A burst of fifty thousand card attempts can generate thousands of dollars in processor fees within minutes, even if your payment risk scoring blocks every single charge.
An edge WAF has the opposite blind spot. It evaluates IP addresses and basic protocol signatures. It knows nothing about shopping carts, merchant inventory balances, or card validity. If an attacker routes requests through clean residential proxy pools, the WAF sees separate requests from domestic consumer ISPs and passes them through. To defend the store, split your defenses: eliminate automated clients at the network and browser layer first, then pass only verified human traffic to the payment risk engine.
## What automated checkout abuse looks like: carding, card cracking, scalping, and hoarding
Automated traffic attacks ecommerce checkouts through four distinct patterns, each with different operational costs.
Carding, also known as card testing, means running thousands of stolen primary account numbers (PANs) against low-value items or donation forms to find out which cards still work. Attackers buy unsorted card batches on dark web marketplaces and use your checkout pipeline as a free validation service. If you want the economics, [what card testing costs merchants](https://www.centinelanalytica.com/blog/what-is-card-testing) outlines how authorization fees and merchant account fines compound quickly.
Card cracking brute-forces credentials instead of testing new cards. Bots take known PANs and cycle through expiration dates and three-digit security codes until the payment gateway returns an approval code. This produces severe authorization velocity spikes against narrow card ranges.
Scalping targets high-demand inventory. Scalper networks deploy automated scripts to monitor inventory drops, add items to baskets, and finish checkout in fractions of a second. Human shoppers can't compete with that speed.
Inventory hoarding combines scraping and cart reservation logic. Bots add items to carts across hundreds of fake accounts or sessions. Many ecommerce engines lock physical stock for ten to thirty minutes while a session is active. By constantly cycling items between carts, attackers starve inventory from legitimate customers without ever submitting payment details. Your site looks sold out to shoppers, and conversion rates collapse.
## Request and session evidence that points to automation
Modern checkout bots rarely announce themselves with curl or python-requests headers. They run customized headless browsers and automation tools behind residential proxy networks. Catching them means inspecting discrepancies between the transport layer, the JavaScript execution context, and user navigation timing.
Start with TLS and HTTP protocol attributes. A request claiming to be modern Google Chrome on Windows 11 sends a distinct TLS client hello message: specific cipher suites, supported elliptic curves, and extension orders. If an attacker routes traffic through an automation binary running on Linux while spoofing a Chrome user-agent string, the TLS fingerprint (such as JA4) conflicts with the declared browser platform. HTTP/2 settings frames and pseudo-header sequences give similar structural evidence.
Session-level telemetry exposes deeper flaws. Real shoppers generate pre-checkout interactions. They view search results, browse category listings, load product pages, select sizes, and spend measurable seconds typing billing addresses. Bots hit cart and checkout endpoints directly, or they fake form filling by writing values straight into the DOM instead of dispatching genuine trusted keyboard and pointer events.
Residential proxy pools complicate this by scattering requests across hundreds of thousands of home internet connections. Catching these attacks means looking past single-IP thresholds toward telemetry inconsistencies, a challenge detailed in [residential proxy scraper detection](https://www.centinelanalytica.com/blog/residential-proxy-scraper-detection-how-to-test-scraping-services). When the transport fingerprint, browser capabilities, and session lifecycle fail to line up, the request is automated, however clean the client IP looks.
## Rate patterns that matter: choosing what to count at checkout
Most engineering teams configure rate limiting at checkout by counting total POST requests per IP per minute. That naive approach fails against distributed attacks and hurts real traffic during flash sales.
If an attacker uses ten thousand rotating residential IPs to run card tests, no single IP ever exceeds two requests per hour. A standard IP-based rate limit never fires. Meanwhile, a flash sale might push hundreds of legitimate buyers through an office corporate gateway or carrier-grade NAT pool, and the WAF blocks real buyers who share an IP address.
Choose counting keys that track application state and resource consumption:
1. Target the payment tokenization endpoint. Count token generation requests per session cookie and device context, not raw page views.
2. Count checkout creation attempts per payment fingerprint or card bin range. If your gateway gets fifty attempts with different card numbers that all share the same issuing bank identifier within two seconds, an automated card test is running.
3. Track cart creation velocity per ASN. Residential proxies are spread across disparate IP ranges, but they often cluster within specific regional consumer autonomous systems. A sudden 400% surge in checkout traffic from a single consumer ISP in a market you don't serve points to proxy abuse.
4. Monitor the time between initial cart creation and final payment submission. If it drops below twelve hundred milliseconds repeatedly across different sessions, scripts are running the transaction pipeline.
## Controls that protect legitimate buyers: allow, challenge, block
Don't force every suspicious request into a hard 403 block or a legacy image puzzle. Hard blocks cause false-positive disasters during high-volume commercial events, and intrusive image CAPTCHAs noticeably hurt checkout conversion rates.
Build a tiered policy based on classification confidence.
Allow clear human traffic immediately. If a browser passes protocol validation, shows a consistent TLS fingerprint, and exhibits genuine user interaction signals during cart navigation, send it straight to payment tokenization without friction.
Challenge sessions that show ambiguous or conflicting signatures. Use invisible proof-of-work challenges or silent browser verification scripts run at the edge before payment fields render. If the client fails the cryptographic proof or returns mismatched browser environment globals, escalate the policy. Centinel Analytica applies allow, challenge, or block policies directly at your existing CDN edge, so you can challenge suspect clients before they touch expensive backend infrastructure.
Reserve hard blocks for unrecoverable technical evidence: completely malformed TLS handshakes, known malicious automation signatures, or clients that fail invisible edge challenges outright. When an endpoint returns a block, avoid generic 500 server errors, which encourage bot frameworks to retry instantly. Return clean 400 or 403 responses, or redirect the client to an out-of-stock holding page to drain the attacker's execution budget.
## Where bot detection stops and payment-risk tools take over
A clean boundary between edge bot defense and payment-risk evaluation keeps your architecture maintainable. Bot defense belongs at the edge and transport layers. Payment risk belongs in the transaction ledger.
Bot protection tools evaluate client execution integrity: is this a real human operating a genuine browser? That layer's job ends once the request is verified as human and reaches the application server. It should never inspect raw PAN data, assess chargeback probabilities, or read financial risk tables. It only makes sure that whatever client submits data to your payment API is genuine.
Payment-risk engines take over at the financial boundary. They look at card-level parameters: whether the billing address matches the issuing bank records (AVS), whether the 3D Secure cryptographic handshake validates, and whether the card has been reported lost or stolen across the merchant network. Payment tools are poorly equipped to analyze JA4 fingerprints or detect headless browser automation in real time, but they excel at spotting stolen cards used by manual, human fraudsters.
When the two layers stay in their lanes, the system runs efficiently. The bot engine purges automated card testing and hoarding before requests reach the payment gateway. The payment engine evaluates financial risk only on clean, human-driven transactions. That protects your merchant reputation, cuts authorization fees, and keeps chargeback ratios below card brand thresholds.
## Adding bot protection alongside an existing CDN: an evidence-led evaluation
Most mid-market and enterprise retailers already route production traffic through Cloudflare, Fastly, Akamai, or AWS CloudFront. Ripping out a CDN and DNS setup just to solve automated checkout abuse adds operational risk and migration friction nobody wants.
Evaluate bot detection software on how cleanly it fits into your existing pipeline. Skip any vendor that makes you change your primary DNS records or reroute your global CDN caching architecture. Modern bot defense should run as an edge worker or lightweight integration right next to your current CDN.
Centinel Analytica is built around this model. Co-founded in Berlin by former bot builders and reverse engineers Frederick Jahn and Simeon Räthel, Centinel Analytica works alongside Cloudflare, CloudFront, Akamai, and Fastly and leaves your core CDN and WAF rules intact. It separates real visitors from automated scrapers and carding scripts, then applies allow, challenge, or block policy through your existing edge layer.
When evaluating any add-on tool, demand a proof-of-concept run on live checkout traffic, not synthetic testing. If you're preparing a pilot, a structured [bot-protection proof of concept guide](https://www.centinelanalytica.com/blog/bot-protection-proof-of-concept) will help you define clear evaluation metrics. Measure the vendor's edge latency overhead, check false-positive rates against authenticated VIP customers, and verify that detection logic holds up against stealth browser tools without brittle, manual signature updates.
## A checklist for a two-week checkout traffic review
Before you spend budget on new vendor contracts or re-architect your checkout flow, audit your current production logs. Run a focused two-week diagnostic across your cart and payment endpoints using this checklist:
1. Calculate your payment authorization failure ratio. Pull gateway logs for the last fourteen days. If failed authorization attempts exceed fifteen percent of total authorization volume, automated card testing is almost certainly running on your checkout endpoint.
2. Map authorization volume to cart page views. In healthy ecommerce flows, product and cart page impressions outnumber payment submissions by wide margins. If payment submissions spike without a matching increase in prior cart page hits, bots are bypassing your frontend UI.
3. Measure transaction velocity per card BIN. Group failed checkouts by the first six to eight digits of the card number. Clusters of dozens of distinct card numbers sharing the same BIN within short windows point to an automated run against a stolen card database dump.
4. Correlate residential ASN traffic with session duration. Find checkout submissions coming from major consumer broadband providers where the time from session creation to checkout submission is under two seconds.
5. Benchmark gateway authorization costs against gross revenue. Calculate the total monthly dollar amount you pay your payment processor strictly for declined transactions. That number is the direct financial waste you can eliminate by filtering bot attempts before authorization.
## Conclusion
Checkout fraud detection is not a feature you toggle inside a payment portal or a WAF configuration file. If you leave automation defense to your payment provider, you pay authorization fees for the privilege of letting attackers validate stolen cards on your infrastructure.
Filter automation at the edge first, and let your payment engine evaluate financial risk only on verified human traffic. Centinel Analytica integrates directly with your existing Cloudflare, Fastly, Akamai, or CloudFront deployment, so you can detect automated browser traffic and enforce edge policies without rebuilding your delivery infrastructure. Inspect your checkout logs, find your authorization failure baseline, and secure your payment flow before the next automated wave hits your margins.
## Frequently asked questions
### How does checkout fraud detection differ from general payment fraud prevention?
Payment fraud prevention analyzes cardholder identities, chargeback histories, and financial parameters. Checkout fraud detection also addresses the automated edge layer, identifying headless browsers, scalper bots, and distributed carding scripts before they submit payment requests to your gateway.
### Can our payment processor handle bot-driven card testing alone?
Payment processors can score transaction risk and decline stolen cards, but they charge authorization fees for every attempt. Centinel Analytica blocks or challenges automated carding scripts at your CDN edge, preventing those requests from hitting your gateway and generating processor fees.
### Will adding edge bot protection slow down checkout page load times?
Modern edge bot protection executes inside serverless runtimes on your existing CDN, evaluating TLS, HTTP, and browser telemetry in single-digit milliseconds. Legitimate human shoppers pass through without experiencing disruptive page delays or traditional puzzle challenges.
### Do we have to replace our existing CDN or WAF to protect our checkout?
No. Modern solutions integrate alongside your current infrastructure. Centinel Analytica works directly with Cloudflare, AWS CloudFront, Akamai, and Fastly, allowing you to maintain your existing routing, caching, and WAF rules while adding specialized bot detection.
---
Page: https://www.centinelanalytica.com/blog/price-scraping-protection-test-controls-on-product-pages
# Price Scraping Protection: Test Controls on Product Pages
Test price scraping protection across product pages before rollout. Map endpoints, design layered limits, and audit false positives on real traffic.
By Centinel Analytica. Published 2026-10-13.
A retailer discovers an aggressive scraper not because conversion drops overnight, but because their pricing team watches competitors undercut every promotional SKU within forty seconds of publication. The first reaction is usually a hasty purchase. An engineering lead signs an enterprise contract, flips an aggressive rule on at the edge, and locks out shoppers browsing on mobile networks or behind corporate VPNs.
Price scraping protection is not a vendor widget you turn on and forget. It's an operational testing discipline. Defending public catalog prices means balancing availability against abuse, because your prices must stay visible to real shoppers, search engine bots, and price-comparison partners. Treat anti-scraping controls as an unvetted silver bullet and you'll break core revenue funnels. Engineering teams should evaluate controls per surface, test them against messy human browsing journeys, and log every anomaly before enforcing a single blocking rule.
## Start with the goal: raise the cost of bulk scraping, not hide public prices
Public ecommerce can't work like a closed banking portal. If a buyer can't see the price tag immediately, they bounce to a competitor. Any security model that hides public pricing behind forced authentication or heavy cryptographic challenges misunderstands the retail business. The goal is to make large-scale, automated catalog extraction economically unsustainable for the scraper. It is not to make it technically impossible for a single human visitor.
In the OWASP Automated Threats framework, this behavior is [OAT-011: Scraping](https://owasp.github.io/www-project-automated-threats-to-web-applications/assets/oats/EN/OAT-011_Scraping). The attacker wants fresh pricing data across fifty thousand SKUs several times an hour. You don't need to stop a rival merchant from looking up twenty items by hand. You need to force automated scrapers to spin up headless browser farms, route through expensive residential proxy pools, solve costly challenges, and handle dynamic rate thresholds. Once acquiring your pricing data costs more than the margin gained by undercutting you, the scraper moves on.
Framing the problem around unit economics changes how you design controls. Instead of chasing a one hundred percent block rate, you aim to degrade the speed, completeness, and margin of automated extraction while keeping human checkout rates intact. For a technical breakdown of how to spot this activity in your server logs, see our guide on [what is price scraping](https://www.centinelanalytica.com/blog/what-is-price-scraping).
## Map the product-page surfaces that expose prices (HTML, JSON/API, search, category, feeds, cart)
Scrapers don't browse like humans. Shoppers go from category lists down to individual product pages, but scrapers take the path of least resistance. If you only rate-limit server-rendered HTML pages, an extractor will hit your internal mobile APIs, dynamic inventory endpoints, or search handlers instead.
Start your testing plan with an inventory of every surface that exposes prices. First is the server-rendered HTML page itself, with its schema markup and microdata. Second are the client-side JSON hydration endpoints, where modern Single Page Application frameworks fetch product data via REST or GraphQL. Third are category and search result pages, which show hundreds of prices in one dense response. Fourth are public product feeds: Google Merchant feeds, Facebook catalog endpoints, and sitemap XML files that carry raw pricing attributes.
Then document the cart and checkout endpoints. Many ecommerce systems only render dynamic pricing, bulk volume discounts, or shipping-adjusted totals once an item lands in a cart session. Advanced scrapers repeatedly post to cart endpoints to pull pricing rules that never appear on static pages. If you evaluate your defenses only against the top-level product detail URL, you'll leave high-volume data conduits wide open.
## Document allowed automation before you test: search engines, partners, price feeds, monitoring, accessibility tools
Before you test against unwanted scrapers, list all approved automation. If your edge protection blocks a competitor bot but also knocks Googlebot or your main affiliate feed offline, the win becomes a business disaster. Define explicit policies for legitimate automated agents.
Search engine crawlers come first. Legitimate indexers like Googlebot, Bingbot, and regional search spiders should be identified with sound methods such as forward and reverse DNS verification, not mutable user-agent headers. Affiliate networks, syndication partners, and price comparison engines like Google Shopping also pull prices programmatically. If these partners use specific API tokens or static IP ranges, document those paths cleanly.
Don't overlook internal engineering monitors, synthetic uptime checkers like Datadog or Pingdom, and automated accessibility scanners. Accessibility crawlers read your markup to verify ADA and WCAG compliance. If your defensive layer adds behavioral traps that trip screen-reader automation or headless accessibility probes, you create real compliance risk. Centinel Analytica sits alongside your CDN, such as Cloudflare, Fastly, CloudFront, or Akamai, and helps teams separate real visitors from automated extractors without disrupting verified partner traffic.
## Design layered controls: exposure reduction, per-session and per-endpoint limits, behavioral and browser signals, graduated responses
A single defensive gate fails every time. Sophisticated scrapers adapt fast by rotating residential IP addresses and simulating human mouse trajectories. The [OWASP Bot Management and Anti-Automation Cheat Sheet](https://cheatsheetseries.owasp.org/cheatsheets/Bot_Management_and_Anti-Automation_Cheat_Sheet.html) recommends combining controls across edge, application, and backend layers, evaluating network and browser signals while using client-side fingerprinting only when necessary.
The first layer is catalog exposure reduction. Don't serve your entire pricing table, volume discount matrix, and warehouse stock levels in the initial public HTML if a customer only needs the basic SKU price. The second layer is endpoint-specific rate limiting. Set tight session budgets on search filters, JSON pagination endpoints, and cart add operations instead of one blunt global limit across your CDN.
The third layer evaluates browser and runtime signals. Scrapers built on headless automation frameworks often give themselves away through inconsistent JavaScript runtime footprints, missing canvas rendering APIs, or odd CDP (Chrome DevTools Protocol) bindings. For a closer look at how commercial extractors mask their footprints, read our breakdown of [residential proxy scraper detection](https://www.centinelanalytica.com/blog/residential-proxy-scraper-detection-how-to-test-scraping-services).
The final layer is graduated response. Never jump straight from allow to a permanent IP ban. Escalate from silent browser fingerprinting, to light background proof-of-work challenges, to dynamic rate throttling, and only then to hard blocks. Hard IP bans do nothing against modern scrapers running through rotating proxy networks with millions of residential addresses.
## Test ordinary shopping journeys: browse, filter, compare, cart, logout, mobile, VPN and shared-IP users, assistive tech
Once the controls are built, test them against real human shopping behavior. Automated suites in a CI pipeline won't reflect the erratic browsing of real consumers. You need deliberate testing that mirrors actual customer habits across difficult network conditions.
Run scripts and manual testers through complex journeys: a user lands on a sale page, applies six faceted filters in three seconds, opens seven product tabs to compare prices, adds items to a cart, and checks out. Fast manual browsing often trips simplistic rate-limiting rules. Test these flows on mobile devices that move between cellular towers and home Wi-Fi, where the IP address changes mid-session.
Next, test shoppers who reach your storefront through commercial privacy VPNs, corporate proxies, and shared university connections. In these environments, hundreds or thousands of users share one egress IP. If your price scraping protection relies on request counts per IP, you'll challenge paying customers because their office mates are also shopping on lunch break.
Finally, test with assistive technologies, including NVDA, JAWS, and keyboard-only navigation. Accessibility tools often traverse the DOM and trigger event listeners in ways that look mechanical to basic behavioral analyzers. If your bot engine flags non-standard keyboard navigation as robotic input, your store blocks vulnerable users.
## Record false positives and build an evaluation log: what to capture, thresholds for go/no-go, monitor-only first
Never roll price scraping defenses into production in active blocking mode. Run your rules in monitor-only mode for at least two to four weeks. In this phase every control evaluates traffic, calculates anomaly scores, and flags suspected bots, but lets the HTTP request complete normally.
Build a structured evaluation log for every flagged interaction. Capture the timestamp, client IP, autonomous system number (ASN), request path, HTTP method, user agent, session identifier, and the specific rule or signal that tripped the alert. Then cross-reference flagged sessions against business metrics. Did the session complete a credit card checkout? Did it log into an authenticated, ten-year-old loyalty account? If flagged traffic regularly converts into legitimate orders, your anomaly threshold is misconfigured.
Set explicit quantitative thresholds for the production go/no-go decision. For example, require the false-positive rate on authenticated sessions and cart checkouts to stay below 0.01 percent for fourteen consecutive days before you turn on automated mitigation. If a control trips legitimate shopping sessions above your acceptable rate, tune the behavioral sensitivity. Deploy without an empirical false-positive log and your edge firewall becomes an unpredictable conversion tax.
## Next steps: re-test after changes, link to the price scraping log guide and run a free site audit
Bot defense is never finished. Every time your front-end team refactors product detail templates, migrates to a new GraphQL backend, or updates mobile checkout libraries, you change the behavioral patterns visible at your edge. After any major UI or architectural update, run your shopping journey tests again to confirm your anti-bot controls haven't regressed.
If you're already looking at suspicious traffic across your storefront, read our companion guide on [what is price scraping](https://www.centinelanalytica.com/blog/what-is-price-scraping) to review raw log samples and extraction signatures. Finding out whether scrapers are siphoning your catalog comes before any edge challenge.
A small Berlin team of former bot builders and reverse engineers founded Centinel Analytica to give companies precise bot and AI crawler protection. Our solution helps teams distinguish real visitors from automated extractors and manage defensive policies alongside existing infrastructure. Your network architecture stays untouched.
## Conclusion
Unchecked catalog scraping erodes retail margins, but an unvetted bot tool that breaks shopping carts erodes revenue much faster. Treat price scraping protection as an engineering system that needs end-to-end testing, clear exception boundaries, and continuous false-positive auditing. To see how much automated crawler traffic is hitting your product pages without disrupting your existing CDN setup, work with Centinel Analytica to inspect your live traffic.
## Frequently asked questions
### What is price scraping protection in ecommerce?
Price scraping protection consists of edge and application controls designed to detect, rate-limit, challenge, or block automated bots that extract catalog pricing data. Effective solutions raise the computational and financial cost of mass scraping while allowing human shoppers, search engines, and verified price-comparison partners to access product pages without interruption.
### How do residential proxies complicate price scraping protection?
Scrapers route requests through millions of real consumer IP addresses assigned by residential internet providers. Because these IP addresses do not belong to known data centers, standard IP-reputation blocklists fail. Defenses must analyze client runtime signatures, TLS fingerprints, browser APIs, and behavioral patterns rather than relying on IP history alone.
### Why should price scraping controls run in monitor mode first?
Running controls in monitor-only mode logs would-be blocks without interrupting HTTP requests. This allows engineering teams to compare flagged sessions against successful purchases and identify false positives among mobile users, corporate VPNs, and shared university networks before enforcing active blocks.
### Can I protect product pages without replacing my current CDN?
Yes. Specialized bot defense solutions integrate directly into your current infrastructure. Centinel Analytica works alongside existing CDNs and WAFs, including Cloudflare, AWS CloudFront, Akamai, and Fastly, allowing you to identify and challenge scrapers without migrating your routing layer.
---
Page: https://www.centinelanalytica.com/blog/residential-proxy-scraper-detection-how-to-test-scraping-services
# Residential Proxy Scraper Detection: How to Test Scraping Services
Evaluate residential proxy scraper detection across scraping services and real traffic. Test detection stacks without blocking legitimate residential users.
By Centinel Analytica. Published 2026-10-11, updated 2026-10-12.
A commercial scraping service can route automated requests through tens of millions of residential broadband connections for a few dollars per gigabyte. A residential IP address used to be a reliable sign of a human. That assumption expired years ago. Commercial scraping platforms now rent residential IP pools from proxy brokers, and those pools walk straight past basic IP reputation lists and static geo-blocking rules.
When security teams test bot detection against these operators, they usually fall into the same trap. They send twenty requests through a residential gateway using curl or a stock browser script, check whether the web application firewall blocked anything, and declare victory or defeat. That's not a test. It's an anecdote. A residential IP address is routing context, not proof of automation.
Testing residential proxy scraper detection properly takes a structured, reproducible framework. Separate the network route from the execution environment, add legitimate users as controls, and measure false positive rates across repeated runs.
## Why an IP address alone does not establish automation
IP reputation databases treat network addresses as static markers of trust. In a modern residential proxy network, an IP address is a transient exit node that an automated script rents for thirty seconds.
Residential proxies work by running background client software on consumer devices. Proxy brokers buy bandwidth from home routers, smart televisions, and personal computers through software development kits bundled into free consumer utilities. When a scraper routes traffic through a residential proxy provider, the request typically exits through an IP address associated with a residential ISP or subscriber. To an edge server that inspects only the autonomous system number (ASN) or IP range, the incoming packet looks identical to a homeowner reading the morning news.
Carrier-grade NAT (CGNAT) makes this worse. Mobile network operators and regional internet service providers routinely put hundreds of independent mobile devices and home routers behind a single public IP address. If a scraper operates from a mobile or residential connection behind CGNAT and shares that IP with five hundred real subscribers, blocking the address penalizes every real user on that subnet.
Academic research confirms that network-level signals alone are unstable. A landmark study on residential proxy infrastructure showed that RTT-based detection is vulnerable to adversarial traffic scheduling, prompting the use of flow-correlation and traffic-analysis features instead ([NDSS Symposium](https://www.ndss-symposium.org/ndss-paper/beyond-rtt-an-adversarially-robust-two-tiered-approach-for-residential-proxy-detection/)). The scraper's execution environment stays on an external cloud server, while the physical transmission comes from a residential gateway. Treating the IP address as a binary verdict gives false confidence against [low-and-slow web scraping](https://www.centinelanalytica.com/blog/detect-low-and-slow-scraping) and degrades real user sessions along the way.
## Proxy use is not the same as malicious behavior
A security policy that equates proxy usage with malice will break core business metrics. Real people use proxies and privacy relays every day, often without realizing it.
Millions of iOS and macOS users browse with Apple iCloud Private Relay enabled. Their traffic routes through an architecture designed to mask the original IP address from destination sites. Corporate telecommuters browse digital storefronts and media publications through centralized enterprise VPNs. Consumer privacy tools like Mullvad and IVPN send everyday web traffic through commercial hosting providers. If your detection rules treat anonymized routing as grounds for a hard block, you'll discard high-value customers along with the scrapers.
Proxy traffic spans a range of intent. At one end sits commercial web scraping, where automated engines pull product catalogs, real estate listings, and proprietary editorial reporting. At the other end sit human subscribers protecting their home networks from tracking scripts. In between are third-party data aggregators, price comparison engines, and legitimate search indexers.
Edge infrastructure providers know this. In its analysis of machine-learning-based proxy classification, Cloudflare noted that residential proxy attacks often manifest as traffic spikes toward sensitive endpoints from many residential ASNs ([Cloudflare Blog](https://blog.cloudflare.com/residential-proxy-bot-detection-using-machine-learning/)). Good residential proxy scraper detection doesn't punish the route. It treats the proxy classification as a risk multiplier and asks for behavioral proof before it enforces anything.
## Three evidence layers: request, browser and connection
Accurate scraper detection inspects traffic across three layers: the connection layer, the HTTP request layer, and the client execution environment. Scrapers behind residential proxies change the connection layer, but they often leak discrepancies across the other two.
The connection layer covers transport parameters: TCP initial window size, IP packet time-to-live (TTL), and TLS handshake attributes. Scrapers can manipulate TLS parameters with custom client libraries, but mismatches between the declared operating system and the negotiated cipher suites show up regularly. Analyzing JA3 and JA4 fingerprints helps identify client software based on connection-handshake characteristics, though these fingerprints are identifiers rather than conclusive proof of client identity ([Cloudflare Docs](https://developers.cloudflare.com/bots/additional-configurations/ja3-ja4-fingerprint/)).
The request layer covers protocol compliance and header behavior. Modern browsers emit HTTP/2 and HTTP/3 frames with precise pseudo-header serialization, header casing, and compression patterns. Scrapers that go through intermediate HTTP proxies often drop or reorder these pseudo-headers (:method, :authority, :scheme, :path) during forwarding. Cryptographic verification such as HTTP Message Signatures ([RFC Editor](https://www.rfc-editor.org/rfc/rfc9421.html)) verifies covered HTTP components against a signing key to detect whether those specific parts were modified in transit.
The browser execution layer interrogates the runtime environment. Advanced scrapers use headless frameworks like Playwright, Puppeteer, or modified Chromium binaries. These runtimes often fail subtle JavaScript environment checks. They leak automation flags in the Chrome DevTools Protocol (CDP), expose inconsistent navigator.plugins arrays, fail WebGL rendering benchmarks, or handle timing loops in ways no human would. OWASP documents these client-side execution boundaries in its guidance on automated threats ([OWASP](https://cheatsheetseries.owasp.org/cheatsheets/Bot_Management_and_Anti-Automation_Cheat_Sheet.html)). Reliable detection correlates all three tiers.
## Designing the test: scraping services, proxy configurations and legitimate-user controls
Testing your defenses takes a repeatable lab setup, not a pile of ad-hoc scripts. You need to simulate the exact tools scrapers use and build control groups that mimic real visitors. When you prepare this evaluation, follow the methodology in our guide on [running a bot-protection proof of concept](https://www.centinelanalytica.com/blog/bot-protection-proof-of-concept).
First, pick scraping test subjects across three categories. Category one is raw headless browsers (Playwright, Puppeteer, Selenium) with community stealth plugins, connected to residential proxy backbones. Category two is commercial scraping APIs (such as ScrapingBee, Zyte, or Bright Data Web Unlocker) that handle browser rendering, header generation, and IP rotation automatically. Category three is lightweight protocol scrapers using Python requests, HTTPX, or curl-impersonate, routed through residential backbones.
Second, configure your proxy matrix systematically. Test rotating residential proxies (every HTTP request gets a fresh IP address) and sticky residential sessions (an IP persists for ten to thirty minutes). Scrapers use sticky sessions to move through stateful checkout flows or paginated archives.
Third, build a strict control group. Collect baseline requests from real human testers on standard residential broadband, 5G mobile data, and corporate VPNs. Have them visit the same product listings or article pages in standard consumer browsers with no automation tools. If your detection stack flags any member of this group, your threshold rules are broken. A system that catches ninety-nine percent of scrapers while blocking two percent of your real mobile readers is a liability.
## Running it: repeated runs, recorded configurations and dated results
A single benchmark snapshot is useless because scraping tools and bot detection engines change constantly. An anti-bot vendor might update its signature database on Tuesday, and a proxy provider might cycle its ASN pool on Thursday.
Run your test suite at regular intervals across at least fourteen days. Execute identical automated jobs every six hours. That cadence shows how your detection stack handles shifting residential proxy quality, regional latency changes, and IP pool rotations. Record the runtime parameters for every request: target path, timestamp, proxy egress IP, client JA4 fingerprint, declared User-Agent, HTTP protocol version, and the edge response code.
Keep careful records of your scraper configurations too. If you test an automation engine against your edge, save the package versions of the framework, the specific stealth patches applied, and the proxy provider session parameters. When a scraper gets past your defenses, you need to replay the identical configuration forty-eight hours later to isolate whether the bypass came from an IP reputation gap or a failure in browser runtime inspection.
Centinel Analytica published its own benchmark of eight scraping tools against seven vendors. Treat those recorded observations as a starting point for testing your own configurations. On a typical news site, 84% of requests claiming to be GPTBot were fake. Testing with recorded configurations means your internal benchmarks reflect real defensive capability, not temporary vendor cache hits.
## Scoring false positives and reading results honestly
When security teams analyze test data, confirmation bias creeps in. They highlight a high catch rate against brute-force headless scripts and ignore the collateral damage to normal visitors. Catching that failure takes a deliberate process to [investigate bot-detection false positives](https://www.centinelanalytica.com/blog/investigate-bot-detection-false-positives).
Don't reduce detection efficacy to one aggregate percentage. Split your scoring into four metrics: Scraper Catch Rate (Recall), Clean Traffic Pass Rate (Specificity), False Positive Rate (FPR), and Challenge Completion Friction. If your edge configuration throws heavy visual CAPTCHAs at forty percent of incoming requests, your scraper block numbers might look great, but your conversion rates will collapse.
Score test traffic by request intent and session depth. Check whether your stack caught the scraper on its first request or whether it pulled five pages of catalog data before the edge stepped in. For content scraping and competitive price monitoring, letting a scraper read five pages per session hands over the whole payload.
Read the results with skepticism. If a commercial scraping service got through, isolate which layer failed. Did the edge miss the residential proxy IP? Did the engine miss the CDP runtime footprint? Or did the edge issue a silent challenge that the scraper's rendering engine solved on its own? Pinning down the failure point keeps teams from buying another network firewall when the real gap is browser-layer telemetry.
## Next step: build your own evidence with a site audit
Synthetic lab testing gives clarity, but live production traffic shows your actual financial and operational exposure. Scrapers aimed at your production assets use custom code tailored to your application structure, asset endpoints, and GraphQL interfaces.
The most practical next step is to audit live production traffic at the edge. Inspect inbound requests for the telltale signs of residential proxy abuse: high volumes of unique residential IP addresses hitting sequential content endpoints, missing or disordered HTTP/2 pseudo-headers, and clients that send desktop user agents with mobile network round-trip latencies. Our recommendations on [how to prevent web scraping](https://www.centinelanalytica.com/blog/how-to-prevent-web-scraping) cover deeper structural hardening.
Centinel Analytica provides bot and AI crawler protection built for this evaluation. A Berlin team of former bot builders and reverse engineers, co-founded by Frederick Jahn and Simeon Räthel, built it to tell real visitors from scrapers and apply allow, challenge, or block policy directly through your existing CDN or edge. Evaluate Centinel against your own scraper configurations and legitimate-user controls before widening enforcement.
## Conclusion
Residential proxies have permanently broken the assumption that an eyeball IP address is a human being. Defending a media or e-commerce property means treating network addresses as routing context and verifying browser integrity and execution behavior. Ad-hoc curl tests will never show your true exposure.
Take control of your bot policy with verified traffic intelligence. Centinel Analytica deploys onto your existing Cloudflare, Fastly, CloudFront, or Akamai infrastructure to identify scrapers hiding behind residential networks, with legitimate-user outcomes included in your evaluation. Request an edge audit with Centinel Analytica today to see the residential scrapers getting past your current defenses.
## Frequently asked questions
### How do residential proxies bypass traditional IP rate limiting?
Residential proxy brokers maintain pools of millions of consumer IP addresses. Scrapers route each subsequent HTTP request through a different consumer connection. Because no single IP address exceeds standard rate-limiting thresholds, traditional volume-based firewalls see each request as an isolated visit from a distinct home subscriber.
### Can residential proxy scraper detection cause false positives for real users?
Yes, if the detection engine relies strictly on IP reputation. Legitimate visitors routinely browse through residential networks, mobile CGNAT gateways, and consumer privacy services like Apple iCloud Private Relay. Accurate detection requires verifying client execution environments and protocol attributes rather than blocking residential ASNs outright.
### Why does TLS fingerprinting alone fail to stop residential scrapers?
Modern scraping engines use tools like curl-impersonate or customized browser binaries that accurately mimic the TLS extension lists, elliptic curves, and cipher suites of consumer browsers. While JA4 signatures catch unsophisticated scripts, advanced residential scrapers match the cryptographic footprint of normal browsers, requiring deeper runtime analysis.
### How does Centinel Analytica integrate with my existing CDN to detect scrapers?
Centinel Analytica works alongside Cloudflare, CloudFront, Akamai, and Fastly without replacing your existing CDN or WAF. It runs at your edge, evaluating incoming traffic to distinguish real visitors from automated scrapers and enforcing allow, challenge, or block policies based on verified client behavior.
---
Page: https://www.centinelanalytica.com/blog/ai-crawler-policy-for-publishers-a-newsroom-access-template
# AI crawler policy for publishers: A newsroom access template
Build an AI crawler policy for publishers across editorial, legal, and tech. Classify bots, verify IPs, and stop scrapers without breaking search traffic.
By Centinel Analytica. Published 2026-10-11.
A publisher's engineering team updated their robots.txt file to block automated AI retrieval and called it settled. Three weeks later, editorial found that syndication partners had lost access to content feeds, commercial saw an active licensing negotiation stall, and security discovered that automated traffic claiming to be conversational models had doubled.
The mistake was treating automated scraping as a server configuration. An AI crawler policy for publishers is not a text file you push to production on a Friday afternoon. It is an operating agreement between editorial directors, licensing teams, infrastructure engineers, and security leads. Each group wants something different. Editorial wants wide distribution and protection from wholesale plagiarism. Licensing wants commercial leverage and clear usage metrics. Infrastructure wants edge compute bills to stay flat. Security has to separate malicious impersonators from legitimate consumers.
When these groups don't align on access rules, the newsroom loses control of its intellectual property. You can't enforce what you haven't categorized, and you can't negotiate licensing terms if you can't prove who is scraping your archives. Here is how media organizations set governance before touching their edge rules.
## Why a crawler policy is a cross-team decision, not a robots.txt edit
Engineers want clear operational rules. Tell an infrastructure engineer to block automated scrapers, and they will add Disallow directives to robots.txt. If that fails, they might write a basic User-Agent block at the content delivery network edge. Neither fixes the business problem.
Robots.txt is an advisory protocol defined in RFC 9309 ([IETF](https://datatracker.ietf.org/doc/html/rfc9309)). It has no cryptographic weight and no technical enforcement. Anyone can read your directives, ignore them, and crawl every article in your archive. Relying on an advisory file means outside organizations set your distribution terms.
Commercial and editorial teams have priorities that pure technical blocks break. A blanket block on automated fetching can kill legitimate syndication feeds, real-time search indexing, and citation features that send referral traffic back to original investigative reporting. Open access with no conditions hands proprietary reporting to commercial large language model trainers with no attribution or payment.
Publishers need a formal governance group. It should include the Chief Technology Officer, Head of Security, Editorial Director, and Chief Commercial Officer. Engineering defines verification mechanisms and technical boundaries. Security audits log anomalies and blocks bad actors. Editorial decides which sections of journalism stay public or protected. Commercial decides which third parties get access under contract. Without this table, any crawler policy collapses into conflicting tickets and emergency rollbacks.
## The four access classes: approved search, declared AI crawlers, user-initiated agents, and unknown or impersonated crawlers
Automated requests hit your edge in a continuous stream. To manage them, split traffic into four classes based on intent, attribution, and business relationship.
The first class is approved search engines. Googlebot and Bingbot parse your pages to populate search results and send organic traffic to your publications. You want them on your public news pages, governed by standard crawling directives and sitemaps.
The second class is declared AI crawlers. Model creators run these systems to collect pre-training or fine-tuning datasets. Examples include OpenAI's GPTBot ([OpenAI Docs](https://platform.openai.com/docs/gptbot)) and Anthropic's ClaudeBot ([Claude Support](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)). These operators publish identity documentation, identify themselves via User-Agent strings, and generally respect standard exclusions. Their purpose is data ingestion, not referral traffic.
The third class is user-initiated retrieval agents. When an end user asks a conversational assistant about breaking news, the system may fetch your live page to cite a current source. ChatGPT-User and SearchBot fetch content to answer an active user prompt. Block them and your reporting disappears from real-time answers. Allow them and you permit real-time summarization, often with no click back to your masthead.
The fourth class is unknown or impersonated crawlers. Headless browsers, commercial scrapers, data brokers, and unauthorized bots that spoof legitimate User-Agent strings all land here. One rule for all four groups guarantees failure. Each class needs its own policy decision.
## Identity evidence by class: published IP ranges, reverse and forward DNS, and why user-agent text alone proves nothing
Anyone with four lines of Python can set their User-Agent header to "Googlebot/2.1" or "GPTBot/1.0". Trusting User-Agent strings as proof of identity is like treating a handwritten nametag as a passport.
Published research from Centinel Analytica found that on a typical news site, 84% of requests claiming to be GPTBot were fake. Attackers and unauthorized data harvesters routinely disguise their scrapers as benign search bots or licensed AI agents to get past basic CDN web application firewall rules. If your security team grants access on request headers alone, your content is leaking continuously.
Real verification needs network-level evidence. Major operators publish verifiable network infrastructure. Google gives explicit instructions for verifying Googlebot using reverse DNS lookups confirmed by forward DNS lookups ([Google Search Central](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests)), alongside published IP lists ([Google Common Crawlers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers)). When a request arrives from an alleged Google IP, you run a reverse DNS lookup to confirm it resolves to a .googlebot.com or .google.com domain, then a forward DNS lookup on that domain to confirm the originating IP matches.
AI companies follow similar patterns. OpenAI publishes discrete IP ranges for its training crawler in [gptbot.json](https://openai.com/gptbot.json), its search engine bot in [searchbot.json](https://openai.com/searchbot.json), and its user-prompted fetcher in [chatgpt-user.json](https://openai.com/chatgpt-user.json). If an incoming request claims to be GPTBot but comes from a residential proxy pool or an unlisted cloud hosting provider, it belongs in class four. It's an impersonator and should be blocked on sight. Read our breakdown on [fake Googlebot](https://www.centinelanalytica.com/blog/fake-googlebot) to see how deep crawler spoofing goes.
## The policy matrix: resource, permitted purpose, identity evidence, allow/challenge/block decision, review owner (with a hypothetical regional newsroom example)
A working AI crawler policy for publishers needs a structured decision matrix. You can't enforce a vague principle like "protect investigative journalism." You have to map specific digital resources to permitted purposes, required verification standards, enforcement actions, and responsible business owners.
Take a hypothetical regional publisher, Metropolitan Media Group. They publish daily news, premium local investigations, high-school sports statistics, and an archived database spanning forty years. Here is how their matrix sets edge decisions:
For breaking news paths (/news/breaking/*), the permitted purpose is organic indexing and live citation. Required evidence is verified rDNS for search engines or verified IPs from searchbot.json. The action is allow, owned by the Editorial Lead.
For those same breaking news paths facing model training ingestion, the required evidence is a verified IP from gptbot.json. The action is block unless a formal contract exists, owned by the Commercial Director.
For investigative reporting (/investigations/premium/*), the permitted purpose is subscriber reading only. The required evidence is a valid subscriber session token. The action is challenge or block for all automated agents, owned by the Head of Subscription Operations.
For the legacy archive (/archives/1980-2024/*), deep search indexing requires verified search engine rDNS, with rate limits enforced. The review owner is the Head of Infrastructure. Any bulk data harvesting attempt or unverified bot targeting the archive gets an immediate edge block, owned by the Head of Security.
For site search endpoints (/api/search/*), internal reader queries require valid CSRF tokens and genuine browser signatures. Any automated request gets a challenge, owned by the Head of Security.
This matrix removes guesswork. When OpenAI introduces a new agent or a commercial crawler targets archive paths, engineering doesn't improvise. They check the matrix. For more on calculating the server overhead from these bots, read our guide on [AI crawler costs](https://www.centinelanalytica.com/blog/ai-crawler-costs-build-a-business-case-from-your-own-traffic-logs).
## Exceptions, licensing conversations and review cadence: who can grant a carve-out and how it expires
A rigid policy with no exception process creates shadow workarounds. A commercial lead finalizes an experimental content agreement with an AI search platform, then finds that engineering's automated edge rules are dropping every one of the partner's verification requests.
Exceptions must be formal, programmatic, and temporary. No single department can grant one alone. If commercial negotiates an evaluation agreement with a model vendor, the carve-out needs three parameters: an authenticated identifier (such as dedicated egress IP ranges or a cryptographic request token), a strict time horizon, and an explicit bandwidth budget.
Carve-outs must expire automatically. Every temporary access permit lives in a central registry with a hard calendar deadline, typically 30, 60, or 90 days. When the evaluation ends, the edge rules revert to block unless the commercial team presents an executed licensing agreement that includes ongoing engineering support.
Run a quarterly crawler governance review. Every three months, the working group inspects four metrics:
First, crawler volume by verified identity versus declared User-Agent.
Second, the financial and performance impact of crawler bandwidth on CDN and origin infrastructure.
Third, the commercial status of active licensing conversations.
Fourth, traffic drift or new crawler types detected in origin server logs.
A recurring quarterly audit keeps newsrooms ahead of fast-moving crawler behavior. When a vendor launches a new retrieval bot, your organization runs it through an established workflow instead of reacting in panic.
## Where robots.txt ends and enforcement begins, including the AIPREF drafts as not-yet-standards
Publishers often ask whether emerging web standards will stop content scraping without active security enforcement. They won't. Protocol initiatives are expanding, but they are declarative preference systems, not access controls.
The Internet Engineering Task Force (IETF) formed the AI Preferences working group ([IETF AIPREF](https://datatracker.ietf.org/wg/aipref/about/)) to standardize ways to express and attach preferences about how content is collected and used for AI, with a scope including mechanisms using robots.txt and HTTP headers. Working drafts like draft-ietf-aipref-vocab ([IETF Vocab Draft](https://datatracker.ietf.org/doc/html/draft-ietf-aipref-vocab-07)) try to define structured vocabularies for AI data usage, separating model training, fine-tuning, search indexing, and real-time inference. Another specification, draft-ietf-aipref-attach ([IETF Attach Draft](https://datatracker.ietf.org/doc/html/draft-ietf-aipref-attach-04)), outlines methods to bind these preferences to HTTP headers, HTML metadata, or origin-level configuration files.
These efforts give a useful policy taxonomy, but they are not ratified standards. Even if finalized, AIPREF stays voluntary. Bad actors, residential proxy botnets, and shadow scrapers will ignore AIPREF directives exactly as they ignore robots.txt. A protocol declaration tells honest vendors what you permit. It does not stop unauthorized operators from stealing your reporting.
Real enforcement starts at the network perimeter. Your web application firewall and bot management layer must judge every automated request on its technical merits. If an agent can't present verifiable IP evidence, shows headless browser signatures, or hits protected investigative assets in high-frequency bursts, your edge must drop the connection no matter what your robots.txt or AIPREF headers say.
## Turn the template into evidence: audit your own traffic against it
Before you change edge configurations or sign a syndication contract, get baseline numbers. Start by auditing 30 days of raw edge access logs. Don't rely on web analytics suites like Google Analytics, which miss automated requests that never run JavaScript.
Map your incoming automated requests against the four access classes. Count how many requests claim to be search engines or verified AI bots, and run IP validation on them. You will almost certainly find heavy crawler impersonation. Unverified headless browsers and commercial scrapers often make up tens of millions of server hits per month on mid-market news domains. Follow our walkthrough on [how to audit AI crawler traffic](https://www.centinelanalytica.com/blog/audit-ai-crawler-traffic) to parse these server logs accurately.
The audit turns your policy matrix from abstract theory into commercial evidence. When licensing executives negotiate with AI labs, they can bring exact numbers on genuine crawl volume versus scraped bandwidth. When engineering configures CDN rules, they have verified IP filters ready to deploy.
Centinel Analytica provides bot and AI crawler protection that tells real visitors from scrapers and applies allow, challenge or block policy through your existing CDN or edge. A small Berlin team of former bot builders and reverse engineers, co-founded by Frederick Jahn and Simeon Räthel, built the platform to solve this visibility problem. It works alongside Cloudflare, CloudFront, Akamai and Fastly. Your CDN or WAF stays in place while you enforce access rules based on verified identity.
## Conclusion
An AI crawler policy for publishers is only as good as the technical enforcement behind it. Drafting access classes, naming review owners, and standardizing licensing carve-outs gives your organization governance. But if your edge can't tell genuine verified crawlers from malicious impersonators, your policy is a wish list.
Stop trusting raw User-Agent headers to protect your journalism. Centinel Analytica integrates directly with your existing Cloudflare, Fastly, Akamai, or CloudFront deployment to evaluate every automated request, verify genuine bots, and block spoofed scrapers before they reach your origin. Test your site against actual scraper behavior and take back control of your content distribution.
## Frequently asked questions
### What is an AI crawler policy for publishers?
An AI crawler policy for publishers is an operational framework that governs how automated AI agents, search engines, and scrapers access journalism assets. It aligns editorial, legal, commercial, and technical teams on which bots are allowed, challenged, or blocked based on verified identity and business agreements.
### Does robots.txt block all AI crawlers?
No. Robots.txt is a voluntary standard defined under RFC 9309. While reputable operators like OpenAI or Anthropic generally respect it, thousands of commercial scrapers, data harvesters, and headless browsers ignore it entirely. Effective protection requires active network enforcement and IP verification at your CDN edge.
### How do I know if an AI crawler request is authentic?
You cannot rely on the User-Agent header alone, as scrapers spoof headers constantly. Legitimate crawlers provide verifiable network evidence. For example, OpenAI publishes authoritative IP ranges in JSON files, and Google requires two-way reverse and forward DNS validation to confirm authenticity.
### Can publishers allow search crawlers while blocking AI training crawlers?
Yes. Publishers can permit search indexing agents while blocking training bots by validating requests against distinct IP lists and user agents. For instance, you can allow OpenAI SearchBot or Googlebot via reverse DNS while blocking GPTBot or unauthorized scrapers through targeted edge rules.
---
Page: https://www.centinelanalytica.com/blog/ai-crawler-costs-build-a-business-case-from-your-own-traffic-logs
# AI crawler costs: Build a business case from your own traffic logs
Calculate your true AI crawler costs from raw server logs and cloud invoices. A worksheet guide covering origin bandwidth, compute spend, and engineering time.
By Centinel Analytica. Published 2026-10-11.
A cloud hosting bill arrives with an unexplained spike in origin egress and dynamic compute instances. In Google Analytics, your audience numbers stayed flat over the same thirty days. The gap between your marketing metrics and your infrastructure invoice is almost certainly filled with automated crawlers scraping article catalogs, product inventories, and editorial archives.
Engineering and media operations teams often know bots are eating infrastructure resources. Knowing isn't enough to convince finance or executive leadership. You can't justify security budgets, rate-limiting policies, or edge blocks with vague assumptions about bots. You need an exact, defensible accounting of what automated traffic extracts from your systems. This guide gives you a repeatable worksheet to calculate your AI crawler costs using your own edge logs, origin metrics, and vendor invoices.
## Why network-wide crawl ratios are not your cost number
Aggregated research reports give a useful view of global traffic trends. Network data from companies like Cloudflare tracks crawl ratios and changes in search engine discovery patterns across millions of websites ([Cloudflare Blog](https://blog.cloudflare.com/ai-search-crawl-refer-ratio-on-radar/)). Those figures help you watch general web trends ([Cloudflare Blog](https://blog.cloudflare.com/radar-2025-year-in-review/)), but they're useless for your internal financial reporting.
Global averages don't reflect your site architecture. Take two publishers with identical visitor counts. The first runs an entirely static site on an edge cache with a 98% hit ratio. When an AI crawler indexes ten thousand pages, the edge absorbs almost every hit. Origin egress barely moves, and server CPU stays flat.
The second publisher runs a headless architecture with dynamic single-page rendering, live inventory checks, and an internal search endpoint. When an AI bot hits that setup, it routinely misses cache. It fires database queries, runs Node.js or Python rendering scripts, and forces origin servers to autoscale. For that publisher, the same ten thousand crawler requests produce a serious origin compute bill.
A CFO will dismiss industry averages in a budget review. Tell leadership that crawlers make up a certain percentage of web requests globally and they'll ask for your actual AWS or Google Cloud expense. To build an airtight business case, drop the third-party assumptions. Every line item in your proposal must come from your own access logs and cloud invoices.
## Step 1: Define the period and bot segment from your logs (link the fake-crawler report on spoofed user agents)
Start by picking an audit window that maps to a closed billing cycle. A standard thirty-day calendar month matches your hosting, content delivery network (CDN), and observability statements. Logs from a partial billing period introduce pro-rating errors that complicate reconciliation with finance.
Next, extract raw web server logs. If you run Nginx at your origin or load balancer layer, the default combined access-log format records the request, response status, `$body_bytes_sent`, and `$http_user_agent`, though a custom log format can also include `$bytes_sent` ([nginx.org](https://nginx.org/en/docs/http/ngx_http_log_module.html)). Filter your log store for the user agent strings declared by commercial AI platforms. Common signatures include GPTBot, ClaudeBot, PerplexityBot, Bytespider, CCBot, and Amazonbot.
Don't assume every request bearing an AI signature is who it claims to be. Anyone running an automated script can insert a custom header string. Published research from Centinel Analytica found that on a typical news site, 84% of requests claiming to be GPTBot were fake. Unofficial scrapers routinely spoof legitimate crawler headers to bypass basic access controls or exploit lax rate limits.
Split your log segment into three buckets: verified AI crawlers, spoofed crawlers, and undeclared automation. Legitimate search and model providers publish specific IP ranges or support reverse DNS lookups. Authentic Googlebot and OpenAI crawlers, for instance, resolve back to verified domains. Put verified IPs in one bucket and unverified or spoofed IPs in another. Separating these segments shows leadership that unauthorized scraping, not just official model crawling, is driving up infrastructure bills.
## Step 2: Origin bandwidth worksheet: sum bytes_sent, subtract cached traffic, apply your invoice rate, reconcile
Bandwidth costs split into two tiers: CDN egress and origin egress. Traffic delivered from an edge cache is billed at CDN distribution rates, which are typically low. The costly part is origin egress: data moving from your cloud virtual machines or storage buckets out to the CDN or straight to the public internet.
Work through this calculation using your filtered bot segment:
1. Sum total bytes sent. Query your origin access logs for all requests from the bot segment during the billing period. Sum the `bytes_sent` (or `body_bytes_sent`) field across all responses.
2. Convert bytes to gigabytes. Divide your total byte sum by 1,073,741,824 (bytes per GiB). For example, 16,106,127,360 bytes equals 15 GiB.
3. Isolate cache misses. If you analyze edge logs rather than origin logs, subtract all responses with a cache status of `HIT`, `STALE`, or `REVALIDATED`. Keep only `MISS`, `BYPASS`, and `DYNAMIC` requests. Only responses served directly by origin compute generate origin network egress.
4. Apply your invoiced network rate. Check your latest hosting invoice for your effective data transfer out (DTO) tier. Cloud providers charge variable pricing based on destination and volume. Google Cloud, for example, bills external internet egress across regional tiers ([Google Cloud](https://cloud.google.com/vpc/network-pricing)). AWS charges standard tiered transfer rates out from EC2 to the internet or across regions. If your company has negotiated enterprise discount pricing, skip the public list prices and use your net cost per gigabyte.
5. Multiply volume by unit rate. If your crawler segment forced 12,000 GiB of origin egress and your net blended egress rate is $0.07 per GiB, your direct crawler bandwidth cost is $840 for the month.
6. Reconcile against your total bill. Compare your calculated crawler bandwidth sum against the total networking line item on your cloud invoice. If your calculated bot egress exceeds the actual bill, your log filters or cache hit calculations are wrong.
## Step 3: Compute worksheet: cache-miss share, allocate compute spend by request share or CPU time, keep separate from egress
Origin compute is usually the biggest financial drain from aggressive web crawling. Network egress scales linearly with payload size, but compute scales with application workload complexity. Crawlers rarely pull static assets like stylesheets and optimized images. They target text endpoints, paginated search results, deep archive indexes, and category filters that force database execution and dynamic template generation.
Treat compute separately from bandwidth. Blend them into a single average cost per request and you get inaccurate numbers that finance will question.
To allocate compute expenses defensibly, pick one of two models based on your logging maturity:
Model A: Request-share allocation. This works well for homogeneous backend applications where individual endpoints consume similar CPU resources. Count the total origin cache-miss requests handled by your web cluster during the month. Then count the cache-miss requests generated by your bot segment. Divide bot misses by total misses to get the bot share percentage. If your application tier cost $14,000 in container or EC2 compute and bots made up 18% of all origin cache misses, allocate $2,520 of compute to crawlers.
Model B: CPU execution time allocation. This one fits if you run application performance monitoring tools such as Datadog, Dynatrace, or New Relic. Measure the mean response time and CPU time of crawler-targeted routes against regular visitor routes. Crawlers pulling historical content often trigger expensive database reads and server-side rendering steps that burn four times the CPU cycles of a typical home page request. Sum total CPU time consumed by the crawler user agents, divide it by total cluster CPU execution time, and apply that ratio to your raw compute bill.
## Step 4: Operational costs: engineering hours, rate-limit tuning, log storage and support, using your own timesheets and leaving unknowns blank
Infrastructure invoices show only part of the expense. Automated scrapers routinely trigger on-call alerts, degrade database performance, and eat platform engineering cycles. Counting staff and operational overhead turns a basic hosting calculation into a complete business case.
Gather data for three operational buckets:
1. Incident response and maintenance hours. Pull historical tickets from Jira or your project management tool. Tally the engineering hours spent investigating bot-driven CPU spikes, writing temporary firewall rules, modifying `robots.txt` directives, and analyzing edge WAF performance. Multiply these hours by your company's loaded engineering cost (salary plus benefits divided by working hours). If platform engineers spent twenty hours on crawler-induced performance incidents at a loaded rate of $95 per hour, document $1,900 in engineering labor.
2. Observability and log ingestion expenses. High-volume scraping creates millions of log lines. If you stream access logs to Datadog, Splunk, or an Elastic cluster, your vendor bills you per gigabyte of ingested and indexed data. Review your observability bill, calculate the percentage of log volume generated by the bot segment, and multiply it by your log ingestion rate.
3. Support and platform tickets. If scraper traffic caused origin timeouts that interrupted internal workflows, record the time it took to triage those issues.
Be rigorous. If you don't track engineering hours on bot maintenance, leave that line item blank. Don't invent an arbitrary figure like twenty thousand dollars of lost productivity. A worksheet with two documented cost items and two blank entries earns immediate respect from finance. An inflated, speculative estimate invites skepticism and weakens your whole case.
## Step 5: Fill-in summary table, ranges and caveats, and weighing cost against referral value from your own logs
Pull your findings into a single summary table. Give a clear low estimate and high estimate to account for caching variations and rate tiers:
| Cost Category | Low Estimate | High Estimate | Primary Data Source |
|---|---|---|---|
| Origin Egress Bandwidth | $620 | $840 | Origin Nginx logs & cloud egress bill |
| Origin Compute (App / SSR) | $1,800 | $2,520 | APM CPU metrics & VM cluster invoice |
| Log Ingestion & Storage | $310 | $420 | Observability vendor invoice |
| Operational Engineering Time | $0 (untracked) | $1,900 | Jira incident tags & loaded hourly rate |
| **Total Monthly Spend** | **$2,730** | **$5,680** | **Consolidated infrastructure ledger** |
Once you have your total expense range, check what your business gets back for this spend. Proponents of unrestricted bot access argue that AI crawlers provide brand discovery and referral traffic. You can audit that claim inside your own web analytics platform.
Filter your web analytics for referral traffic from AI domains such as chatgpt.com, claude.ai, and perplexity.ai. Research across the publishing industry shows the ratio of crawler requests to click-through referrals is badly lopsided ([Cloudflare Blog](https://blog.cloudflare.com/crawlers-click-ai-bots-training/)). Many content sites handle hundreds of thousands of crawler requests and get only a handful of referral sessions back.
Calculate your cost per referral visit: divide your crawler infrastructure expense by the referral sessions you actually recorded. If crawlers cost your systems $4,000 per month and delivered 200 incoming visits, you're paying $20 per referral session. Set real costs against incoming traffic value and the argument stops being an abstract technical debate. It becomes a plain return-on-investment calculation.
## Turning the worksheet into a proposal, with the free audit page as a next step
A completed worksheet gives you immediate leverage with technical and financial leadership. Instead of asking for tooling against speculative threats, you're proposing a clear infrastructure cost reduction. Frame the project as server capacity recovery: blocking or enforcing policy on automated scrapers recoups cloud compute, frees engineering capacity, and cuts monthly networking invoices.
Your proposal doesn't need to mandate a blanket ban on every crawler. A sound policy might allow verified search indexing bots while challenging or blocking aggressive commercial scrapers, bulk data extractors, and spoofed user agents. Those controls don't require an expensive architectural overhaul.
Centinel Analytica provides bot and AI crawler protection that tells real visitors from scrapers and applies allow, challenge, or block policies through your existing CDN or edge. The platform works alongside Cloudflare, CloudFront, Akamai, and Fastly, so your current CDN and WAF stay in place. It was built by a small Berlin team of former bot builders and reverse engineers, co-founded by Frederick Jahn and Simeon Räthel.
Before you finalize your business case, audit your live traffic. Upload your access logs or route sample traffic to see how much of your inbound crawler volume comes from verified bots versus spoofed scrapers. A proposal grounded in your own log data gives executive leadership the exact financial justification it needs to approve enforcement.
## Conclusion
Uncontrolled automated crawling is an unbudgeted infrastructure subsidy that content publishers pay to third-party model operators and scrapers. Aggregate web statistics and marketing claims will never convince leadership to allocate security resources. Run your access logs through this worksheet, pull your real cloud rates, and calculate your exact monthly crawler bill.
Once you have your numbers, stop absorbing unnecessary origin spend. Centinel Analytica integrates directly into your existing Cloudflare, Fastly, Akamai, or CloudFront distribution, so you can tell authentic visitors from scrapers and enforce granular allow, challenge, or block rules without touching your architecture. Deploy Centinel Analytica today to reclaim your origin compute and stop the automated infrastructure drain.
## Frequently asked questions
### How do I identify AI crawler costs in my access logs?
Filter origin web server logs by crawler user agents, isolate non-cached cache-miss responses, and sum total bytes sent. Multiply this bandwidth volume by your invoice data transfer rate, then calculate the bot share of total origin compute requests to allocate compute spending accurately.
### Why do AI crawlers cause higher compute costs than human visitors?
Human visitors frequently browse popular pages served directly from edge CDN caches. AI crawlers systematically traverse deep pagination, complex taxonomy filters, and historical archives. These requests miss edge caches, forcing backend databases and server-side rendering engines to generate expensive responses.
### How do I know if an AI crawler user agent in my logs is legitimate or spoofed?
Legitimate crawlers publish their IP ranges or validate through reverse DNS lookups. Research from Centinel Analytica shows that 84% of requests claiming to be GPTBot on typical news sites are spoofed. Centinel Analytica inspects technical client signals at the edge to verify actual visitors from scrapers.
### Can I block AI crawlers without replacing my existing CDN?
Yes. Centinel Analytica works alongside Cloudflare, CloudFront, Akamai, and Fastly, allowing your existing CDN and WAF to stay in place. It evaluates incoming requests at the edge and enforces allow, challenge, or block policies based on verified client behavior.
---
Page: https://www.centinelanalytica.com/blog/ai-browser-agents
# AI browser agents: what a website can verify
Agentic browsers for site owners: ChatGPT's cloud browser signs its requests, while Comet and Claude in Chrome arrive as the user's own browser.
By Frederick Jahn. Published 2026-10-11.
An AI browser agent is a browser that a model drives to finish a user's task: it opens
pages, clicks, fills in forms and signs in. Vendors also call these products agentic
browsers. What a website can learn about one depends on where the browser runs.
Three of the agents below run in the vendor's cloud. ChatGPT's cloud browser and
Google-Agent publish a check, either a signed request or a list of IP ranges. The Manus
pages we read describe neither. The other four run inside the user's own browser: Perplexity
Comet, Claude in Chrome, Gemini in Chrome and the Manus browser extension. Their requests
carry the user's IP address, cookies and browser, and the pages we read describe no signal
a website can check. Gemini is a partial exception, because it can move a task to a remote
browser.
| Agent | Where the browser runs | What the site receives | Published check |
| --- | --- | --- | --- |
| ChatGPT cloud browser (ChatGPT Work) | OpenAI's cloud | Requests signed with Web Bot Auth as `https://chatgpt.com` | Verify the signature against OpenAI's key directory |
| Google-Agent | Google infrastructure | A `Google-Agent` user agent; some requests signed as `https://agent.bot.goog` | Google's `user-triggered-agents.json` ranges, and the signature when present |
| Manus Cloud Browser | Manus's cloud | Requests from data center IP addresses | None in the Manus pages we read |
| Perplexity Comet | The user's device | The user's browser, session and IP address | None described in the court opinion or Perplexity's post |
| Claude in Chrome | The user's Chrome | The user's tabs, sign-ins and IP address | None in the Anthropic pages we read |
| Gemini in Chrome (auto browse) | The user's Chrome, or a remote browser when the device is unavailable | A tab in the user's Chrome; the help page does not describe the remote browser's requests | None in the Google help page we read |
| Manus Browser Operator | The user's Chrome or Edge | The user's logins, sessions and local IP address | None in the Manus pages we read |
Six rows were checked against the vendor's own documentation on October 11, 2026. The Comet
row rests on a court's description of the product. OpenAI alone has withdrawn three agent
products, so recheck a row before you build a rule on it.
## Agents that run in the vendor's cloud
A cloud agent's browser sits on the vendor's computers. Requests reach you from the
vendor's network, and the vendor can attach proof of where they came from.
### ChatGPT's cloud browser
Two earlier OpenAI agents are gone. The help page for
[ChatGPT agent](https://help.openai.com/en/articles/11752874-agent) now says "ChatGPT agent
is no longer available" and sends readers to ChatGPT Work. The same page says Operator was
folded into agent mode: "The Operator website is no longer accessible."
The agent that browses for ChatGPT Work today is its cloud browser. OpenAI's
[cloud browser page](https://help.openai.com/en/articles/20001280-using-cloud-browser-in-chatgpt)
says it "gives ChatGPT Work its own browser on a separate computer in the cloud." It keeps
its own cookies and sessions and "does not use your personal browser's open tabs, browsing
history, saved passwords, cookies, extensions, or existing sign-ins."
OpenAI's two pages disagree about sign-in. The cloud browser page describes a form that
sends the user's password to the remote browser and a session that stays signed in for
later tasks. The
[allowlisting page](https://help.openai.com/en/articles/11845367-chatgpt-works-cloud-browser-allowlisting)
says "at launch, Cloud browser cannot sign in to websites or complete payments." We would
plan for sign-ins. One would arrive from OpenAI's network, on a browser your site has not
seen.
The allowlisting page names no user-agent token for the cloud browser. It identifies the
traffic by signature: "ChatGPT Work's Cloud browser uses Web Bot Auth to sign outbound HTTP
requests, allowing website operators to verify that requests genuinely originate from
ChatGPT." Each request carries three headers:
```
Signature: ...
Signature-Input: ...
Signature-Agent: "https://chatgpt.com"
```
The public keys are at `https://chatgpt.com/.well-known/http-message-signatures-directory`.
We fetched that address on October 11, 2026. It returned one Ed25519 key whose `exp` field
was October 18, 2026, so a verifier has to refetch the directory and cannot pin the key.
OpenAI's instruction for a CDN it does not list is to check that `Signature-Agent` matches
`"https://chatgpt.com"` "including the quotation marks", verify the other two headers under
[RFC 9421](https://www.rfc-editor.org/rfc/rfc9421.html), and allow the request only after
the signature verifies.
Vendors still file this traffic under the retired names. OpenAI lists it as "ChatGPT Agent"
in Akamai and HUMAN, as bot tag `chatgpt-agent` in Cloudflare (found by searching for
"ChatGPT Operator"), and as `chatgpt-operator` in Vercel. A search for "cloud browser" in
those consoles may find nothing.
When real cloud browser traffic is blocked, OpenAI's first troubleshooting step is to
"confirm that intermediate proxies preserve" the three signature headers. A proxy that
drops unknown headers turns a verifiable request into an anonymous one.
### Google-Agent
Google lists `Google-Agent` among its
[user-triggered fetchers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers):
it "is used by agents hosted on Google infrastructure to navigate the web and perform
actions upon user request." The desktop user agent is:
```
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Google-Agent; +https://developers.google.com/crawling/docs/crawlers-fetchers/google-agent) Chrome/W.X.Y.Z Safari/537.36
```
Google also lists a mobile string with the same `Google-Agent` token. The page says
user-triggered fetchers "generally ignore robots.txt rules" and carries a caution: "The
user agent string can be spoofed." So check the source address against the ranges in
[user-triggered-agents.json](https://developers.google.com/static/crawling/ipranges/user-triggered-agents.json).
Google also signs part of this traffic. Its
[Web Bot Auth page](https://developers.google.com/crawling/docs/crawlers-fetchers/web-bot-auth)
calls the work experimental: "A subset of requests made by the `Google-Agent` are signed
with Web Bot Auth," with `Signature-Agent` set to `g="https://agent.bot.goog"`. Google tells
sites to keep the address check as well, "as not all requests are signed."
OpenAI's page shows the header as a bare quoted string. Google's shows a labeled member,
`g="..."`. The
[IETF draft](https://datatracker.ietf.org/doc/draft-ietf-webbotauth-httpsig-protocol/)
requires the labeled form from signers ("Signers MUST send the dictionary form"). Of the
older string form it says that "deployments still send it" and that a verifier "MAY accept
that form" as a dictionary with one member. To verify both vendors as their pages stand, a
verifier has to parse both.
### Manus Cloud Browser
Manus documents where its
[Cloud Browser](https://manus.im/docs/features/cloud-browser) connects from: it "operates
from data center IP addresses, not residential IPs." The page tells users to expect more
CAPTCHAs and security checks as a result. When one appears, Manus prompts the user to "Take
Over" the browser, solve it, and hand control back. A site that challenges this browser
should expect a person to solve the challenge and the agent to carry on.
## Agents that run in the user's own browser
An in-browser agent has no network of its own. The user's browser loads each page and sends
each request, so the site sees the customer's usual device, address and session cookie.
OpenAI shipped a browser of this kind, Atlas, in October 2025. The
[announcement](https://openai.com/index/introducing-chatgpt-atlas/) now carries the note
"Atlas has since been deprecated."
### Perplexity Comet
A court opinion describes how Comet works. In
[Amazon.com Services v. Perplexity AI](https://law.justia.com/cases/federal/appellate-courts/ca9/26-1444/26-1444-2026-08-04.html),
decided on August 4, 2026, the Ninth Circuit described Comet as "a web browser that operates
like Google Chrome, running locally on a user's machine and enabling the user to navigate
the Internet." When a user sends the assistant to Amazon, it "takes screenshots of the
browser view, sends those screenshots from the user's computer to Perplexity's servers,"
and gets instructions back.
On identification the opinion says: "At the core of the dispute was Perplexity's decision
not to use a 'user-agent string,' a mechanism 'that would communicate that the user has
activated an AI agent.'" A footnote adds that the parties dispute whether Perplexity
"knowingly altered the Assistant's user-agent string" after Amazon first identified and
blocked the Assistant. So the agent could be told apart at one point, and the opinion does
not say whether it still can. Perplexity had stated its position in a
[blog post](https://www.perplexity.ai/hub/blog/bullying-is-not-innovation) when the dispute
began: "Your AI assistant must be indistinguishable from you."
The court vacated the preliminary injunction that Amazon had won and sent the case back to
the district court. Its conclusion gives two reasons. Amazon "is unlikely to succeed on the
merits of the 'access' prong of the CFAA and CDAFA analysis" (the federal and California
computer-access statutes), and "the equitable factors do not otherwise strongly favor an
injunction" either. On access the panel wrote, "It is the user who 'accesses' Amazon's
computers, with the help of the Assistant to carry out specific acts on Amazon.com." It
limited the holding to that question and that record: "We do not establish a new legal
regime governing agentic AI." This article describes the ruling and gives no legal advice.
### Claude in Chrome
Claude in Chrome is a browser extension. Anthropic's
[desktop documentation](https://claude.com/docs/third-party/claude-desktop/browser) says it
"lets Claude work in the user's own Google Chrome or Microsoft Edge, with the tabs and
sign-ins the user already has there." The
[setup page](https://support.claude.com/en/articles/12012173-getting-started-with-claude-in-chrome)
names Chrome only and says other Chromium browsers are not supported. It lists the
extension's permissions: the `debugger` permission "is what allows Claude to actually
control your browser – clicking buttons, typing text, and taking screenshots – when you ask
it to complete tasks for you."
The controls Anthropic documents belong to the user's side. Claude cannot open some site
categories, it "asks for permission before accessing financial sites," and an organization's
admin can set an allowlist or a blocklist of sites
([safety page](https://support.claude.com/en/articles/12902428-using-claude-in-chrome-safely)).
None of the three pages describes a header, a user-agent token or an opt-out for the
website being visited.
### Gemini in Chrome
The [auto browse help page](https://support.google.com/chrome/answer/16821166?hl=en)
describes a task that runs in a tab of the user's Chrome: "While Gemini in Chrome is
performing the task, there's an auto browse icon on the tab that it's using." It tells the
user: "You're responsible for Gemini's actions during a task, including mistakes and
unexpected results like purchases."
The task does not always stay on the user's device. The same page says Gemini Spark, the
agent that uses auto browse, "might also use a remote browser (which is a separate, new
browser instance, not your local browser in the cloud)," for example when the user closes
the device before the task is finished. The page names no user agent and no IP range for
either mode, and it does not say whether the remote browser sends `Google-Agent`.
### Manus Browser Operator
Manus ships the same idea as an extension for Chrome and Edge. Its
[Browser Operator page](https://manus.im/docs/integrations/manus-browser-operator) says the
agent works in the user's browser "complete with your existing logins, sessions, and local
IP address," and states the consequence for the site: "Because the activity appears
legitimate to websites, it clears standard access barriers automatically and maintains
active sessions." The Cloud Browser page lists "Website blocks data center IPs" among the
reasons to switch to this mode.
## What a signature proves
Web Bot Auth is HTTP Message Signatures
([RFC 9421](https://www.rfc-editor.org/rfc/rfc9421.html)) applied to automated clients. The
[working-group draft](https://datatracker.ietf.org/doc/draft-ietf-webbotauth-httpsig-protocol/)
dated September 1, 2026 defines the `Signature-Agent` header, a key directory in JWKS format
and the well-known address that serves it. Its authors work at Cloudflare and Google, and
its header gives the intended status as Standards Track. It is still a draft and expires on
March 5, 2027.
The draft states the limit of a valid signature. It "proves that a holder of a key that URL
publishes signed the covered message." It "says nothing about who operates the Agent,
whether the Agent is benign, or whether the request is authorized. Those are origin policy."
A signature under `https://chatgpt.com` shows that the request was signed with a key
chatgpt.com publishes. Which user sent the agent, and whether that user may take the
action, are questions your own login and permission checks have to answer.
Signing is voluntary, and the draft says so: "If an agent wishes not to identify itself,
this is not the right choice of protocol for it." An unsigned request therefore tells you
nothing about whether an agent sent it.
Few sites need to write the verifier themselves. OpenAI's page gives steps for Akamai,
Cloudflare, HUMAN and Vercel. Google's page says: "Major bot-detection services, CDNs, and
WAFs support Web Bot Auth." [How to verify AI agents](https://www.centinelanalytica.com/blog/how-to-verify-ai-agents) covers
the address checks and the identity results to record.
## Set policy by what you can verify
| What arrives | What you can establish | A starting rule |
| --- | --- | --- |
| A signed request that verifies, or a `Google-Agent` request from a published range | The vendor's signing domain or network | Apply the rule you chose for that vendor on that route. Login, checkout and account changes still go through normal authorization. |
| A `Google-Agent` user agent from another address with no valid signature, or a signature that fails | Nothing; the claim is unverified | Do not apply the vendor's allow rule. Before you call it a false claim, confirm that you compared the client's address and not your CDN's, that your copy of the range file is current, and that no proxy changed the signed headers. |
| A browser from a data center address with no identity | Only that the address is hosted. Corporate VPN and cloud desktop users match too. | Use your default policy for hosted automation. |
| A signed-in session that an agent may be driving | Only the customer | Apply the same rule to every session: per-account limits, and a confirmation step on purchases, exports and credential changes. |
Evidence in the browser can sometimes show that software is driving a session, and
[How to detect browser automation](https://www.centinelanalytica.com/blog/detect-browser-automation) covers what that
evidence supports. It does not name the product, and blocking the session blocks a
signed-in customer. Decide which actions you are willing to let software take for a
customer, and enforce that at the action.
[How to control AI agent access](https://www.centinelanalytica.com/blog/how-to-control-ai-agent-access) has the full policy
model, and [What is agentic traffic?](https://www.centinelanalytica.com/blog/what-is-ai-agent-traffic) explains how these
sessions differ from crawlers and fetchers.
## Check your own traffic
Three checks need only your edge logs.
1. Log the `Signature-Agent` request header and count requests by its value. In nginx
[a request header is a variable](https://nginx.org/en/docs/http/ngx_http_core_module.html#var_http_),
here `$http_signature_agent`:
```
log_format agents '$remote_addr "$http_user_agent" '
'"$http_signature_agent" $status $request_uri';
access_log /var/log/nginx/agents.log agents;
```
Behind a CDN, `$remote_addr` is the CDN's address. Log the client address your CDN
passes on instead. nginx
[escapes quotation marks](https://nginx.org/en/docs/http/ngx_http_log_module.html#log_format)
in a logged value as `\x22`, so OpenAI's header appears as `\x22https://chatgpt.com\x22`.
2. Search the user agent field for `Google-Agent` and compare the client addresses with
`user-triggered-agents.json`.
3. Send a request with test signature headers through your CDN and look for all three in
the origin log. A CDN that verifies signatures may reject the test request, which also
tells you who is checking:
```
curl -s -o /dev/null -w '%{http_code}\n' https://www.example.com/ \
-H 'Signature-Agent: "https://signer.example"' \
-H 'Signature-Input: sig1=("@authority" "signature-agent");created=1700000000;tag="web-bot-auth"' \
-H 'Signature: sig1=:AA==:'
```
If a bot management product sits in front of the site, look up its entries for the names in
the ChatGPT section and see how each is set. OpenAI notes that on Vercel "no additional
configuration is required to allow Cloud browser traffic," so the current setting may be
one that nobody on your team chose.
## Questions for a bot management vendor
Ask any vendor, Centinel included, three things about the agents on this page: which signed
agents the product verifies, how it treats a request whose signature fails, and what it
reports for a browser session whose vendor cannot be established. Centinel is bot
management for websites and applications, and the [bot management guide](https://www.centinelanalytica.com/bot-management)
has the full evaluation checklist.
Declared crawlers are a separate question from browser agents. The
[bot directory](https://www.centinelanalytica.com/bots) lists what each crawler operator publishes, and the
[crawler IP checker](https://www.centinelanalytica.com/verify-crawler) tests one address against it.
## Sources
- [OpenAI: ChatGPT Work's Cloud browser allowlisting](https://help.openai.com/en/articles/11845367-chatgpt-works-cloud-browser-allowlisting)
- [OpenAI: Using cloud browser in ChatGPT](https://help.openai.com/en/articles/20001280-using-cloud-browser-in-chatgpt)
- [OpenAI: ChatGPT agent](https://help.openai.com/en/articles/11752874-agent)
- [OpenAI: Introducing ChatGPT Atlas](https://openai.com/index/introducing-chatgpt-atlas/)
- [Google: user-triggered fetchers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers)
- [Google: Authenticate requests with Web Bot Auth](https://developers.google.com/crawling/docs/crawlers-fetchers/web-bot-auth)
- [Google Chrome Help: auto browse](https://support.google.com/chrome/answer/16821166?hl=en)
- [Manus: Cloud browser](https://manus.im/docs/features/cloud-browser)
- [Manus: Browser Operator](https://manus.im/docs/integrations/manus-browser-operator)
- [Anthropic: Built-in browser and Claude in Chrome](https://claude.com/docs/third-party/claude-desktop/browser)
- [Anthropic: Get started with Claude in Chrome](https://support.claude.com/en/articles/12012173-getting-started-with-claude-in-chrome)
- [Anthropic: Use Claude in Chrome safely](https://support.claude.com/en/articles/12902428-using-claude-in-chrome-safely)
- [Ninth Circuit: Amazon.com Services v. Perplexity AI, No. 26-1444](https://law.justia.com/cases/federal/appellate-courts/ca9/26-1444/26-1444-2026-08-04.html)
- [Perplexity: Bullying is not innovation](https://www.perplexity.ai/hub/blog/bullying-is-not-innovation)
- [IETF: HTTP Message Signatures for automated traffic](https://datatracker.ietf.org/doc/draft-ietf-webbotauth-httpsig-protocol/)
- [RFC 9421: HTTP Message Signatures](https://www.rfc-editor.org/rfc/rfc9421.html)
- [nginx: embedded variables](https://nginx.org/en/docs/http/ngx_http_core_module.html#var_http_)
- [nginx: log_format](https://nginx.org/en/docs/http/ngx_http_log_module.html#log_format)
---
Page: https://www.centinelanalytica.com/blog/what-is-card-testing
# What is card testing? Signs, costs and defenses
Card testing is scripted checking of stolen card details through a merchant's checkout or add-card flow. How it works, the signs, and what stops it.
By Frederick Jahn. Published 2026-09-30, updated 2026-10-11.
Card testing is the use of scripts to find out which stolen or guessed payment card details
still work. The attacker submits card after card through a merchant's checkout, add-card form,
or payment API, usually with small amounts or card-verification checks, and reads which
attempts the issuer approves. The valid cards are then used for fraud or sold. The merchant
never loses the cards, but it pays for the declines, fees, disputes, and payment-network
scrutiny that follow.
[Stripe's card-testing guide](https://docs.stripe.com/disputes/prevention/card-testing) lists
"carding," "account testing," "enumeration," and "card checking" as other names for the same
activity. The payment networks draw a finer line, which the next section explains.
## Card testing, enumeration and carding
[Visa's guidance on enumeration attacks and account testing](https://usa.visa.com/content/dam/VCOM/global/support-legal/documents/visa-guidance-to-guard-against-enumeration-attacks.pdf)
separates two schemes that both run on automated scripts:
| Term | What the attacker has | What the script does |
| --- | --- | --- |
| Account testing | Card details obtained illegally | Submits one or two low-amount transactions to see if each account is active |
| Enumeration | A card range (BIN), without the full details | Iterates through card numbers, expiry dates, CVV2 and postal codes until the issuer approves |
The [OWASP automated threats list](https://owasp.org/www-project-automated-threats-to-web-applications/)
uses a similar split: OAT-001 Carding tests bulk stolen card data, and OAT-010 Card Cracking
guesses the missing values. For a merchant the
difference matters less than the shared pattern: many automated payment attempts, most of
them failing, sent through a flow that was built for customers.
## How a card-testing attack runs
1. **Get the cards.** The list comes from a breach, a phishing kit, or a criminal marketplace.
For enumeration, the attacker starts from a card range instead.
2. **Find a weak merchant.** Visa's
[merchant guidance on enumeration](https://usa.visa.com/dam/VCOM/global/support-legal/documents/anti-enumeration-and-account-testing-best-practices-merchant.pdf)
names legitimate online merchants with weak fraud controls as the most common target.
Guest checkout and card-saving forms with no limits are attractive.
3. **Script the attempts.** Stripe describes testers using both card setup and payments. They
prefer card setup, because a validation during setup usually does not appear on the
cardholder's statement. Payment-based tests use small amounts that cardholders are less
likely to notice. Visa notes that attack transactions are often under 10 US dollars.
4. **Read the responses.** Approvals, declines, and 3D Secure results tell the script which
cards are live.
5. **Use or sell the results.** Visa says the valid details are sold on underground carding
sites or used for fraud elsewhere, often at another merchant.
Not every attack goes through a shop's front end. Scripts can call a payment API directly
with a leaked or public key, and Visa also describes attackers opening merchant accounts under
false identities or taking over a merchant's gateway login to test cards.
## What it costs the merchant
The cards belong to someone else, but the merchant carries the side effects. Stripe lists:
- disputes on the test charges that succeed;
- a higher decline rate, which can make issuers treat the merchant's legitimate payments as
riskier even after the attack ends;
- extra fees, such as authorization and dispute fees;
- load on infrastructure from the burst of requests;
- test payments that look like new customers in the business data.
The payment networks also watch for it. Under Visa's
[Acquirer Monitoring Program (VAMP)](https://corporate.visa.com/content/dam/VCOM/corporate/visa-perspectives/security-and-trust/documents/visa-acquirer-monitoring-program-fact-sheet-2025.pdf),
in effect since June 2025, Visa monitors enumeration every month alongside fraud and disputes.
Acquirers must keep merchants below an enumeration threshold: 20% of authorization attempts
identified as enumerated, with at least 300,000 such attempts in the month. Visa's merchant
guide puts the losses from enumeration across the payments ecosystem at more than a billion
dollars a year.
## Signs of card testing
Most of the evidence sits in payment data, not in page views. Look for several of these
together:
| Sign | Where you see it | Also consistent with |
| --- | --- | --- |
| A sudden rise in failed or blocked payments | Payment provider dashboard, decline logs | A provider incident or a broken release |
| Many low-value attempts or zero-amount card verifications | Authorization records | A free trial or card-on-file flow working normally |
| Declines for invalid numbers, expired cards or failed CVV, grouped by card range | Decline codes by BIN | A card issuer's own outage |
| Many different cards from one account, session, device, or email | Your order and account data | Families or businesses sharing a device |
| Nonsense customer names and email addresses on small payments | Payment details | Test orders from your own team |
| Reversals sent right after approvals | Authorization and reversal logs | Legitimate order cancellations |
| Payment calls that skip the checkout page | API and server logs | A mobile app or partner integration |
Stripe's guide also flags a rise in requests failing with its `402` card errors and the
`generic_decline` result. It warns that testers vary their techniques, so a firewall rule or
filter based on one heuristic, such as IP address, is usually not enough on its own.
Watch for your own automation too. Stripe notes that aggressive retries of failed subscription
payments can look like card testing when they produce spikes with a low success rate.
## How to stop it
The controls work at two layers: before the request reaches the payment provider, and inside
the payment flow.
- Use the provider's current integration and send it the risk data it asks for. Stripe says
its card-testing protection depends partly on the information the integration supplies,
such as IP address, email, name, and billing address.
- Limit the operations that validate cards. Cap cards added per account and per session, new
accounts per IP address, and attempts per time window. Visa and Stripe both recommend these
limits, and Visa extends them to the add-card and save-card steps.
- Check the client before the payment call. A challenge or bot check has to run on every
request that can validate a card or create a payment, and has to be verified on the server.
A check that only guards the checkout page does nothing against a script that calls the API.
- Require the card checks. Visa recommends CVV2 and address verification on card-not-present
transactions, 3D Secure, and network tokens where available.
- Protect your keys. Stripe warns that testers can reuse a publishable key and that an exposed
secret key lets them create payments directly. Visa suggests rotating API keys during an
attack that targets the API.
- After an attack, refund successful test charges to head off disputes, and do not retry cards
attached to fraudulent customers.
The [carding prevention guide](https://www.centinelanalytica.com/blog/how-to-prevent-carding-attacks) walks through these
controls in order, including retry budgets and how to test changes without real cards.
## What bot detection can and cannot tell you
Bot detection answers a question about the client: was this request sent by a person in a
browser, by a script, or by an automated browser, and does it repeat a workflow across many
sessions? That is useful here, because card testing only pays off at scale, and scale needs
automation.
It does not answer questions about the card. A bot signal does not prove a card is stolen, a
payment is fraudulent, or a cardholder was involved. Those answers come from the issuer and
the payment provider. Legitimate automation reaches payment flows too: subscription billing,
your own monitoring, and partner integrations. And a human-looking browser session does not
prove that a payment is safe.
Keep the two decisions apart. Use automation evidence to decide which requests may reach the
payment step, and let the provider's fraud checks decide the payment. Label findings as
"automated card-verification attempts," not "fraud," until the payment data supports that.
## Where Centinel fits
In supported web checkouts, Centinel evaluates browser and request evidence on checkout and
card routes, so a site can challenge or block scripted attempts before they reach the payment
provider. It does not score cards or transactions and does not replace the provider's fraud
checks. The [checkout and carding use case](https://www.centinelanalytica.com/use-cases/checkout-and-carding-fraud) describes
that boundary, and [How to detect browser automation](https://www.centinelanalytica.com/blog/detect-browser-automation) covers
the evidence involved.
---
Page: https://www.centinelanalytica.com/blog/cloudflare-bot-fight-mode-vs-bot-management
# Cloudflare Bot Fight Mode vs. Bot Management
What Cloudflare's Bot Fight Mode, Super Bot Fight Mode, and Enterprise Bot Management each do, where each one stops, and when to move to the next level.
By Frederick Jahn. Published 2026-09-30, updated 2026-10-11.
Cloudflare sells three bot products that are easy to confuse: Bot Fight Mode, Super Bot Fight
Mode, and Bot Management. They differ in what they detect, what you can configure, and which
plan you need. This guide summarizes each one from Cloudflare's documentation, then covers the
limits that usually push a site from one level to the next.
## Three products, three plan levels
- **Bot Fight Mode** is on the [Free plan](https://developers.cloudflare.com/bots/plans/free/).
- **Super Bot Fight Mode** is included with the
[Pro](https://developers.cloudflare.com/bots/plans/pro/),
[Business, and Enterprise](https://developers.cloudflare.com/bots/plans/biz-and-ent/) plans.
An Enterprise zone without the Bot Management add-on gets the Business version.
- **Bot Management** is a
[paid Enterprise add-on](https://developers.cloudflare.com/bots/plans/bm-subscription/) that
your Cloudflare account team enables. Zones with Bot Management do not see Bot Fight Mode or
Super Bot Fight Mode.
## What Bot Fight Mode does
[Bot Fight Mode](https://developers.cloudflare.com/bots/get-started/bot-fight-mode/) identifies
traffic that matches patterns of known bots and issues computationally expensive challenges to
it. Cloudflare's Free plan page says it detects simple bots from cloud hosting providers and
headless browsers.
There is almost nothing to configure. Bot Fight Mode applies to the whole domain, with no way
to limit it to certain endpoints. You cannot customize it, skip it, or bypass it with WAF custom
rules or Page Rules, because it runs outside the Ruleset Engine. It does not trigger when an IP
Access rule matches first. Challenged requests appear in Security Analytics with the label
"Bot Fight Mode".
## What Super Bot Fight Mode adds
[Super Bot Fight Mode](https://developers.cloudflare.com/bots/get-started/super-bot-fight-mode/)
lets you choose an action per bot category, protect static resources, and see bot analytics. It
runs after your WAF custom rules, so you can exempt traffic with a custom rule that uses the Skip
action.
What you can act on depends on the plan. On Pro, you can allow, block, or challenge the
"Definitely automated" and "Verified bots" groups, and analytics are limited to a Bot Report.
Blocking or challenging "Likely automated" traffic is not available on Pro. Business and
Enterprise add the "Likely automated" group, detection of many sophisticated bots, and dedicated
Bot Analytics. Cloudflare suggests considering Bot Management when the Bot Report shows a
double-digit percentage of automated traffic.
## What Enterprise Bot Management adds
[Bot Management](https://developers.cloudflare.com/bots/get-started/bot-management/) gives every
request a [bot score](https://developers.cloudflare.com/bots/concepts/bot-score/) from 1 to 99. A
score of 1 means automated, 2 to 29 means likely automated, and 30 to 99 means likely human.
Verified bots are a separate group. The detection engines that produce the score (heuristics,
machine learning, and JavaScript Detections) apply only to Bot Management.
You act on the score in WAF custom rules or Workers, so a rule can target a single path, method,
or IP range instead of the whole domain. Bot Management also exposes JA3 and JA4 fingerprints,
bot tags, and detection IDs for rules and logs.
## Side-by-side comparison
| Question | Bot Fight Mode | Super Bot Fight Mode | Bot Management |
| --- | --- | --- | --- |
| Plan | Free | Pro, Business, Enterprise | Enterprise add-on |
| Detects | Simple bots, headless browsers | Pro: simple bots, headless browsers. Business and up: many sophisticated bots | Scored 1 to 99 on every request |
| Action | Computationally expensive challenge | Allow, block, or challenge per bot category | Any WAF custom rule or Worker action |
| Scope | Whole domain | Whole domain, with Skip exceptions | Any rule expression: path, IP, and more |
| Exceptions | IP Access rules only | WAF custom rules with Skip | Written into your own rules |
| Analytics | Security Analytics events | Bot Report (Pro) or Bot Analytics (Business and up) | Bot Analytics, score, and detection IDs |
## The challenge script Bot Fight Mode injects
Bot Fight Mode turns on
[JavaScript Detections](https://developers.cloudflare.com/cloudflare-challenges/challenge-types/javascript-detections/),
and you cannot disable it while Bot Fight Mode is on. Cloudflare injects an invisible JavaScript
snippet into HTML page responses. It is not added to AJAX responses. The script loads from
`/cdn-cgi/challenge-platform/`, and a passing browser receives a `cf_clearance` cookie. A result
lasts 15 minutes, and Cloudflare injects the code again before it expires.
Your Content Security Policy must allow `/cdn-cgi/challenge-platform/`. Cloudflare adds nonces
it parses from the CSP response header, but nonces set in a `` tag are not supported. On
the free and Pro tiers the detection runs, but you cannot enforce on its result: a rule on
`cf.bot_management.js_detection.passed` requires Bot Management. The first request from a new
visitor also carries no JavaScript Detections result yet.
## Limits to plan around
- **Whole-domain scope.** Bot Fight Mode cannot be limited to certain paths, so it applies to
your API and login routes as much as to your marketing pages.
- **API and mobile app traffic.** Cloudflare warns that Bot Fight Mode may challenge API or
mobile app traffic. A native app or a server-to-server client cannot solve a browser
challenge.
- **No bypass.** Because WAF custom rules and Page Rules cannot skip it, the only exception
mechanism is an IP Access rule. That rarely helps with partners or monitoring services whose
IP addresses change.
- **Upgrade path.** To move to Super Bot Fight Mode you have to turn Bot Fight Mode off first.
If legitimate users get stuck repeating a challenge, see
[how to fix bot challenge loops](https://www.centinelanalytica.com/blog/fix-bot-challenge-loops).
## When to move beyond Bot Fight Mode
Move to Super Bot Fight Mode when you need exceptions for APIs, partners, or monitoring, or when
you want to block rather than only challenge. Move to Bot Management when you need per-route
rules, a per-request score, or enforcement on JavaScript Detections.
Some teams instead want to know whether a specialist product catches bots that their Cloudflare
configuration misses. [Centinel vs. Cloudflare](https://www.centinelanalytica.com/compare/centinel-vs-cloudflare)
compares the two, including Centinel's September 2026 benchmark observations, and
[Centinel vs. CDN and WAF bot protection](https://www.centinelanalytica.com/compare/centinel-vs-cdn-bot-protection) explains how to
test whether another layer is worth it. The [bot management overview](https://www.centinelanalytica.com/bot-management) lists
the requirements to check, and the
[bot management software comparison](https://www.centinelanalytica.com/blog/best-bot-management-software) covers other vendors.
---
Page: https://www.centinelanalytica.com/blog/best-bot-management-software
# Best bot management software: 10 tools compared
Ten bot management tools compared by deployment model, client-side signals, mobile and API coverage, and fit. Sourced from each vendor's own documentation.
By Frederick Jahn. Published 2026-09-30, updated 2026-10-11.
No bot management product is best for every site. The right choice depends on where your
traffic already flows, which clients you need to protect (browsers, native apps, APIs), and how
much integration work your team can take on.
This guide compares ten products by what their own documentation says about deployment, signal
collection, and response options. The products are listed alphabetically. The order is not a
ranking. Centinel publishes this guide and is one of the ten products; its section lists its
limits as well as its strengths.
## How to choose bot management software
Use these questions to narrow the list before you talk to any vendor.
- **Where does the decision run?** Some products run inside a CDN you may already use. Others
sit in your web server, load balancer, or application code, or call an external API from one
of those places. A control can only act on traffic that passes through it, so check direct
origin paths and APIs as well as your main hostname.
- **Does it need a script in the page?** Most products collect browser signals with
JavaScript. That affects your Content Security Policy, page weight, and consent review, and
it only covers traffic that loads HTML.
- **Do you need native mobile app coverage?** Some vendors document mobile SDKs; others do not.
An API used only by your app needs a different signal source than a web page.
- **How are declared crawlers verified?** A user-agent string is easy to copy. Ask which
operator sources (IP ranges, reverse DNS, signed requests) the product uses to verify search
engines and AI crawlers.
- **Which responses are available?** Identify, observe, rate-limit, challenge, allow, and block
are different decisions. Check which ones each route supports.
- **What happens on failure?** Ask whether the product fails open or closed when its own
service is slow or unreachable, and what timeout applies.
- **Can you review its errors?** You need reason codes or logs to investigate false positives
and to prove the product is doing what the contract says.
The [bot management overview](https://www.centinelanalytica.com/bot-management) turns these questions into a requirements
checklist.
## Deployment models at a glance
| Product | Where it runs, per vendor documentation | Client-side component |
| --- | --- | --- |
| Akamai Bot Manager | Akamai edge | Not detailed on the public product page |
| Arcjet | Your application code, through an SDK | None required for bot rules |
| Centinel | Validation API called from an edge, web server, application, or CMS integration | Optional browser script |
| Cloudflare Bot Management | Cloudflare network; Enterprise add-on | JavaScript Detections |
| DataDome | Edge, CDN, or server module that calls DataDome | JavaScript Tag; Android and iOS SDKs |
| F5 Distributed Cloud Bot Defense | F5 Distributed Cloud HTTP load balancer, or BIG-IP | JavaScript injected on chosen pages |
| Fastly Bot Management | Fastly network, pre-cache or post-cache (post-cache needs Next-Gen WAF) | JavaScript snippet for advanced detections |
| HUMAN Bot Defender | Enforcer on your CDN, load balancer, or web server; cloud Detector | Sensor JavaScript |
| Imperva Advanced Bot Protection | Not detailed on the public product page | Not detailed on the public product page |
| Kasada | Not detailed publicly; product documentation requires a login | Not detailed publicly |
"Not detailed" means the public source did not describe it. It does not mean the product lacks
it. Ask the vendor.
## Akamai Bot Manager
[Akamai Bot Manager](https://www.akamai.com/products/bot-manager) detects and mitigates bots at
Akamai's edge. Akamai describes a bot score from 0 (human) to 100 (bot) that starts with the
first request, response segments labeled Cautious, Strict, and Aggressive, and actions beyond
allow and block. It includes a directory of known bots and lists mobile app coverage and APIs.
**Fits:** teams whose traffic already runs through Akamai and who want bot decisions made at
the same edge.
**Consider:** the detailed technical documentation sits behind an Akamai Control Center login,
so plan time with the account team to confirm configuration details.
See [Centinel vs. Akamai](https://www.centinelanalytica.com/compare/centinel-vs-akamai) for a direct comparison.
## Arcjet
[Arcjet](https://docs.arcjet.com/bot-protection/concepts) configures bot rules in application
code through an SDK. Rules allow or deny named bots and categories drawn from Arcjet's
open-source list of well-known bots, and allow rules can verify a bot by IP address and reverse
DNS. The [reference documentation](https://docs.arcjet.com/bot-protection/reference) says the
SDK fails open, with a default timeout of 2,000 ms.
**Fits:** developer teams that want bot rules in code, next to the routes they protect, without
a CDN change.
**Consider:** protection covers only the routes that call the SDK, and decisions happen after
the request reaches your application. Arcjet now positions itself more broadly as a runtime
security platform for AI agents.
## Centinel
Centinel is a validation API. Your edge, web server, or application sends request details to
`POST /validate` and receives allow, block, or redirect. The
[validation documentation](https://docs.centinelanalytica.com/install/validation) recommends a
client timeout of 300 to 500 ms and describes the failure behavior: fail open on 5xx and
transport errors, fail closed on 4xx. An
[optional browser script](https://docs.centinelanalytica.com/install/scripts) adds browser and
network signals.
Documented [integrations](https://docs.centinelanalytica.com/platforms) cover Cloudflare
Workers, Fastly VCL and Compute, Akamai EdgeWorkers, AWS CloudFront Lambda@Edge, F5 BIG-IP
iRules, Nginx and OpenResty, Apache, HAProxy, Varnish, Next.js, ASP.NET Core, Azure Functions,
WordPress, and Drupal. The [crawler catalog](https://docs.centinelanalytica.com/concepts/crawler-catalog)
verifies declared crawlers by Web Bot Auth signatures, published IP ranges, and reverse DNS.
**Fits:** teams that want specialist detection of browser automation and scraper tools while
keeping their current CDN or WAF, and teams that need crawler-by-crawler policy.
**Limits:** no native mobile SDK is documented, so app-only APIs rely on server-side signals.
Every protected request adds a validation call, so measure latency in your own path. The
default policy rule enforces nothing; your team has to set the policy. Centinel is not a CDN
or WAF; it runs through the edge or server you already operate.
## Cloudflare Bot Management
Cloudflare sells three levels of bot protection. Bot Fight Mode is free,
[Super Bot Fight Mode](https://developers.cloudflare.com/bots/get-started/super-bot-fight-mode/)
comes with Pro, Business, and Enterprise plans, and
[Bot Management](https://developers.cloudflare.com/bots/get-started/bot-management/) is a paid
Enterprise add-on. Bot Management gives each request a
[bot score](https://developers.cloudflare.com/bots/concepts/bot-score/) from 1 to 99 that you
act on through WAF custom rules or Workers, and exposes JA3 and JA4 fingerprints and detection
IDs.
**Fits:** sites already on Cloudflare that want bot scores inside the same rule engine as their
WAF.
**Consider:** the detection engines behind the bot score apply only to the Enterprise add-on,
not to the free and Pro tiers. The
[Bot Fight Mode vs. Bot Management guide](https://www.centinelanalytica.com/blog/cloudflare-bot-fight-mode-vs-bot-management)
explains the differences, and [Centinel vs. Cloudflare](https://www.centinelanalytica.com/compare/centinel-vs-cloudflare) compares
the two products.
## DataDome
DataDome's [getting started guide](https://docs.datadome.co/docs/getting-started) recommends an
edge integration and lists server-side modules for CDNs and web servers, including Akamai
EdgeWorkers, Fastly, Apache, and HAProxy. Client-side signals come from the
[JavaScript Tag](https://docs.datadome.co/docs/javascript-tag), which DataDome describes as
required for optimal detection, and from Android and iOS SDKs. DataDome also sells Account
Protect and Ad Protect.
**Fits:** teams that need web and native app coverage from one vendor and can deploy a module
at the edge.
**Consider:** server-side modules call DataDome during the request. The
[Fastly module](https://docs.datadome.co/docs/module-fastly), for example, sends a preflight
request to the nearest DataDome endpoint. Check latency and failure settings for your module.
See [Centinel vs. DataDome](https://www.centinelanalytica.com/compare/centinel-vs-datadome) for a direct comparison.
## F5 Distributed Cloud Bot Defense
F5 Distributed Cloud Bot Defense runs on an F5 Distributed Cloud HTTP load balancer or on
BIG-IP through a native module or iApp, per the
[Bot Defense quickstart](https://docs.cloud.f5.com/docs-v2/bot-defense/quickstarts/bot-defense-quickstart).
You define protected endpoints, and F5 injects JavaScript into the pages you choose. Mobile
traffic needs its own configuration, and mitigation actions include a log-only option.
**Fits:** teams already running F5 load balancers or BIG-IP who want protection scoped to
specific endpoints such as login and checkout.
**Consider:** protection is endpoint-by-endpoint, so coverage depends on the list you maintain.
See the [F5 product page](https://www.f5.com/products/distributed-cloud-services/bot-defense) for
the commercial packaging.
## Fastly Bot Management
[Fastly Bot Management](https://docs.fastly.com/products/bot-management) inspects requests
before the cache or, with Next-Gen WAF, after it. Documented features include JA3 and JA4
fingerprinting, verified bots, and client challenges (interactive, non-interactive
proof-of-work, and dynamic). Advanced client-side detections need a JavaScript snippet in your
HTML. Fastly bills it per million requests, separately from delivery and Next-Gen WAF.
**Fits:** sites already on Fastly, especially those using Next-Gen WAF.
**Consider:** the Next-Gen WAF Essential platform is not supported, per the same page. See
[Centinel vs. Fastly](https://www.centinelanalytica.com/compare/centinel-vs-fastly) for a direct comparison.
## HUMAN Bot Defender
[HUMAN Bot Defender](https://docs.humansecurity.com/applications/bd-overview) has three parts: a
JavaScript sensor, a cloud Detector, and an Enforcer that runs inline on your CDN, load
balancer, or web server. HUMAN documents more than forty pre-built integrations and coverage for
web, mobile apps, and APIs. The
[getting started guide](https://docs.humansecurity.com/applications/getting-started-with-bot-defender)
walks through adding the sensor and deploying an Enforcer.
**Fits:** teams that need web, mobile, and API coverage and want enforcement in their own
infrastructure rather than at a specific CDN.
**Consider:** the setup touches both the page and the server path. See
[Centinel vs. HUMAN Security](https://www.centinelanalytica.com/compare/centinel-vs-human-security).
## Imperva Advanced Bot Protection
Imperva, now part of Thales, says
[Advanced Bot Protection](https://www.imperva.com/products/advanced-bot-protection-management/)
protects websites, mobile apps, and APIs and covers all 21 OWASP automated threats. That
coverage statement is Imperva's own claim. The public product page does not describe the
deployment model in detail.
**Fits:** teams already using Imperva application security products.
**Consider:** ask for the integration architecture and the client-side requirements before a
trial. Centinel did not test Imperva in the September 2026 benchmark.
## Kasada
[Kasada](https://www.kasada.io/) lists three products: Bot Defense, Account Intelligence, and AI
Agent Trust, and says Bot Defense works without CAPTCHAs. It covers web and API traffic. Kasada's
technical documentation requires a customer login, so its deployment model is not publicly
documented.
**Fits:** teams that want a bot-focused vendor with account protection and AI agent products
alongside bot defense.
**Consider:** the claims on the public site are Kasada's own. Ask for a proof of concept on your
traffic. See [Centinel vs. Kasada](https://www.centinelanalytica.com/compare/centinel-vs-kasada).
## September 2026 benchmark observations
Centinel tested seven of these products against eight scraping and browser-automation tools in
September 2026: Browser-use, Firecrawl, Browserbase, kernel.sh, Playwright Stealth, Camoufox,
Zyte, and Ulixee Hero. These are Centinel's observations from the tested configurations. Centinel
designed and ran the test, and it is one of the products tested.
| Product | Tools detected, of 8 |
| --- | --- |
| Centinel | 8 |
| DataDome | 5 |
| Kasada | 4 |
| Cloudflare | 3 |
| Akamai | 2 |
| HUMAN Security | 2 |
| Fastly | 0 |
Arcjet, F5, and Imperva were not tested. The result says nothing about plans or configurations
outside the test, about false positives, or about traffic other than these eight tools. The
[benchmark scope](https://www.centinelanalytica.com/compare/centinel-vs-cdn-bot-protection#september-2026-benchmark-scope) explains
the method, and the [per-tool results](https://www.centinelanalytica.com/research/2026-09-bot-detection.csv) are available as a
CSV.
## Run your own proof of concept
Shortlist two or three products that fit your traffic path and client mix, then test them on
your own traffic. Measure detection on the automation that reaches your site, false positives
on real users and trusted crawlers, added latency, and the effort to change a policy. The
[proof-of-concept scorecard](https://www.centinelanalytica.com/blog/bot-protection-proof-of-concept) gives a structure for that
test, and [Centinel vs. CDN and WAF bot protection](https://www.centinelanalytica.com/compare/centinel-vs-cdn-bot-protection)
helps you decide whether a specialist layer adds detection to the CDN you already run.
---
Page: https://www.centinelanalytica.com/blog/pay-per-crawl-explained
# Pay per crawl explained: HTTP 402 and TollBit
Pay per crawl charges AI crawlers per request. How Cloudflare, TollBit, RSL and x402 work, and why it depends on knowing which crawler is asking.
By Frederick Jahn. Published 2026-09-30, updated 2026-10-11.
Pay per crawl is a model in which a website charges an AI crawler for each page it fetches,
instead of only allowing or blocking it. A crawler without a payment arrangement gets an
HTTP `402 Payment Required` response or a paywall; a crawler that has agreed to pay gets the
page, and a marketplace settles the charges. Every version assumes the site already knows which
crawler is asking.
This article describes how the main systems work and where they are still weak. Centinel does
not sell, meter, or broker crawler access. Its part is identifying the crawler, covered at the
end.
## HTTP 402 was reserved for decades
HTTP/1.1 listed 402 in [RFC 2068](https://www.rfc-editor.org/rfc/rfc2068.html) in 1997 as
"reserved for future use," and it was never given a meaning after that. The current HTTP
standard, [RFC 9110](https://www.rfc-editor.org/rfc/rfc9110.html#name-402-payment-required),
still says only: "The 402 (Payment Required) status code is reserved for future use."
That leaves each pay-per-crawl system to define its own headers around the code. The status
code tells a crawler that payment is possible. It does not say how much, in what currency, or
how to pay. Those details are different in each system below.
## How Cloudflare Pay per crawl works
Cloudflare [announced pay per crawl in July 2025](https://blog.cloudflare.com/introducing-pay-per-crawl/).
Its [documentation](https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/what-is-pay-per-crawl/)
describes it as a feature of AI Crawl Control that is in closed beta, with Cloudflare acting as
merchant of record. A site owner sets one price per zone and chooses, crawler by crawler,
whether to allow, charge, or block.
The exchange uses custom headers. When a crawler requests a page it has not agreed to pay for,
Cloudflare returns the price:
```
HTTP/2 402
crawler-price: USD 0.01
```
The crawler can retry with `crawler-exact-price` to accept that price, or send
`crawler-max-price` up front with the most it will pay. If the price fits, the page is served
with a `crawler-charged` header, and Cloudflare records a billing event. According to the
announcement, Cloudflare aggregates those events, charges the crawler, and pays the publisher.
Only a verified crawler can be charged. Cloudflare's
[crawler setup guide](https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-ai-owner/verify-ai-crawler/)
requires Web Bot Auth signatures and registration as a verified bot. Cloudflare's blog calls
preventing anyone from spoofing a paying crawler "an incredibly important technical challenge."
Security rules come first: a block from Cloudflare's WAF or Bot Management overrides the charge
setting. The
[crawler selection guide](https://developers.cloudflare.com/ai-crawl-control/features/pay-per-crawl/use-pay-per-crawl-as-site-owner/select-crawlers-to-charge/)
also warns that charging or blocking search engine crawlers can hurt indexing.
## How TollBit works
TollBit uses a separate gateway instead of a status code on the main site. Its
[architecture documentation](https://docs.tollbit.com/docs/platform-architecture) describes the
setup: the publisher creates a `tollbit` subdomain, and edge logic at the CDN "checks each
request against a simple list of known bot user agents." Matching requests are redirected to
the subdomain.
There, TollBit checks for a valid token. Its [platform documentation](https://tollbit.com/docs)
describes the token as a short-lived signed JWT that the AI company generates with its TollBit
API key. A bot without a token sees a paywall page telling it how to get one. A bot with a
token gets the content, fetched from the publisher, at the
[rates the publisher sets](https://tollbit.com/bot-paywall). TollBit's documentation describes
the redirect being set up in a CDN or in a bot-management product.
## Other approaches
RSL (Really Simple Licensing) is a machine-readable license format, defined in the
[RSL 1.0 specification](https://rslstandard.org/rsl). A site links to its license file from `robots.txt` with a
`License` directive, from HTTP headers, or from the page. The license can state terms such as
pay per crawl or pay per inference, and a server may answer with `401` or `402` when access
depends on a license. The specification also defines a protocol for obtaining a license, and
lets a license name a payment protocol such as x402.
The [x402 protocol](https://docs.x402.org/core-concepts/http-402) is a general scheme for machine payments over HTTP. The server returns `402` with a `PAYMENT-REQUIRED`
header that describes accepted payment schemes and the price, and the client pays and retries
with a `PAYMENT-SIGNATURE` header. It is not specific to crawlers.
| System | How it is enforced | How the crawler is identified | Who settles payment |
| --- | --- | --- | --- |
| Cloudflare Pay per crawl | At Cloudflare's edge, on the site's own URLs | Web Bot Auth signature and verified-bot registration | Cloudflare, as merchant of record |
| TollBit | Redirect to a `tollbit` subdomain | User-agent match for the redirect, TollBit token for access | TollBit |
| RSL | Wherever the site enforces its license | Left to the site or an RSL license server | License server or the named payment protocol |
| x402 | In the server or its middleware | Payment payload, not crawler identity | The payment scheme named in the response |
## The open problems
### A price only applies to crawlers that ask
Every system above works on traffic that identifies itself. A scraper that sends a browser
user agent from a residential address never matches a crawler list, so it never meets the
paywall. Only bot detection can catch it.
### User agents are easy to copy
A redirect or rule keyed on the user-agent string treats every request that uses the name the
same way. That is a real problem at the edge: on the news and media sites in our
[fake crawler report](https://www.centinelanalytica.com/blog/fake-crawler-report), 1 in 7 requests calling themselves
[Googlebot](https://www.centinelanalytica.com/bots/googlebot) on the typical site did not come from Google, and on the typical site 84% of
requests using the [GPTBot](https://www.centinelanalytica.com/bots/gptbot) name were fake. A fake sent to a paywall does little harm. The
same fake under a rule that allows crawlers by name gets the content for free.
### Signed identity is new and not universal
[Web Bot Auth](https://datatracker.ietf.org/doc/draft-ietf-webbotauth-httpsig-protocol/) builds
on [HTTP Message Signatures (RFC 9421)](https://www.rfc-editor.org/rfc/rfc9421.html). A crawler
signs each request with a private key and publishes the public key in a directory. The IETF
working group published the first version of its protocol draft on 1 September 2026, so it is
not yet a finished standard. Crawlers that do not sign fall back to the older checks: the
operator's published IP ranges or a reverse and forward DNS lookup.
A valid signature proves that the holder of a known key signed the request. It does not tell
the site how the crawler will use the content. The verifier also has to enforce the signature's
creation and expiry times to stop replays, and has to decide which key directories it trusts.
### Search crawlers are a separate decision
Charging or blocking a search crawler can remove pages from search results, as Cloudflare's own
documentation warns. Google also separates crawling from use: the
[Google-Extended robots.txt token](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers#google-extended)
controls whether content Google crawls may be used for Gemini training and grounding, but it
has no user agent of its own. Crawling is done with Google's existing user agents, so a price
on a user agent cannot separate those uses.
### There is no common protocol yet
Cloudflare uses `crawler-price` headers, x402 uses `PAYMENT-REQUIRED`, TollBit uses a redirect
and a token, and RSL points to a license file. A crawler has to implement each one it wants to
pay through. Until 402 gets a standard meaning, a site that changes provider changes protocol
too.
## Where crawler verification fits
Allowing, blocking, rate-limiting, and charging all need the same input: an identity that has
been checked. Centinel identifies automated traffic and verifies declared crawlers with each
operator's documented method: a signed request where the operator supports it, published IP
ranges, or reverse and forward DNS. It also detects automation that does not declare itself.
Centinel does not set prices, process payments, or negotiate access.
To check one address yourself, use the [crawler verification tool](https://www.centinelanalytica.com/verify-crawler). The
[AI agent verification guide](https://www.centinelanalytica.com/blog/how-to-verify-ai-agents) covers each operator's method,
and [How to control AI agent access](https://www.centinelanalytica.com/blog/how-to-control-ai-agent-access) covers setting
different responses by identity. For the scraping side of the problem, see
[scraping and AI crawler protection](https://www.centinelanalytica.com/use-cases/scraping-and-ai-crawlers).
---
Page: https://www.centinelanalytica.com/blog/fake-googlebot
# Fake Googlebot: how to spot an impersonator
Any client can send a Googlebot user agent. Verify the IP with reverse and forward DNS or Google's published ranges, and avoid the Google Cloud trap.
By Frederick Jahn. Published 2026-09-30.
A request that says Googlebot in its user agent is only claiming to be Googlebot. The header
is text the client chooses, and Google's
[crawler documentation](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers)
warns that the user agent can be spoofed. The claim becomes an identity only when the address
the request came from checks out against what Google publishes: a reverse DNS name under a
Google domain that resolves back to the same address, or a match in Google's published IP
range files.
Fakes are common. On the news and media sites in our
[fake crawler report](https://www.centinelanalytica.com/blog/fake-crawler-report), 1 in 7 requests calling itself Googlebot
on the typical site did not come from Google. To check a single address now, paste it into the
[crawler IP checker](https://www.centinelanalytica.com/verify-crawler). The rest of this guide shows how to run the same checks
yourself and where they go wrong.
## Why a fake Googlebot is worth catching
Sites treat Googlebot differently from other clients. Rules often let it past rate limits,
paywalls, login walls and bot challenges, because blocking the real crawler costs search
visibility. That exception is what an impersonator is after. A scraper that sends the
Googlebot user agent inherits whatever the site grants the name.
In the report's data, the fakes looked like scrapers rather than broken crawlers:
- 88% of fake Googlebot requests came from datacenter addresses, about a third of them from a
single European hosting provider.
- 95% of fake Googlebot requests asked for pages rather than images, scripts or stylesheets,
against 85% for the real Googlebot.
- 94% of fake Googlebot requests claimed the real Googlebot's current version string, so the
header itself gives nothing away.
Those figures cover news and media sites from late June to 21 September 2026. Each is a share,
not a count, and each comes from at least five sites. Other kinds of site will see different
proportions, but the mechanism is the same anywhere a rule trusts the name.
## Verify an IP with reverse and forward DNS
Google's
[verification guide](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests)
describes the manual check. It takes two lookups.
First, a reverse lookup of the address from your log:
```
$ host 66.249.66.1
1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com.
```
Second, a forward lookup of the name you got back:
```
$ host crawl-66-249-66-1.googlebot.com
crawl-66-249-66-1.googlebot.com has address 66.249.66.1
```
The request is from Google when both hold: the name ends in `googlebot.com` or `google.com`,
and the forward lookup returns the address you started with. The second lookup is the one that
matters. Whoever controls an address can set its reverse record to any name, including
`crawl-1-2-3-4.googlebot.com`. Only Google can make a `googlebot.com` name resolve forward to
that address.
A fake usually fails at the first step:
```
$ host 203.0.113.7
Host 7.113.0.203.in-addr.arpa. not found: 3(NXDOMAIN)
```
No reverse record, or a name under someone else's domain, means the address is not a Google
crawler, whatever the user agent says.
## The googleusercontent.com trap
Google's guide lists a third domain, `googleusercontent.com`. Accepting it as proof of
Googlebot is a mistake.
Google Cloud gives a virtual machine's external address a reverse DNS name under
`googleusercontent.com` unless the customer sets their own, as the
[Compute Engine PTR documentation](https://cloud.google.com/compute/docs/instances/create-ptr-record)
describes. Those names forward-confirm. An address on a rented machine resolves to a name such
as `.bc.googleusercontent.com`, and that name resolves back to the same address. Both
lookups pass, and the request came from whoever rented the machine.
That matters for fake crawlers in particular. At least half of the fake [GPTBot](https://www.centinelanalytica.com/bots/gptbot), [ClaudeBot](https://www.centinelanalytica.com/bots/claudebot),
[PerplexityBot](https://www.centinelanalytica.com/bots/perplexitybot) and [ChatGPT-User](https://www.centinelanalytica.com/bots/chatgpt-user) requests in the report came from Google Cloud. A check that
accepts `googleusercontent.com` would have called many of them Google.
Google uses that domain for some of its
[user-triggered fetchers](https://developers.google.com/static/crawling/ipranges/user-triggered-fetchers.json),
such as tools that fetch a page because a person asked for it. Those are not Googlebot. To
verify one, match the address against that range file rather than trusting the domain.
## Match against Google's published IP ranges
Reverse DNS costs two lookups per address. For log analysis or a firewall rule, Google
publishes the same information as JSON files, each a list of IPv4 and IPv6 prefixes:
- [common-crawlers.json](https://developers.google.com/static/crawling/ipranges/common-crawlers.json)
covers Googlebot and the other common crawlers.
- [special-crawlers.json](https://developers.google.com/static/crawling/ipranges/special-crawlers.json)
covers the special-case crawlers described below.
- user-triggered-fetchers.json, linked above, covers fetchers acting on a user's request.
Each file carries a `creationTime`, and Google changes the ranges over time. Fetch them on a
schedule, not once. An address inside a prefix in the right file is from that group of Google
crawlers. An address outside all of them is not, and a user agent naming Googlebot on that
request is false.
The files and the DNS check agree in practice. In the report's data, 99.99% of requests that
passed verification were inside the operators' published ranges, and 98% of requests that
failed were outside them.
## Special-case crawlers verify differently
Google's
[special-case crawlers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-special-case-crawlers),
such as [AdsBot-Google](https://www.centinelanalytica.com/bots/adsbot-google) and [Mediapartners-Google](https://www.centinelanalytica.com/bots/mediapartners-google), work for a specific Google product where the
site and the product have an agreement about the access. Google says they may ignore
robots.txt rules, and AdsBot ignores the
global `User-agent: *` group, so only a rule that names it applies.
Their reverse DNS names end in `google.com`, in the form
`rate-limited-proxy-66-249-90-77.google.com`, not `googlebot.com`. Their addresses are in
special-crawlers.json, not common-crawlers.json. A check that only accepts `googlebot.com`, or
only loads the common-crawlers file, will reject real AdsBot traffic.
## What to do with a fake
A request that fails the check is not Google. Treat it like any other unknown client: apply
your normal rate limits, challenges and access rules, without the exceptions you grant to
search crawlers. Blocking it by the user agent alone would also block the real Googlebot, so
key the exception on verified identity instead.
For the real crawler, verification tells you who sent the request, not whether you want it.
The [Googlebot profile](https://www.centinelanalytica.com/bots/googlebot) lists its robots.txt token, what blocking it does and
its verification details. For other operators and a broader method, see
[how to verify AI agents](https://www.centinelanalytica.com/blog/how-to-verify-ai-agents).
---
Page: https://www.centinelanalytica.com/blog/what-is-ticket-scalping
# What is ticket scalping? How scalper bots work
Ticket scalping is buying limited tickets to resell at a markup, today mostly by bot. How scalper bots beat purchase limits, and what the BOTS Act prohibits.
By Frederick Jahn. Published 2026-09-24, updated 2026-10-11.
Ticket scalping is buying tickets to an event likely to sell out and reselling them above face
value. Online, it is mostly done with bots: software that joins the on-sale faster than fans can
and buys across many accounts to get around purchase limits. In the US, using bots to evade a
ticket seller's controls is illegal under the BOTS Act, even where resale itself is allowed.
[OWASP's automated threat entry for scalping (OAT-005)](https://github.com/OWASP/www-project-automated-threats-to-web-applications/blob/master/assets/oats/EN/OAT-005_Scalping.md) defines it as obtaining "limited-availability and/or preferred goods/services by unfair methods." It notes that scalping is "most well known in the ticketing business," and lists ticket scalping, ticket touting, queue jumping, and sneaker bot among its other names.
This article explains how scalper bots work and what the law says about them. If you run ticketing and want the route-by-route controls, see [bot protection for ticketing and live events](https://www.centinelanalytica.com/industries/ticketing-and-live-events).
## How a scalper bot buys tickets
A scalper operation automates the steps a fan takes by hand, and repeats them many times at once:
1. **Watch.** The bot monitors the event page so it can act the second inventory is released.
2. **Get in line many times.** It opens many sessions, each with its own account, browser profile, and network address, so the operator holds many places in a queue instead of one.
3. **Select and hold.** Each session picks seats and adds them to a cart faster than a person can read the seat map.
4. **Pay.** Checkout runs with card details spread across the accounts, so the purchases do not look linked to one buyer.
5. **Relist.** The tickets go up on a resale marketplace at a markup.
The FTC's descriptions of real cases show the same pattern. In its [first cases under the BOTS Act in 2021](https://www.ftc.gov/news-events/news/press-releases/2021/01/ftc-brings-first-ever-cases-under-bots-act), the agency said three brokers used software to search for and reserve tickets automatically, software to conceal their IP addresses, and hundreds of fictitious Ticketmaster accounts and credit cards to get around posted ticket limits. In a [2025 complaint](https://www.ftc.gov/news-events/news/press-releases/2025/08/ftc-takes-action-against-ticket-resellers-using-illegal-tactics-bypass-ticket-limit-protections), the FTC alleges a broker operation used thousands of accounts, thousands of card numbers, proxy or spoofed IP addresses, and SIM boxes to receive verification codes for those accounts.
## Scalping vs. seat spinning
Some bots never buy at all. Seat spinning holds seats in carts over and over so they can't be sold to anyone else, then lets them go. OWASP files this under [OAT-021 Denial of Inventory](https://github.com/OWASP/www-project-automated-threats-to-web-applications/blob/master/assets/oats/EN/OAT-021_Denial_of_Inventory.md). It lists seat spinning, inventory hoarding, and hold-all attack among the other names, and it names queue positions as one of the things that can be exhausted.
| Compared on | Scalping (OAT-005) | Seat spinning (OAT-021) |
| --- | --- | --- |
| Does the bot pay? | Yes | No |
| What it takes from fans | The tickets | The chance to buy them while the holds last |
| Where it shows up | Checkout and payment | Seat selection, cart, and hold routes |
| Typical goal | Resale at a markup | Pushing buyers toward resale, or holding stock for later |
The two often run together. A hold buys the scalper time to pay with another account.
## Is ticket scalping illegal?
Reselling a ticket is legal in many places. Using bots to get around a ticket seller's controls is not, in the US. According to the [FTC's summary of the Better Online Ticket Sales (BOTS) Act](https://www.ftc.gov/legal-library/browse/statutes/better-online-ticket-sales-act), the law "prohibits the circumvention of a security measure, access control system, or other technological control measure used online by a ticket issuer." It also prohibits selling tickets obtained that way if the seller took part in, could control, or should have known about the violation. It covers concerts, theater, sporting events, and similar events at venues with more than 200 seats.
The FTC's [2025 refresher on the Act](https://www.ftc.gov/business-guidance/blog/2025/04/bots-act-compliance-time-refresher) gives two plain examples of what it makes illegal: bypassing maximum ticket purchase limits "through technological means or by buying tickets under fake identities."
Enforcement has picked up. A
[March 2025 executive order](https://www.whitehouse.gov/presidential-actions/2025/03/combating-unfair-practices-in-the-live-entertainment-market/)
directs the FTC to "rigorously enforce" the BOTS Act. In September 2025 the FTC and seven
states [sued Live Nation and Ticketmaster](https://www.ftc.gov/news-events/news/press-releases/2025/09/ftc-sues-live-nation-ticketmaster-engaging-illegal-ticket-resale-tactics-deceiving-artists-consumers).
The complaint alleges that brokers routinely bypassed Ticketmaster's limits "by creating
thousands of Ticketmaster accounts and using proxy IP addresses," and that an internal review
found five brokers controlling 6,345 accounts. These are allegations, and the court will decide
the case.
State laws on resale and price caps vary, and this is not legal advice. The point for a ticket seller is narrower: the law is aimed at people who get around your controls, so those controls have to exist and work at the moment of the on-sale.
## Why purchase limits and CAPTCHAs are not enough
A limit of four tickets per account stops one account. The operators in the FTC's cases got around that with many accounts and many cards. A limit per card or per address runs into proxies and virtual card numbers.
A CAPTCHA at checkout adds a step for every fan on the busiest minute of the year. Paid CAPTCHA-solving services exist, so the extra step can cost fans more than it costs an operator. See [CAPTCHA alternatives](https://www.centinelanalytica.com/blog/captcha-alternatives) for how teams move checks away from the visible puzzle.
A waiting room controls how many people reach the store at once. It doesn't decide which of them are scripts. If a bot operator holds a thousand places in line, the queue admits a thousand bots in fair order.
## What works better
The useful question on an on-sale is how each request was made, not which account or card it came with. A few practices follow from that:
- **Check the session before the scarce step.** Evaluate browser and request evidence on the event page, seat selection, and checkout, and challenge or block sessions with weak evidence there. [How to detect browser automation](https://www.centinelanalytica.com/blog/detect-browser-automation) covers what that evidence looks like.
- **Give holds their own limits.** Seat-hold and add-to-cart routes need a separate policy from browsing, so repeated holds from one automated workflow are slowed before they take inventory off sale.
- **Treat account creation as part of the on-sale.** A spike in new accounts in the days before a big release is often how operators prepare. [Detecting automated account creation](https://www.centinelanalytica.com/blog/detect-automated-account-creation) walks through that review.
- **Protect accounts that already hold tickets.** Existing fan accounts get taken over with [credential stuffing](https://www.centinelanalytica.com/blog/detect-credential-stuffing) and used for purchases or transfers.
- **Measure fans, not only blocks.** Track how many real buyers complete checkout and how many hit a challenge. A drop in blocked requests means little if fans still lose out.
None of this replaces the ticketing system's own rules. Purchase limits, hold expiry, and transfer rules stay where they are; bot evidence decides which sessions reach them.
## Where Centinel fits
Centinel evaluates how each request was made, so a ticketing platform can apply a different response to scripted buying than to fans. It is not a waiting room, it does not set purchase limits, and it does not decide whether a buyer is a scalper by identity. The [ticketing and live events page](https://www.centinelanalytica.com/industries/ticketing-and-live-events) describes where it sits on each route.
---
Page: https://www.centinelanalytica.com/blog/what-is-price-scraping
# What is price scraping? How to spot it in logs
Price scraping is the automated collection of prices from product, listing, or fare pages. See who runs it, how it shows up in logs, and which responses fit.
By Frederick Jahn. Published 2026-09-24, updated 2026-10-11.
Price scraping is the automated collection of prices, and usually the availability and listing details around them, from product, search, listing, or fare pages. Competitors, aggregators, and resellers run it to track what you charge and react faster than they could by hand, often through rotating proxies so each request looks like a new shopper.
It is a narrow case of general web scraping. [OWASP's automated threat entry for scraping (OAT-011)](https://github.com/OWASP/www-project-automated-threats-to-web-applications/blob/master/assets/oats/EN/OAT-011_Scraping.md) describes collecting accessible data "for subsequent use such as exploitation of proprietary data, or price-fixing" and lists "comparative shopping" and "bargain hunting" among its other names. The same entry notes that scraping can be ongoing or periodic, and that it may use accounts or need none at all.
This article is about recognizing price scraping and choosing a response. The route-by-route control plan lives in [how to prevent web scraping](https://www.centinelanalytica.com/blog/how-to-prevent-web-scraping).
## Who scrapes prices, and why
The people behind price collection usually want one of a few things:
- **A competing store or marketplace** wants to match or undercut your prices, or to see when you run out of stock.
- **An aggregator or comparison site** wants your catalog and prices without a feed agreement.
- **A reseller** watches for items priced below what they can sell them for elsewhere, and needs to see the change before other buyers do.
- **A travel metasearch or booking reseller** wants fares and availability, which change often and are costly to generate on your side.
What they have in common is a need for freshness. A price that is a day old is worth much less than one from ten minutes ago, so price scrapers revisit the same items on a schedule. That repeat pattern is often the most useful evidence you have.
## Not all price collection is unwanted
Plenty of automated price reading is expected. Google's Shopping crawler, [Storebot-Google](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers), crawls pages for Google Shopping surfaces. Comparison partners may pull prices under an agreement. Your own pricing team may run a monitoring tool against your site to check that published prices match the catalog.
So the question is not "is this a bot reading prices?" The questions are who the client claims to be, whether that claim holds up, and whether your policy allows that client to collect at that rate.
A user-agent string settles none of that. Google itself warns that the [user agent can be spoofed](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers) and documents how to [verify a Google crawler](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests): a reverse DNS lookup on the source IP, a check that the hostname belongs to `googlebot.com`, `google.com`, or `googleusercontent.com`, and a forward lookup that must return the original IP. Google also publishes IP range files for automated matching. A request that says [Googlebot](https://www.centinelanalytica.com/bots/googlebot) but fails that check is not Googlebot.
Robots.txt does not decide this either. [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309) states that robots.txt rules "are not a form of access authorization." A scraper that ignores the file faces nothing from it.
## What price scraping looks like in logs
The example below is illustrative. The paths, parameters, and addresses are invented for this article and are not Centinel data.
Suppose a store serves its category page as HTML and loads prices from a JSON endpoint behind it. An ordinary visit fetches the page, its scripts and images, then the price API with a session cookie, and sometimes moves on to a product page or the cart. The requests below arrive at the price API during one hour:
```
203.0.113.14 GET /api/v2/listings?cat=headphones&sort=price_asc&page=1 200 no cookie
198.51.100.72 GET /api/v2/listings?cat=headphones&sort=price_asc&page=2 200 no cookie
192.0.2.201 GET /api/v2/listings?cat=headphones&sort=price_asc&page=3 200 no cookie
203.0.113.90 GET /api/v2/listings?cat=headphones&sort=price_asc&page=4 200 no cookie
...
198.51.100.9 GET /api/v2/listings?cat=headphones&sort=price_asc&page=38 200 no cookie
```
No single address sends more than a handful of requests, so a per-IP rate limit never triggers. Looked at together, the requests tell a clearer story:
| Observation | Why it matters | What else could explain it |
| --- | --- | --- |
| Pages 1 to 38 are covered in order, with the same sort and filter | A person browsing rarely reads every page of a category | A partner integration or your own monitoring job |
| Price API calls with no matching page load or asset requests | The client may be calling the data endpoint directly | Your CDN may serve assets, so the origin never sees them |
| No session cookie carried between requests | The client is not keeping browser state | Privacy settings, or a first visit on each request |
| The same sequence returns every six hours | Scheduled revisits to keep prices fresh | A cache-warming or feed job you forgot about |
| Addresses spread across many networks with one shared request pattern | Proxy rotation to spread volume | Many real shoppers following the same popular link |
Each row is a lead, not proof. Put them together over a long enough window and you have a reasonable case for a closer look. Group the requests by pattern (endpoint, parameters, header set, timing) rather than by IP. [Detecting low-and-slow scraping](https://www.centinelanalytica.com/blog/detect-low-and-slow-scraping) covers how to choose that window and how to avoid turning a weak grouping key into a named person.
Before acting, rule out your own traffic. Check the addresses and patterns against your monitoring tools, feed partners, and any verified crawler you intend to allow.
## Choosing a response
Public prices are visible to anyone who loads the page. No control makes them secret, and a determined collector can fall back to slower, more human-looking collection. A realistic goal is narrower: keep scraping from getting the freshest data at the lowest cost, protect your origin from the load, and keep approved clients working. Where the line sits is your decision, and so is the choice to allow some collection.
**Observe.** Start by recording which clients a proposed rule would affect, without changing their responses. This is how you find the partner integration or internal job before you break it.
**Rate-limit.** Limit the price endpoint by what it returns, not only by request count per IP. A budget for distinct items or pages per session, credential, or request pattern over hours catches coverage that a per-minute limit misses. [OWASP's bot management cheat sheet](https://cheatsheetseries.owasp.org/cheatsheets/Bot_Management_and_Anti-Automation_Cheat_Sheet.html) recommends applying rate limits "at multiple keys, not just IP," and notes that per-IP limits are defeated by residential proxy networks.
**Serve differently.** Some sites give suspected automation a slower or less fresh response, such as cached prices or a challenge before the price API answers. The same OWASP cheat sheet goes further and suggests returning "plausible but slightly wrong data (price ±1%, fake stock counts)" to scrapers. That carries a real cost: a misclassified shopper sees a price you will not honor. Talk to whoever owns pricing and customer terms before trying it.
**Offer a sanctioned path.** If aggregators want your prices and you are willing to share them, a feed or partner API with its own credentials and limits gives you control over what they get. It also makes the remaining unapproved collection easier to separate out.
**Block.** Blocking fits clients that fail identity checks, such as a [fake Googlebot](https://www.centinelanalytica.com/blog/fake-googlebot), and clients that keep collecting after softer responses. Keep verified search crawlers and named partners on their own allow policy so a block rule cannot reach them.
Whatever you choose, detection does not tell you the operator's intent or whether their collection is lawful. It tells you how the request was made and whether it matches the policy you set for that route.
## Where Centinel fits
Centinel evaluates how a request was made. In supported browser deployments, it combines request and browser evidence on search and item routes instead of relying on one address or header. It can compare a declared crawler identity with the verification data the operator publishes, so a copied crawler name gets no automatic trust. Your integration then applies the response configured for that route, such as allow, rate-limit, challenge, or block.
One limit matters for price scraping in particular. A client that calls your JSON endpoint directly does not run your page, so it supplies no browser evidence. Decisions on that path rely on request evidence and on whether the endpoint expects a browser session at all.
The [retail and ecommerce industry page](https://www.centinelanalytica.com/industries/retail-and-ecommerce) covers catalog scraping alongside checkout and login abuse, and the [marketplace industry page](https://www.centinelanalytica.com/industries/marketplaces) shows how this applies to search, item, sign-up, and messaging routes. The [scraping and AI crawler use case](https://www.centinelanalytica.com/use-cases/scraping-and-ai-crawlers) covers crawler verification and per-client policy.
---
Page: https://www.centinelanalytica.com/blog/what-is-stagehand
# What is Stagehand? The AI browser automation SDK
Stagehand is Browserbase's open-source SDK for driving Chromium with natural-language steps. How it works, where it runs, and what a target site can see.
By Frederick Jahn. Published 2026-09-24, updated 2026-09-30.
Stagehand is an open-source SDK from Browserbase for automating a web browser with a mix of ordinary code and natural-language instructions. A developer can write `page.goto()` for a step with a stable selector and `act("click the sign-in button")` for a step where a language model should find the target.
This article is about the software library, not the theater job. It covers what Stagehand does, where its browser runs, and what that means for a site that receives its traffic.
## What Stagehand provides
The [Stagehand repository](https://github.com/browserbase/stagehand) is published under the MIT license. The [current documentation](https://docs.stagehand.dev/v4/first-steps/introduction) describes version 4, with matching SDKs for TypeScript, Python, and Go.
The library has two layers. The first is a set of Playwright-style page methods such as `goto`, `click`, `type`, and `locator`. The second is three AI primitives:
- **act** performs one action on the page. According to the [act documentation](https://docs.stagehand.dev/v4/basics/act), passing an instruction runs model inference to find the target, while passing a previously observed action replays it with no inference. The docs recommend chaining single-step calls rather than writing multi-step instructions.
- **observe** returns candidate actions for the current page, each with a selector, description, and method, as described in the [observe documentation](https://docs.stagehand.dev/v4/basics/observe). A developer can inspect one and hand it back to `act`.
- **extract** pulls structured data from a page and validates it against a schema the developer defines, per the [extract documentation](https://docs.stagehand.dev/v4/basics/extract).
Earlier versions also had an `agent()` method that looped over these tools and let a model choose each step. The [v3 to v4 migration guide](https://docs.stagehand.dev/v4/migrations/v3) says `agent()` is gone in v4. Browserbase now recommends that a coding assistant write a Stagehand script once, or that developers plug Stagehand into their own tool-calling loop. The introduction describes Stagehand as "the hands" and the developer's agent framework as "the brain."
## How the language model fits in
Stagehand does not require one model provider. The [models documentation](https://docs.stagehand.dev/v4/configuration/models) lists OpenAI, Anthropic, and Google as supported providers and also accepts a client-side callback for a custom model. When the browser runs on Browserbase and no model is set, Browserbase's Model Gateway picks a model for each call.
Caching changes how often a model is involved. The [caching guide](https://docs.stagehand.dev/v4/best-practices/caching) says Browserbase caches `act`, `observe`, and `extract` results on its servers. A repeated call with the same instruction, page content, and options returns without inference, and a cached `act` replays the recorded selector. This cache only exists for Browserbase-hosted browsers; the same page says local browsers cannot use it.
For a defender, this matters because a mature Stagehand job may call a model rarely or never. Its per-step timing can look like a plain script rather than an agent pausing to think.
## Where the browser runs
The [browser configuration page](https://docs.stagehand.dev/v4/configuration/browser) documents three options:
- launch a cloud browser on Browserbase;
- launch a local Chromium browser on the developer's machine;
- attach over the Chrome DevTools Protocol (CDP) to a Chromium browser that is already running.
Stagehand drives the browser over CDP and, according to its introduction, has no Playwright or Puppeteer dependency. In v4 its runtime runs inside the browser through an extension. For local launches, the documentation lists the Chrome arguments Stagehand adds by default, including `--enable-unsafe-extension-debugging` so the SDK can attach to that extension's service worker. Those are launch settings on the client, not headers a website receives.
## Stealth, proxies, and CAPTCHAs belong to Browserbase
Stagehand's own docs point to Browserbase for identity and network features. The browser configuration page says the Browserbase environment offers "stealth mode, proxy support, and persistent contexts," and shows `proxies`, `solveCaptchas`, and a `verified` setting passed through `browserbase.launch()`.
Those features are documented by Browserbase, not implemented in the Stagehand library:
- [Browserbase proxies](https://docs.browserbase.com/platform/identity/proxies) can route a session through managed residential proxies or a customer's own HTTP proxies.
- [Browserbase CAPTCHA solving](https://docs.browserbase.com/platform/identity/captcha-solving) is enabled by default for sessions.
- [Browserbase Verified sessions](https://docs.browserbase.com/platform/identity/verified-customization) use a purpose-built browser with fingerprints that Browserbase says are recognized by its bot protection partners.
A local Stagehand run gets none of these by default. It uses whatever browser, profile, and network the operator provides. The docs also note that authenticated proxies are not yet supported for local browsers.
## What a target site can observe
A site does not receive a "Stagehand" label. It sees a Chromium browser, a network path, and a sequence of requests and interactions. That picture changes with the deployment:
- **On Browserbase:** the session carries whatever proxy, CAPTCHA, and identity settings the operator chose. In our September 2026 benchmark, Centinel detected Browserbase. That result covers Browserbase sessions; Stagehand itself was not part of the benchmark.
- **Locally or over CDP:** the traffic comes from the operator's own machine or infrastructure, with their browser build and network. Nothing from the Browserbase result applies to these runs.
In both cases, detection works from evidence rather than the tool name. Check whether the browser's claims, the connection, and the request headers agree. Look at how the session moves through a flow and whether the same task pattern repeats across different IPs or identities. [How to detect browser automation](https://www.centinelanalytica.com/blog/detect-browser-automation) describes that evidence model in more detail, and [What is Browserbase?](https://www.centinelanalytica.com/blog/what-is-browserbase) covers the hosted side.
## Deciding what Stagehand traffic may do
The same SDK can run a test suite, fill a form for a user, or extract data at scale. Detecting it answers one question: is this client automated? Whether it may continue depends on what it is doing.
Set the rule by resource. Automated reads of a public page need a different response from automated logins, checkout attempts, or bulk extraction of listings. Where an agent operator has agreed to identify itself, [verify the agent](https://www.centinelanalytica.com/blog/how-to-verify-ai-agents) and give it an allow path. For the rest, start by observing which routes the traffic reaches, then apply the least disruptive control that stops the abuse. [Centinel bot management](https://www.centinelanalytica.com/bot-management) describes how that route-level policy works for automated browser sessions on your own site.
## Sources
- [Stagehand repository and license](https://github.com/browserbase/stagehand)
- [Stagehand v4: Introduction](https://docs.stagehand.dev/v4/first-steps/introduction)
- [Stagehand v4: Act](https://docs.stagehand.dev/v4/basics/act)
- [Stagehand v4: Observe](https://docs.stagehand.dev/v4/basics/observe)
- [Stagehand v4: Extract](https://docs.stagehand.dev/v4/basics/extract)
- [Stagehand v4: Models](https://docs.stagehand.dev/v4/configuration/models)
- [Stagehand v4: Caching actions](https://docs.stagehand.dev/v4/best-practices/caching)
- [Stagehand v4: Browser configuration](https://docs.stagehand.dev/v4/configuration/browser)
- [Stagehand: Migrate v3 to v4](https://docs.stagehand.dev/v4/migrations/v3)
- [Browserbase: Proxies](https://docs.browserbase.com/platform/identity/proxies)
- [Browserbase: CAPTCHA solving](https://docs.browserbase.com/platform/identity/captcha-solving)
- [Browserbase: Verified](https://docs.browserbase.com/platform/identity/verified-customization)
---
Page: https://www.centinelanalytica.com/blog/what-is-nodriver
# What is nodriver? Python CDP automation explained
nodriver is an async Python library that drives Chrome over the DevTools Protocol without Selenium or WebDriver. What it changes and what gives it away.
By Frederick Jahn. Published 2026-09-24.
nodriver is an open-source, asynchronous Python library that automates Chrome and other Chromium-based browsers by talking to them directly over the Chrome DevTools Protocol (CDP). It does not use Selenium or the chromedriver binary, and its maintainer presents it as the successor to undetected-chromedriver.
## How nodriver controls the browser
Most Python browser automation has gone through Selenium. Selenium sends WebDriver commands to a driver program such as chromedriver, which then controls the browser. The [nodriver README](https://github.com/ultrafunkamsterdam/nodriver) removes that middle layer: its feature list opens with "No chromedriver binary or Selenium dependency" and the tagline "No more webdriver, no more selenium."
Instead, nodriver launches a locally installed browser and connects to its DevTools endpoint. The [CDP documentation](https://chromedevtools.github.io/devtools-protocol/) describes that endpoint: a browser started with remote debugging enabled exposes a WebSocket URL at `/json/version`, and a client sends protocol commands over it. The [nodriver source](https://github.com/ultrafunkamsterdam/nodriver/blob/main/nodriver/core/browser.py) reads that WebSocket URL and holds the connection itself. The README says the project works with Chromium, Chrome, Edge, and Brave, and lets users call any CDP domain, method, or event directly.
A few other design choices are documented:
- **Async API.** The README calls the library "fully asynchronous," unlike undetected-chromedriver. Scripts are written with `async` and `await`.
- **Fresh profiles.** By default each run [creates a temporary profile directory](https://github.com/ultrafunkamsterdam/nodriver/blob/main/nodriver/core/config.py) and removes it on exit, unless the user points it at an existing profile.
- **Launch defaults.** The same config file passes a fixed set of Chrome command-line flags, such as skipping the first-run screen, and adds `--headless=new` when headless mode is requested.
- **Convenience helpers.** Element lookup by CSS selector, XPath, or visible text, including inside iframes, plus cookie saving and loading so a login does not have to be repeated.
The package is published on [PyPI as nodriver](https://pypi.org/project/nodriver/), requires Python 3.9 or later, and is licensed under the [GNU Affero General Public License v3.0](https://github.com/ultrafunkamsterdam/nodriver/blob/main/LICENSE.txt). The current release at the time of writing is 0.50.3.
## What its documentation claims
The [nodriver documentation](https://ultrafunkamsterdam.github.io/nodriver/) is open about the goal. It says the package is optimized "to stay undetected for most anti-bot solutions" and that talking to the browser directly gives better resistance against web application firewalls. Those are the maintainer's own claims. The project does not publish a test method or results that would let a reader check them.
The README also documents where its own features raise the risk of detection. An optional expert mode disables web security and origin trials and forces shadow roots open, and the README warns that this "makes you more detectable." It also documents a helper built to find and click a verification checkbox. That tells a defender the tool is designed to interact with challenge pages, not only to read public content.
## nodriver and undetected-chromedriver
undetected-chromedriver is the same author's earlier project. Its [README](https://github.com/ultrafunkamsterdam/undetected-chromedriver) describes it as an "optimized Selenium Chromedriver patch": it downloads chromedriver, patches it, and keeps the Selenium programming model. A 3.4.0 changelog entry says the anti-detection approach moved from removing and renaming variables to stopping them from being injected in the first place. That project is licensed under [GPL-3.0](https://github.com/ultrafunkamsterdam/undetected-chromedriver/blob/master/LICENSE).
nodriver goes a step further. Instead of patching the WebDriver path, it drops it. The README calls nodriver "the official successor" and includes a utility to convert a running undetected-chromedriver instance into a nodriver browser, so existing scripts can move over gradually.
The older README contains a limit that applies to both tools. In bold, it says the package "DOES NOT hide your IP address," and that a script run from a datacenter or from a home connection with a poor reputation is unlikely to pass. Changing how a browser is driven does nothing to the network it runs on.
The [patched Chromium guide](https://www.centinelanalytica.com/blog/patched-chromium-browsers) covers how driver-level, framework-level, and binary-level changes differ.
## What removing WebDriver does and does not change
The [WebDriver specification](https://w3c.github.io/webdriver/#interface) defines `navigator.webdriver` as a report that the browser is under WebDriver remote control. nodriver does not start a WebDriver session, so a check that looks only for WebDriver or chromedriver artifacts is looking for a control path this library does not use. That is the same weakness a single-property rule has against [Playwright Stealth](https://www.centinelanalytica.com/blog/what-is-playwright-stealth).
Other evidence is still there:
- **CDP-driven automation.** The browser is still controlled through a debugging protocol. The client, the commands it sends, and the order it sends them in can leave traces that depend on the Chromium version and the kind of observation.
- **Environment consistency.** A stock browser on a server, a headless session, or a fresh profile with no history still has to agree with its claimed platform, screen, fonts, and graphics. nodriver does not generate fingerprints the way an [antidetect browser](https://www.centinelanalytica.com/blog/what-is-an-antidetect-browser) does.
- **Network consistency.** IP reputation, connection properties, and whether the request headers match the browser's claims are unchanged by the automation library, as the undetected-chromedriver README itself admits.
- **Behavior.** Navigation order, timing, form completion, and repetition across sessions reflect the script, not the transport.
None of these is proof on its own. Legitimate tools use CDP too, including test runners and monitoring systems. The [browser automation detection guide](https://www.centinelanalytica.com/blog/detect-browser-automation) explains how to combine evidence from layers that fail in different ways.
## What to do with it as a defender
Treat nodriver as a test case, not a signature. Add a pinned nodriver version to your automation regression cohort next to stock Selenium and Playwright, and record which evidence changes between them. If your detection only works because chromedriver was present, nodriver will show it.
Decide the response by route and action. A nodriver session reading a public page and one repeatedly submitting a signup form carry different risk. Keep an allow path for your own and your partners' approved automation, and log the evidence behind each decision so a false positive can be reversed and the test rerun when nodriver or Chrome changes. Centinel's [bot management](https://www.centinelanalytica.com/bot-management) applies these responses per route.
## Sources
- [nodriver repository and README](https://github.com/ultrafunkamsterdam/nodriver)
- [nodriver documentation](https://ultrafunkamsterdam.github.io/nodriver/)
- [nodriver launch configuration source](https://github.com/ultrafunkamsterdam/nodriver/blob/main/nodriver/core/config.py)
- [nodriver on PyPI](https://pypi.org/project/nodriver/)
- [undetected-chromedriver repository and README](https://github.com/ultrafunkamsterdam/undetected-chromedriver)
- [Chrome DevTools Protocol documentation](https://chromedevtools.github.io/devtools-protocol/)
- [WebDriver specification for navigator.webdriver](https://w3c.github.io/webdriver/#interface)
---
Page: https://www.centinelanalytica.com/blog/what-is-crawl4ai
# What is Crawl4AI? The open-source AI crawler
Crawl4AI is an open-source Python crawler that turns web pages into Markdown for LLMs. What it can change, what your site sees, and how to assess it.
By Frederick Jahn. Published 2026-09-24.
Crawl4AI is an open-source Python library for running your own [AI crawler](https://www.centinelanalytica.com/blog/what-is-an-ai-crawler). It crawls websites and converts pages into clean Markdown or structured data for large language models. It usually drives a real browser through Playwright, and it runs wherever its user installs it, so there is no single Crawl4AI network or service behind the traffic.
## What Crawl4AI does
The [project repository](https://github.com/unclecode/crawl4ai) describes Crawl4AI as an "LLM Friendly Web Crawler & Scraper" and publishes it under the Apache License 2.0. A developer installs it with pip, and the setup step installs Playwright and its browsers. The [browser configuration reference](https://docs.crawl4ai.com/core/browser-crawler-config/) lists Chromium, Firefox, and WebKit as browser types, with Chromium as the default.
The output is the point. The [Markdown generation docs](https://docs.crawl4ai.com/core/markdown-generation/) say the library was built to keep a page's actual content and discard boilerplate. Content filters score text density, link density, and page structure to drop navigation and footers, which produces what the project calls "fit markdown." The same page can also come back as HTML, JSON from an extraction strategy, a screenshot, or a PDF.
Crawl4AI does not always need a browser. Version 0.5.0 added an [HTTP-only crawler strategy](https://docs.crawl4ai.com/blog/releases/0.5.0/) for pages that do not need JavaScript rendering. The same tool can therefore reach your site as a full browser session or as plain HTTP requests, depending on how the operator configured it.
## How people use it
The common pattern is a pipeline: crawl a list of URLs or a whole site, turn each page into Markdown, and feed the result into a retrieval index, an agent, or a dataset. Crawl4AI can also run as a self-hosted server. The [installation page](https://docs.crawl4ai.com/core/installation/) describes a Docker image that exposes the crawler over HTTP, which lets other applications send it jobs.
Unlike a hosted scraping API such as [Firecrawl](https://www.centinelanalytica.com/blog/what-is-firecrawl), Crawl4AI has no vendor-operated fetch fleet. The requests your site receives come from whatever machine, cloud account, or proxy the operator chose. An IP list for "Crawl4AI" does not exist in any useful sense.
## Settings that matter to defenders
Several documented options change what a target website sees:
- **User agent.** The operator can set any user-agent string or set `user_agent_mode` to `"random"`, which the [parameter reference](https://docs.crawl4ai.com/api/parameters/) describes as a way to randomize the user agent. The string your logs record is whatever the operator picked.
- **Stealth mode.** Setting `enable_stealth=True` applies [playwright-stealth](https://www.centinelanalytica.com/blog/what-is-playwright-stealth). The [undetected browser page](https://docs.crawl4ai.com/advanced/undetected-browser/) says stealth mode removes the `navigator.webdriver` flag and modifies browser fingerprints. A separate undetected browser adapter applies deeper patches aimed at bot detection.
- **Simulated interaction.** The run configuration includes `simulate_user`, documented as simulating mouse movements to avoid bot detection, and `override_navigator`, which overrides `navigator` properties in JavaScript.
- **Proxies.** The [proxy docs](https://docs.crawl4ai.com/advanced/proxy-security/) cover authenticated HTTP, HTTPS, and SOCKS5 proxies, plus a round-robin strategy that rotates proxies per request.
- **Persistent identity.** [Identity-based crawling](https://docs.crawl4ai.com/advanced/identity-based-crawling/) stores cookies, local storage, and other session data in a browser profile so a crawl can reuse a logged-in session without repeating the login.
The project is candid about the limits of these modes. The undetected browser page says evasion is "Not 100% Guaranteed" and that headless mode can be detected more easily. It suggests running with a visible browser window instead.
## Crawl rate and robots.txt
Crawl4AI includes pacing controls, but the operator chooses them. The [multi-URL crawling docs](https://docs.crawl4ai.com/advanced/multi-url-crawling/) describe a rate limiter that waits a random one to three seconds between requests to the same domain by default and backs off when the site returns 429 or 503. Dispatchers cap concurrent tasks, with documented defaults of 10 or 20 depending on the dispatcher.
robots.txt is opt-in. The parameter reference lists `check_robots_txt` with a default of `False`. When it is enabled, the crawler checks robots.txt rules before each URL. Without that flag, a default Crawl4AI run does not consult your robots.txt at all. robots.txt is a voluntary policy in any case, so it tells you what a polite crawler should do, not what a given client will do.
## What a target website can observe
Because the operator controls the user agent, network path, and browser patches, one field rarely settles the question. What you can observe depends on the mode:
- **HTTP-only runs** send requests without executing JavaScript. They load no scripts or images and leave no browser signals to inspect. A browser user agent on such a request is a claim you can check against its headers and TLS handshake, which come from an HTTP client rather than a browser.
- **Browser runs** execute your pages in Playwright-driven Chromium, Firefox, or WebKit. Stealth patches change some JavaScript-visible properties. Consistency checks still apply: does the claimed browser match the TLS fingerprint, header order, rendering behavior, and platform details? A randomized user agent on top of an unchanged browser build is a common source of mismatches. [How to detect browser automation](https://www.centinelanalytica.com/blog/detect-browser-automation) walks through that kind of evidence.
- **Request patterns** are often the clearest signal. A Markdown pipeline typically fetches documents, skips assets it does not need, follows links systematically, and moves through a section faster and more evenly than a reader does.
## How to assess Crawl4AI traffic on your site
The question is less "is this Crawl4AI?" and more "what is this client doing, and is that allowed?" When reviewing a suspected crawl:
- Which routes did it request, and were any behind a login or a paywall?
- Did it reuse cookies from an authenticated session?
- Does the same traversal order repeat across changing IPs or user agents?
- Did it request robots.txt, and did it follow the rules it found?
- Is there an approved integration, partner, or internal team running a crawler?
Then pick a response that fits the resource. A public documentation page may be fine to crawl at a modest rate. A search endpoint, a pricing page that competitors track, or a signed-in workflow may warrant rate limits, a challenge, or a block. [How to prevent web scraping](https://www.centinelanalytica.com/blog/how-to-prevent-web-scraping) covers those options in more depth. The [scraping and AI crawler use case](https://www.centinelanalytica.com/use-cases/scraping-and-ai-crawlers) shows how Centinel applies them per route.
## Sources
- [Crawl4AI repository, README, and license](https://github.com/unclecode/crawl4ai)
- [Crawl4AI: Installation](https://docs.crawl4ai.com/core/installation/)
- [Crawl4AI: Browser, crawler and LLM config](https://docs.crawl4ai.com/core/browser-crawler-config/)
- [Crawl4AI: Parameter reference](https://docs.crawl4ai.com/api/parameters/)
- [Crawl4AI: Markdown generation](https://docs.crawl4ai.com/core/markdown-generation/)
- [Crawl4AI: Undetected browser mode](https://docs.crawl4ai.com/advanced/undetected-browser/)
- [Crawl4AI: Proxy and security](https://docs.crawl4ai.com/advanced/proxy-security/)
- [Crawl4AI: Identity-based crawling](https://docs.crawl4ai.com/advanced/identity-based-crawling/)
- [Crawl4AI: Multi-URL crawling](https://docs.crawl4ai.com/advanced/multi-url-crawling/)
- [Crawl4AI v0.5.0 release notes](https://docs.crawl4ai.com/blog/releases/0.5.0/)
---
Page: https://www.centinelanalytica.com/blog/fake-crawler-report
# Fake Googlebot and AI crawlers: 2026 data
On the typical news site, 1 in 7 requests claiming to be Googlebot failed verification, and 84% of GPTBot claims were fake. Data, method and CSV download.
By Frederick Jahn. Published 2026-09-22, updated 2026-09-30.
Anyone can put "[Googlebot](https://www.centinelanalytica.com/bots/googlebot)" in a user-agent header. From late June to 21 September 2026 we
verified every request that named a well-known crawler on the news and media sites Centinel
protects. On the typical site, 1 in 7 requests calling itself Googlebot did not come from
Google. For several [AI crawlers](https://www.centinelanalytica.com/blog/what-is-an-ai-crawler), most requests using the name
were fake. To check one request yourself, see
[how to spot a fake Googlebot](https://www.centinelanalytica.com/blog/fake-googlebot).
The report in four numbers1 in 7requests claiming to be Googlebot were fake on the typical site84%of GPTBot claims were fake on the typical site99%of Perplexity-User claims were fake on the typical site94%of fake Googlebot requests copied the real version stringNews and media sites protected by Centinel, late June to 21 September. The typical site is the median across sites.
## Most impersonated crawler identities
A request is fake when the operator's published IP ranges or DNS check contradict it.
"Typical site" is the median across sites. "All traffic" pools every request, so the busiest
sites weigh more.
Share of crawler claims that failed verificationTypical siteAll trafficGoogle-Extended>99%>99%Perplexity-User99%96%Claude-SearchBot99%2%CCBot88%44%GPTBot84%2%Meta-ExternalAgent77%2%ClaudeBot40%6%Claude-User25%24%PerplexityBot20%19%Googlebot14%6%ChatGPT-User12%1%OAI-SearchBot12%4%Bingbot, Applebot, Amazonbot≤1%<1%Typical site is the median across sites with a meaningful number of claims. All traffic pools every request, so the busiest sites count for more. Each figure comes from at least five sites.
The two measures differ because real crawler traffic concentrates on a few busy sites. Those
sites pull the pooled share down. On the typical site the real crawler rarely visits, so most
requests using its name are fake. For [GPTBot](https://www.centinelanalytica.com/bots/gptbot), that is 84%.
## Google-Extended is never a real visitor
[Google-Extended](https://www.centinelanalytica.com/bots/google-extended) is a robots.txt token, not a crawler. Google's
[crawler documentation](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers#google-extended)
says it "doesn't have a separate HTTP request user agent string". Any request that sends it as
a user agent is fake and can be blocked without a lookup.
## Where the fakes come from
88% of fake Googlebot requests came from datacenter addresses, about a third of them from a
single European hosting provider. At least half of the fake GPTBot, [ClaudeBot](https://www.centinelanalytica.com/bots/claudebot), [PerplexityBot](https://www.centinelanalytica.com/bots/perplexitybot)
and [ChatGPT-User](https://www.centinelanalytica.com/bots/chatgpt-user) requests came from Google Cloud.
Share of fake requests sent from datacenter addressesPerplexityBot99%GPTBot98%ClaudeBot98%Claude-SearchBot98%CCBot98%Perplexity-User97%ChatGPT-User96%OAI-SearchBot96%Googlebot88%Claude-User80%Meta-ExternalAgent63%Requests that failed verification, pooled across sites. Datacenter means the address belongs to a hosting or cloud network.
Fakes target articles. 95% of fake Googlebot requests asked for pages rather than images,
scripts or stylesheets, against 85% for the real Googlebot.
## The version string does not give fakes away
94% of fake Googlebot requests claimed the real Googlebot's current version. For ChatGPT-User,
[Bingbot](https://www.centinelanalytica.com/bots/bingbot), [Amazonbot](https://www.centinelanalytica.com/bots/amazonbot) and Claude-SearchBot, every fake did.
Fake requests that claimed the real crawler's current versionChatGPT-User>99%Bingbot>99%Amazonbot>99%Claude-SearchBot>99%Googlebot94%A version check catches almost none of them.
CCBot is the exception: 99% of its fakes sent the bare crawler token without the usual
`Mozilla/5.0` prefix. Across all identities, 90% of fake requests used the full
browser-style string.
## Allowing crawlers by name lets the fakes in
A common bot rule allows search and AI crawlers by user-agent string, so that blocking never
costs search visibility. Under that rule, every fake request in this report gets the same
treatment as the real crawler.
Verify first, then allowA request says `Googlebot` in its user agentCheck the address against the operator's published IP list, or run its reverse and forward DNS checkVerifiedThe address matches the operator. Apply your policy for that crawler.ContradictedThe operator's method rules the claim out. This is an impersonator.UnknownThe operator publishes no method. Treat it as an unverified bot.Keep the three results apart. A contradicted claim is stronger evidence of intent than an unknown one.
1. Check the address against the operator's published IP list or its reverse and forward DNS
check. Google, Bing, OpenAI, Anthropic and Perplexity publish machine-readable lists. The
[AI agent verification guide](https://www.centinelanalytica.com/blog/how-to-verify-ai-agents) covers each operator's
method, and the [crawler verification tool](https://www.centinelanalytica.com/verify-crawler) checks a single address.
2. Keep verified, contradicted and unknown as separate results.
3. Tie the allow rule to the verified operator, not to the string. Keep your
[robots.txt policy](https://www.centinelanalytica.com/blog/robots-txt-ai-bots-guide) for the real crawlers.
For 15 to 21 September, we compared Centinel's verification with the operators' own lists for
Google, Bing, OpenAI and Perplexity. 99.99% of verified requests came from the published
ranges, and 98% of unverified requests came from outside them. Against Anthropic's list, every
unverified Claude request came from outside it.
Centinel's verification against the operators' own IP listsVerified requests inside the published ranges99.99%Unverified requests outside the published ranges98%Google, Bing, OpenAI and Perplexity, 15 to 21 September. A one-day check against Anthropic's list found every unverified Claude request outside it.
## Automated browsers
Automated browsers are headless or scripted browsers running real rendering engines. They
don't name a crawler; they claim to be ordinary visitors. 56% claimed to be Chrome and 70%
came from datacenter addresses. No published list covers them, so catching them takes
behavioral and browser evidence. The [AI agent traffic guide](https://www.centinelanalytica.com/blog/what-is-ai-agent-traffic) covers the
agents in this group.
Which browser automated browsers claimed to beChrome56%No browser identified30%Firefox6%Safari4%Edge3%Other1%Requests classified as automated browsers, pooled across sites. 70% of them came from datacenter addresses.
## Method
Data: requests to news and media sites protected by Centinel, late June to 21 September 2026,
excluding test and staging hosts. Each figure is a median across sites or a share of all
traffic, as labelled, and comes from at least five sites.
Claim: the user agent contains the crawler's documented token.
Verification: the operator's documented method, in this order: a signed request where
supported, published IP ranges, then reverse and forward DNS. A request fails when the method
contradicts the claim. Facebook's link-preview fetcher and Semrush are excluded because their
checks were unreliable.
Networks: "datacenter" means the address belongs to a hosting or cloud network, per Centinel's
IP intelligence.
The [crawler audit guide](https://www.centinelanalytica.com/blog/audit-ai-crawler-traffic) shows how to run the same check on
your own logs.
## Data and citation
[Download the shares behind every chart (CSV)](https://www.centinelanalytica.com/research/2026-09-fake-crawlers.csv). The charts
and figures may be reused with this attribution:
> Centinel Analytica, "Fake Googlebot and AI crawlers: 2026 verification data",
> September 2026, [https://www.centinelanalytica.com/blog/fake-crawler-report](https://www.centinelanalytica.com/blog/fake-crawler-report)
Until 30 September 2026 this report was titled "Fake crawlers: how often Googlebot and AI bots
are not who they claim". Citations made under that title refer to the same report and data.
---
Page: https://www.centinelanalytica.com/blog/respond-to-layer-7-bot-attacks
# How to respond to a layer 7 bot attack
Contain an application-layer bot flood by finding the exhausted resource, narrowing traffic controls, and checking recovery. Includes an incident worksheet.
By Frederick Jahn. Published 2026-09-05, updated 2026-10-11.
During a suspected layer 7 bot attack, first identify the resource that is failing and the requests consuming it. Containment should reduce work before that bottleneck while preserving a usable path for legitimate traffic.
A large request count is not required. [AWS documents that application-layer attacks can overload specific functions with relatively low traffic volumes](https://docs.aws.amazon.com/whitepapers/latest/aws-best-practices-ddos-resiliency/application-layer-attacks.html). Request rate and application work are different measurements.
If the network link is saturated before your application can receive traffic, involve the network or upstream protection provider. An application rule cannot recover capacity it never gets to use.
## Establish impact and assign one change owner
Record the affected user journey, start time, known deployment changes, and the service that first shows distress. Name an incident lead and one owner for traffic-policy changes so two teams do not unknowingly counteract each other.
Preserve a small, redacted sample of request evidence. Keep the full operational logs in their existing restricted system. Do not copy credentials, raw request bodies, or private customer records into an incident chat.
Separate confirmed observations from assumptions. “Search workers saturated while one query family increased” is useful. “A huge botnet attacked us” requires evidence you may not yet have.
## Find the scarce resource
[OWASP's denial-of-service guidance](https://cheatsheetseries.owasp.org/cheatsheets/Denial_of_Service_Cheat_Sheet.html) distinguishes application, session/protocol, and network exhaustion. Use that distinction to choose the response owner and enforcement point.
| Observed bottleneck | Inspect next | Candidate containment |
| --- | --- | --- |
| Search workers | Query family and concurrency | Bound expensive search operations |
| Database capacity | Routes and query work | Restrict the contributing operation |
| Background queue | Admission rate and oldest job | Bound new jobs and duplicate submissions |
| Origin requests | Cache behavior and route mix | Restore safe caching or limit origin-bound work |
| Network bandwidth | Upstream and edge visibility | Escalate to the network protection path |
These are options to evaluate against your architecture, not commands to apply blindly during an incident.
## Make one narrow change and observe it
Prefer a clearly scoped intervention: one expensive route, one client class, or one operation whose allowance is being exceeded. Record the configuration version and rollback action before enforcement.
Where supported, reduce admitted work before expensive processing begins. A rule evaluated after the database query may reject the response without relieving the database.
Do not create an unlimited queue as an alternative to rejection. That can postpone the failure and make recovery harder. Decide how the application communicates temporary unavailability to cooperative clients.
If identity is uncertain, use it as an investigation lead rather than a reason to block a broad shared network. Keep verified integrations and critical workflows visible in the impact review.
## Check recovery from the user's side
A healthier CPU graph does not establish recovery if legitimate users are now blocked at the edge. Verify the affected journey through the normal public entry point using an authorized test account.
Compare successful work, failure rate, backlog, and the requests affected by the new control. Also check whether retries or traffic moved the load to a different route.
If the intervention does not reduce the bottleneck, revisit the causal assumption. Do not stack unrelated rules merely because the incident is still active.
## Leave a usable incident record
Record what failed, what evidence linked the requests to that failure, which control changed, and what recovered. Keep unresolved questions explicit. Schedule removal or review of temporary exceptions and restrictions.
The [traffic-spike diagnostic guide](https://www.centinelanalytica.com/blog/bot-traffic-vs-traffic-spike) helps when the cause remains uncertain. The [traffic flood use case](https://www.centinelanalytica.com/use-cases/ddos-and-traffic-floods) frames the product boundary, including the distinction between application automation controls and upstream capacity protection.
Close the incident when the intended user journey is working and the emergency configuration has an owner, not simply when the request graph looks quieter.
---
Page: https://www.centinelanalytica.com/blog/detect-cache-busting-bots
# How to detect cache-busting bots
Investigate query-string variation, falling cache hits, and rising origin work. Review cache keys safely without merging meaningfully different responses.
By Frederick Jahn. Published 2026-09-05, updated 2026-10-11.
Cache-busting bots create request variations that may prevent a cache from reusing an existing response. Investigate them by comparing request variants, cache outcomes, and the work reaching the origin.
Start with the actual cache key. Different URLs do not necessarily mean different cache entries in every deployment, and a low hit rate does not automatically mean an attack.
[AWS describes query-string variation as a mechanism in cache-busting HTTP floods](https://docs.aws.amazon.com/whitepapers/latest/aws-best-practices-ddos-resiliency/application-layer-attacks.html). Legitimate asset versioning and application filters also vary URLs, so the mechanism alone does not establish intent.
## Choose one route and reconstruct its cache policy
Record the route, cache layer, relevant request parameters, response directives, and whether content differs by account, language, or another context. Include recent cache and application changes.
Do not begin by removing every query parameter from the key. [RFC 9111](https://www.rfc-editor.org/rfc/rfc9111) describes cache selection using the target URI and relevant variation. Reusing the wrong response can create a correctness or data-exposure problem.
Confirm whether the response is eligible for shared caching at all. A personalized result may appropriately miss a shared cache even when the request rate is high.
## Compare variants with the origin work they create
Group a bounded sample by normalized route and parameter names. Keep sensitive parameter values out of ordinary reports. Compare distinct variants, cache outcomes, origin requests, and origin execution time.
A global cache-hit percentage can hide a small, expensive route. Review that route's request volume and work separately from static assets.
| Observation | Question to answer |
| --- | --- |
| Many unique values for an unfamiliar parameter | Does the application use it, ignore it, or reject it? |
| Hit rate falls after a release | Did the cache key or response policy change? |
| Origin work rises without more edge requests | Did request cost or cache eligibility change? |
| One account receives private responses | Is bypassing shared cache intentional? |
| New versioned asset URLs appear | Does the pattern match a normal deployment? |
Preserve alternative explanations until the application and caching owners have checked them.
## Classify parameters before normalizing anything
Consider a hypothetical result endpoint with a language parameter, a filter parameter, and an unrelated tracking parameter. The first two may change the response. The third might not.
That is a hypothesis to verify against the implementation. Compare responses under controlled conditions, including authentication and locale. A parameter ignored by one handler may affect another handler or a downstream service.
If the team chooses to exclude a parameter from the key, document why doing so preserves response meaning. Prefer an explicit, reviewed allowlist of meaningful variation over a broad emergency transformation.
Also inspect unknown-parameter handling. Rejecting unsupported input may be clearer than accepting unlimited variants whose behavior nobody owns.
## Apply protection before unnecessary origin work
Where your architecture supports it, enforce route-specific request or work budgets before forwarding expensive requests. Keep approved crawlers and client integrations within documented allowances.
If a rule is enforced after the origin has generated the response, it may change what the requester receives without relieving the origin. Check the request path, not just the presence of a block event.
Avoid making private responses public as a performance fix. If a route cannot safely use shared caching, its capacity and admission controls need to reflect that.
## Verify both load and response correctness
After a change, compare cache reuse and origin work for the selected route. Check that language, filters, authentication, and error responses still behave correctly. Include an ordinary deployment's versioned assets in the review.
Record the previous cache configuration and a rollback owner. Cache changes can affect many users even when the initial investigation concerns one suspected client.
Use the [layer 7 response guide](https://www.centinelanalytica.com/blog/respond-to-layer-7-bot-attacks) if overload is active. For broader protection scope, see the [traffic flood use case](https://www.centinelanalytica.com/use-cases/ddos-and-traffic-floods). Bring the reviewed cache key and one origin-work comparison; those are more useful than a list of suspicious query strings.
---
Page: https://www.centinelanalytica.com/blog/bot-traffic-vs-traffic-spike
# Bot traffic or traffic spike? How to tell
Compare traffic surges with request behavior, route cost, and real workflow completion. Avoid blocking a flash crowd or overlooking a bot-driven overload.
By Frederick Jahn. Published 2026-09-05, updated 2026-10-11.
To distinguish bot traffic from a legitimate traffic spike, compare the requests with the work users are completing and the resources the service is consuming. Volume, geography, and user-agent strings are clues. None settles the cause by itself.
Keep three hypotheses open at the start: real demand increased, automated activity increased, or a service change made ordinary traffic more expensive. More than one can be true.
## Begin with the affected journey
Identify what changed for users: slow search, failed login, delayed downloads, or general unavailability. Compare the time of that change with traffic, deployments, campaigns, and dependency incidents.
[Google's SRE monitoring guidance](https://sre.google/sre-book/monitoring-distributed-systems/) uses latency, traffic, errors, and saturation to describe service health. It also separates symptoms from causes. Apply that discipline before assigning an attack label to a busy graph.
Use the same time window and observation point when comparing metrics. Edge requests, origin requests, application sessions, and analytics events count different things.
## Test competing explanations
| Hypothesis | Evidence to seek | Evidence that weakens it |
| --- | --- | --- |
| Legitimate demand | Expected entry pages and completed workflows | Little useful activity after repeated expensive requests |
| Approved automation | A known owner and documented workload | Unexplained operations outside its scope |
| Abusive automation | Repeated policy violations or extraction behavior | A confirmed permitted integration matching the pattern |
| Application regression | Cost or failures rise after a change | Stable execution cost while external demand changes |
| Retry amplification | Repeated operations after errors or timeouts | Independent successful operations explain the volume |
Treat “evidence that weakens it” as an invitation to investigate, not as a definitive exclusion rule.
Avoid inferring intent from a new country or a shared network. A public link can bring an unfamiliar audience. A mobile provider may place many legitimate visitors behind the same address.
## Work through a hypothetical launch
Suppose a feature announcement brings visitors to a page that immediately requests search suggestions. Search latency rises, browsers retry, and the origin receives more calls than the public page-view count suggests.
That sequence could resemble a bot flood if the investigation begins and ends with repetitive API requests. Before blocking, inspect the frontend request behavior, error timing, and relationship between visits and search operations.
Now suppose a separate client repeatedly traverses search results without performing the announced workflow. That may be an independent automation problem. Split the populations rather than forcing the whole incident into one explanation.
The example is a diagnostic exercise, not a reported Centinel incident.
## Compare completed work, not only engagement proxies
For an application, use a relevant server-side outcome: a successful search result, an authorized record read, or a completed workflow. Analytics engagement can add context, but missing analytics does not prove automation.
Conversely, a bot can complete a workflow. A completed operation establishes what happened, not whether it was permitted. Compare the result with the client's allowed purpose and usage.
For analytics-specific questions, use [bot traffic in Google Analytics](https://www.centinelanalytica.com/blog/how-to-identify-bot-traffic-in-google-analytics). Keep event filtering separate from a decision about serving requests.
## Contain the harm while uncertainty remains
You may need to protect an overloaded operation before attribution is complete. [OWASP's denial-of-service guidance](https://cheatsheetseries.owasp.org/cheatsheets/Denial_of_Service_Cheat_Sheet.html) emphasizes the resource and architectural layer under strain.
Choose a bounded, reversible control on the affected work. Monitor the legitimate journey while it is enforced. Record whether the change reduced the actual bottleneck and who was affected.
If the evidence supports an application-layer attack, continue with the [layer 7 response guide](https://www.centinelanalytica.com/blog/respond-to-layer-7-bot-attacks). The [traffic flood use case](https://www.centinelanalytica.com/use-cases/ddos-and-traffic-floods) is the product context. Keep the diagnosis record: it prevents a temporary hypothesis from becoming a permanent blocking rule.
---
Page: https://www.centinelanalytica.com/blog/prevent-graphql-abuse
# GraphQL rate limiting for expensive queries
Bound GraphQL query depth, breadth, pagination, and cost. Learn why HTTP request limits alone miss expensive operations.
By Frederick Jahn. Published 2026-09-05, updated 2026-10-11.
GraphQL rate limiting should account for the work inside an operation, not only the number of HTTP requests. One request can select many fields, traverse relationships, or ask for large lists.
Start with server-enforced bounds on returned data and operation shape. Then add client budgets that reflect estimated cost and actual service capacity. A bot assessment can inform policy, but it cannot calculate resolver cost or enforce object permissions on its own.
[GraphQL's security guidance](https://graphql.org/learn/security/) covers pagination, depth and breadth limits, and query complexity analysis. These controls address different forms of demand; none is a universal substitute for the others.
## Review what a client can ask for
For each important operation, record its list sizes, relationship traversal, and expensive resolver dependencies. Include operations used by older application versions and approved integrations.
Do not assume that a shallow query is cheap. A broad selection can request many independent fields. A single expensive resolver can also dominate work even when depth is low.
Likewise, a persisted operation is not automatically safe merely because its text is stored. Review which documents are actually approved and whether client-supplied variables can expand their work beyond intended bounds.
## Match each limit to its failure mode
| Control | What it bounds | What still needs review |
| --- | --- | --- |
| Page-size cap | Items returned from a list | Number of lists selected |
| Depth limit | Nested relationship traversal | Broad shallow operations |
| Breadth or alias limit | Repeated parallel selections | Cost of each selection |
| Complexity allowance | Estimated operation work | Whether estimates match resolver behavior |
| Concurrency limit | Operations running together | Cost and duration of each operation |
| Timeout | Allowed execution duration | Whether downstream work actually stops |
Treat the table as a design worksheet. The correct combination depends on the implementation and the operations your clients need.
Keep authorization at the resource boundary. [OWASP's REST guidance](https://cheatsheetseries.owasp.org/cheatsheets/REST_Security_Cheat_Sheet.html) describes endpoint access control; for GraphQL, apply the same separation of traffic policy and access permission to the operations and resources being resolved.
## Make cost estimates explainable
If you assign weights to fields, document the reason: database work, result size, or a constrained dependency. Avoid a scoring system that cannot explain why a normal operation is refused.
Compare estimated cost with observed execution time and downstream calls in a controlled environment. A schema change can make an old estimate inaccurate. Include cost review when a resolver gains a new dependency or a list changes its behavior.
Decide how variables affect cost before execution. If the calculation ignores a client-selected list size, the nominal budget may not bound the work you intended.
## Review batching and retries explicitly
If the server accepts multiple operations in one HTTP request, establish whether each operation consumes an allowance. Check whether concurrent requests share the same account or tenant budget.
A retry after a timeout may arrive while earlier work is still running. Trace cancellation and duplicate mutation handling instead of assuming that the failed client request means the server did nothing.
Return an error the client understands. An HTML challenge in a GraphQL response path needs an explicit integration design; it should not appear unexpectedly inside a client expecting a structured response.
## Test the approved client contract
Review a normal application screen, a mobile client, and any permitted reporting integration. Include maximum supported pagination and a rejected operation. Use synthetic data and bounded requests in an authorized environment; generating an uncontrolled load spike is unnecessary for this review.
After a change, monitor rejected operations by reason, successful completion, and backend saturation. Keep a route back to the previous configuration if approved clients break.
Use [API rate-limit design](https://www.centinelanalytica.com/blog/api-rate-limiting-for-bot-protection) for broader counting scopes and the [API abuse use case](https://www.centinelanalytica.com/use-cases/api-abuse) for the product discussion. Bring a representative operation and its resource costs so the evaluation addresses the GraphQL workload, not just the HTTP endpoint.
---
Page: https://www.centinelanalytica.com/blog/detect-api-bot-abuse
# How to detect bot abuse on APIs
Investigate API automation using credentials, routes, resource use, and outcomes. Distinguish approved clients from abusive requests.
By Frederick Jahn. Published 2026-09-05, updated 2026-10-11.
API bot detection begins with an awkward fact: many API clients are supposed to be automated. The useful task is to identify requests that exceed their permissions, consume unacceptable resources, or misuse a workflow.
For a native API client, missing browser JavaScript is expected. Investigate the evidence the server actually receives: validated credentials, routes, operation sequences, returned data, and downstream outcomes.
Keep authentication and authorization mandatory. [OWASP's REST security guidance](https://cheatsheetseries.owasp.org/cheatsheets/REST_Security_Cheat_Sheet.html) places access control at API endpoints; an automation assessment is additional context.
## Name the clients you intend to serve
Separate browser-backed requests, mobile applications, partner integrations, internal services, and public anonymous clients. Record which authentication and resource policies apply to each.
An API key identifies a credential presented to the service. It does not guarantee that the holder is using it correctly or that one person controls every request. A browser-looking user-agent is even less conclusive.
For approved integrations, keep the owner, allowed operations, intended workload, and revocation route in an operational record. Unknown automation should remain unknown until verified; do not label it malicious merely because it lacks a familiar name.
## Know where the observation was made
| Evidence | Useful for | Boundary to preserve |
| --- | --- | --- |
| Validated credential | Accountable client scope | A credential may be misused |
| Source address at the trusted edge | Network context | Shared networks and proxies complicate attribution |
| Route and method | Requested operation | Route names alone omit resource cost |
| Response and workflow outcome | What the client actually achieved | A success status may not mean a completed action |
| Browser telemetry | Browser-backed flows where collected | Native API clients do not provide it |
If a reverse proxy forwards request metadata, establish which hop is trusted to supply it. Do not accept arbitrary client-supplied forwarding headers as verified origin identity.
The same caution applies to connection fingerprints. Observe a connection at a proxy and you may be describing the proxy's connection rather than the original client's.
## Find excessive use of legitimate operations
A client may be authenticated, authorized, and still use an API in a way the service does not intend. [OWASP API6](https://owasp.org/API-Security/editions/2023/en/0xa6-unrestricted-access-to-sensitive-business-flows/) covers this class of sensitive-workflow abuse.
Choose one operation and compare its completed work with the client's allowance. For reads, inspect resource coverage and data returned. For writes, inspect state changes, duplicate actions, and downstream effects. Keep retries separate from distinct intended operations.
Build the investigation around a bounded time window and one client class. Combining partner sync jobs with anonymous requests makes both the baseline and the response harder to interpret.
## Return a response the client understands
A non-browser integration generally needs a documented API response, not an unexpected interactive page. Define retry behavior and error handling for each control you introduce.
Use [API rate-limit design](https://www.centinelanalytica.com/blog/api-rate-limiting-for-bot-protection) to choose resource and client budgets. For valid credentials extracting excessive data, review [authenticated scraping](https://www.centinelanalytica.com/blog/prevent-authenticated-scraping). These controls answer different parts of the same request path.
Do not silently disable authentication when the detection service is unavailable. Decide the fallback for automation assessment separately from the non-negotiable permission check.
## Check approved automation after each change
Replay a small, authorized staging workflow for a partner job, a mobile client, and an internal service. Include retries after a timeout and credentials with narrower permissions. Confirm that the application preserves its intended contract.
After rollout, monitor useful completed operations and rejected work by client class. A large “bot traffic” percentage may be normal for an API. The outcome that matters is whether the permitted clients can do their jobs without giving other clients unrestricted access.
For a product evaluation, use the [API abuse use case](https://www.centinelanalytica.com/use-cases/api-abuse). Start with one operation and a permitted client that must keep working; use those to check the proposed integration.
---
Page: https://www.centinelanalytica.com/blog/api-rate-limiting-for-bot-protection
# API rate limiting for bot protection
Choose API rate limits around client identity, route cost, bursts, and retries. Avoid one global rule that blocks users but misses expensive abuse.
By Frederick Jahn. Published 2026-09-05, updated 2026-10-11.
A useful API rate limit needs more than a number. Define what is counted, whose allowance is consumed, the period it covers, and what happens when it is exhausted.
For bot protection, those decisions should reflect the operation's cost and the legitimate client's job. One global requests-per-second rule can be too strict for a harmless batch client and too generous for an expensive endpoint.
[HTTP 429](https://www.rfc-editor.org/rfc/rfc6585) communicates excessive requests and may include a Retry-After header. The standard does not choose the identity key, counting method, or business allowance for you.
## Start with the resource you want to protect
List the operation and the constrained resource: request-handling capacity, database work, a third-party call, or access to a valuable dataset. If one request can create many units of work, count those units as well as incoming requests.
A read of one cached object and a request for a large generated report should not automatically consume identical allowances. You may still need a coarse request cap, but it should not be your only bound.
Also distinguish rate from concurrency. A slow operation can occupy workers long after it was admitted. Limiting starts per minute does not directly limit the number of operations still running.
## Choose independent counting scopes
[OWASP's bot-management guidance](https://cheatsheetseries.owasp.org/cheatsheets/Bot_Management_and_Anti-Automation_Cheat_Sheet.html) describes account, source, and endpoint limits. Each answers a different question.
Consider a hypothetical login service. A counter keyed only by the combination of source IP and username creates a fresh allowance for each pair. It does not independently bound all attempts against one account or all accounts attempted by one source.
The broader design lesson is to write each intended constraint separately before choosing storage keys.
| Budget | Intended bound | Legitimate case to review |
| --- | --- | --- |
| Per credential | Work attributable to one client | Scheduled partner batch |
| Per account or tenant | Aggregate account consumption | Several users working together |
| Per route | Demand on one service function | A product launch affecting that route |
| Per source | Coarse anonymous request pressure | Shared office or mobile network |
| Concurrent operations | Work still occupying capacity | Slow but permitted jobs |
Not every endpoint needs every budget. Add the scopes that correspond to an actual risk or capacity constraint.
## Allow the burst the workflow needs
A client may legitimately load several resources together or reconnect after a brief interruption. Choose burst and sustained allowances using representative traffic, not an example copied from another service.
Record the period used for observation and the client classes included. Check whether average rates hide short spikes or whether the observed sample excluded your busiest approved integration.
For expensive work, consider queue admission or a job interface instead of asking clients to retry the synchronous operation repeatedly. Keep the queue bounded too.
## Make rejection and retries agree
Return the documented status and response shape. If a retry delay is exposed, make it useful for cooperative clients without disclosing unnecessary account or detector details.
Review retry behavior in official clients and partner SDKs. Immediate automatic retries can turn a controlled refusal into additional load. An unknown outcome for a state-changing operation needs retry-safe business semantics, not simply a longer delay.
Do not cache a personalized limit decision as though it were a public page. Check the actual CDN and application behavior around error responses.
## Roll out a policy you can explain
Before enforcement, record which requests a candidate budget would reject. Review a busy legitimate workflow and a suspected abuse sequence. Write down the reason for each selected threshold and how to roll it back.
After enforcement, inspect completed work, rejected requests, and customer impact together. A lower request rate is useful only if the protected operation remains available to the intended clients.
Continue with [API abuse investigation](https://www.centinelanalytica.com/blog/detect-api-bot-abuse) to distinguish policy violations from approved automation. The [API abuse use case](https://www.centinelanalytica.com/use-cases/api-abuse) provides the product context; the budget worksheet gives that evaluation a concrete requirement.
---
Page: https://www.centinelanalytica.com/blog/prevent-promo-code-abuse
# How to prevent automated promo-code abuse
Protect promotion eligibility and redemption from bots. Review repeated claims, concurrent requests, and retry handling without relying on IP alone.
By Frederick Jahn. Published 2026-09-05, updated 2026-10-11.
Prevent promo-code abuse by enforcing eligibility and recording redemption in the same trusted workflow. Bot controls can reduce automated attempts, but a human-looking request must still obey the promotion's rules.
Start by writing those rules precisely. “One use per customer” is not an implementation detail. The application needs a defined eligibility unit, a trusted way to evaluate it, and an intentional response when two requests arrive together.
## Define what one redemption means
Separate the promotion itself from its code. One campaign may have many codes, and one code may be intentionally reusable. Decide whether limits apply per account, order, invitation, organization, or another supported business entity.
Do not substitute an IP address or browser fingerprint for that definition. A household may share a network; one user may change devices. If the business requires identity verification, that needs its own supported process.
Write down when the benefit is reserved, awarded, reversed, and made available again. Cancellation and refund behavior belong in the initial design, not only in incident cleanup.
## Distinguish retries, repeated claims, and races
Consider three hypothetical cases:
- A client retries the same redemption after a timeout because it never received the result.
- An account submits a second, separate redemption after already receiving the benefit.
- Two requests simultaneously observe that the benefit is still available.
The first may be normal recovery behavior. The second may violate eligibility. The third tests whether the implementation enforces the rule under concurrency.
[OWASP's business-logic guidance](https://cheatsheetseries.owasp.org/cheatsheets/Business_Logic_Security_Cheat_Sheet.html) describes atomic checks for sensitive operations and retry-safe external actions. Use those principles to ensure a successful response corresponds to one intended state change.
A bot score cannot repair a race between checking eligibility and recording use.
## Review the redemption lifecycle
| Stage | Question to answer |
| --- | --- |
| Eligibility | Which trusted records determine whether the request qualifies? |
| Reservation | Can simultaneous requests reserve the same allowance? |
| Award | Is the benefit issued only once for the operation? |
| Retry | Can the original result be retrieved without a second award? |
| Reversal | Does cancellation restore eligibility, and under what rule? |
| Audit | Can support explain why a specific operation was accepted? |
Keep public error messages useful without disclosing a detailed anti-abuse rulebook. Preserve the detailed reason in a restricted operational record.
## Look beyond the redemption endpoint
A promotion can be abused through account creation or referral relationships upstream. [OWASP's account-creation threat description](https://github.com/OWASP/www-project-automated-threats-to-web-applications/blob/master/assets/oats/EN/OAT-019_Account_Creation.md) covers accounts created for subsequent misuse.
Connect the signup and redemption timelines using the account reference. Review repeated benefit claims and their actual eligibility outcomes. Similar names or shared browser properties can support investigation, but they do not prove common ownership.
Check every route that awards the same entitlement, including administrative tools and older APIs. A stricter public endpoint does not help if another path records the benefit differently.
## Apply automation controls to a defined failure
For excessive redemption attempts, consider route and account budgets with a clear retry response. For automated signup linked to repeated benefits, review [signup bot detection](https://www.centinelanalytica.com/blog/detect-automated-account-creation). For incorrect eligibility or duplicate awards, fix the application rule.
Keep a legitimate repeated customer, a network timeout, and a cancelled operation in your staging review. Use test entitlements; do not exercise real promotions or payment flows merely to generate evidence.
After rollout, compare invalid awards and legitimate redemption completion. A lower request count may simply mean the promotion is harder to use.
Review the [promo abuse use case](https://www.centinelanalytica.com/use-cases/promo-abuse-and-multi-accounting) alongside the eligibility definition. Assign the traffic-policy change and any redemption-logic repair to their respective owners before the promotion runs again.
---
Page: https://www.centinelanalytica.com/blog/detect-automated-account-creation
# Fake accounts: how to detect automated signups
Fake accounts are registered in bulk, usually by scripts, for spam, fake reviews, or repeated sign-up credits. How to detect them from signup to first use.
By Frederick Jahn. Published 2026-09-05, updated 2026-10-11.
Fake accounts are accounts registered in bulk, usually by scripts, to be misused: to post spam,
claim sign-up credits, or get around per-account limits. The FTC, for example, alleges that
ticket brokers bypassed Ticketmaster's purchase limits
["by creating thousands of Ticketmaster accounts"](https://www.ftc.gov/news-events/news/press-releases/2025/09/ftc-sues-live-nation-ticketmaster-engaging-illegal-ticket-resale-tactics-deceiving-artists-consumers).
To detect them, follow new registrations into verification and first use. A surge of signups is
a lead; the stronger evidence is accounts that repeat a workflow the service does not permit.
This is not the same as attacks on existing accounts, and it is not proof that a person is fake.
"Suspected automated signup" is often a defensible label where "fake person" is not.
## What a fake account is
For a site operator, a fake account is one registered to be misused rather than used the way a customer would. OWASP's name for the automated version is [OAT-019 Account Creation](https://github.com/OWASP/www-project-automated-threats-to-web-applications/blob/master/assets/oats/EN/OAT-019_Account_Creation.md), defined as "create multiple accounts for subsequent misuse," and "fake account" is one of the other names it lists. The misuse it names includes content spam, laundering cash and goods, spreading malware, and skewing SEO, reviews, and surveys. [OWASP's API6 entry](https://owasp.org/API-Security/editions/2023/en/0xa6-unrestricted-access-to-sensitive-business-flows/) adds a referral example, where a script automates registration so that each new user adds credit to the attacker's wallet.
The form itself rarely separates these accounts from real sign-ups. Real users arrive spread over time, finish verification at varied speeds, and do different things first. A batch of fake accounts may share a creation burst, a client configuration, or one repeated first action, often the one that grants something: a credit, a post, a review. Compare the cohort with a normal onboarding period before you read much into any of these.
None of those traits proves fraud, and bot detection doesn't either. It can show that registrations were automated. Whether an account breaks your rules is shown by what it does next, which the rest of this guide follows.
## Follow the account beyond the form
Build a bounded cohort of accounts created during the period under investigation. Compare it with a legitimate onboarding period using the same product flow.
Record registration outcome, verification state, time to first useful action, and the actions that consume resources or award benefits. Use internal account references, not raw passwords, codes, or unnecessary profile data.
The first-use step matters because different abuse goals leave different traces. Accounts created to claim a benefit should be reviewed against that benefit's rules. Accounts created to post repetitive content need a content and workflow investigation. A signup score alone cannot settle both.
## Separate weak similarities from stronger evidence
| Observation | What it supports | What it does not establish |
| --- | --- | --- |
| Shared source IP | A shared network origin | One operator or one person |
| Similar browser properties | A common client configuration | Unique device identity |
| Fast registration | A short completion time | Malicious intent |
| Repeated first-use sequence | A behavior worth grouping | That every grouped account is abusive |
| Repeated violation of a server-side entitlement rule | A concrete policy failure | Who owns the accounts |
Use weak signals to choose a sample for review. Avoid converting a cluster into a permanent denylist without examining the actual behavior.
An enterprise administrator importing users, a classroom onboarding together, or a permitted partner process may legitimately create accounts in a burst. Maintain an explicit owner and scope for those workflows.
## Put controls where misuse becomes possible
Consider separating account creation from permission to perform costly or abuse-prone actions. A newly created account may need to complete a relevant verification step before sending invitations, posting broadly, or receiving a promotional entitlement.
Do not add friction without naming the action it protects. Requiring an extra step on every signup may damage completion while leaving a poorly protected benefit endpoint unchanged.
[OWASP's sensitive-business-flow guidance](https://owasp.org/API-Security/editions/2023/en/0xa6-unrestricted-access-to-sensitive-business-flows/) includes automated registration connected to referral abuse. The general lesson is to review the complete value-granting flow, not just its first screen.
## Give legitimate bulk onboarding a defined route
Approved automation should have bounded permissions and an accountable owner. It should not depend on an unlimited exception for a shared IP or a copied user-agent string.
Document who can invoke the onboarding path, how many accounts it may create, which privileges those accounts receive, and when the exception is reviewed. Keep that path separate from public self-service signup where possible.
For suspicious public registrations, start with a reversible restriction on the affected first-use action. Preserve enough evidence for review before deleting accounts or their data. Some users may be caught by association rather than behavior.
## Measure completed useful work
Track signup completion, verification completion, and the misuse outcome separately. Blocking more registration requests is not sufficient evidence that the service is healthier.
Inspect where rejected users stop and whether they can recover. Also check whether unwanted activity simply moved to older accounts or another entry point.
The [promo abuse and multi-accounting use case](https://www.centinelanalytica.com/use-cases/promo-abuse-and-multi-accounting) is the relevant product context. If benefits are being granted repeatedly, continue with [promo-code abuse prevention](https://www.centinelanalytica.com/blog/prevent-promo-code-abuse) and take the signup cohort into that review.
---
Page: https://www.centinelanalytica.com/blog/how-to-prevent-carding-attacks
# Card testing: how to prevent carding attacks
Card testing, or carding, uses scripts to check stolen card details against payment and card-setup flows. How to spot it and stop it before the provider call.
By Frederick Jahn. Published 2026-09-05, updated 2026-10-11.
To reduce carding, protect the requests that create payment attempts as well as the payment-processing step itself. Use your provider's current fraud controls, constrain automated retries, and check every flow that can validate or save a payment method.
A checkout page is only one possible entry point. An add-card form or a direct API call may reach the same verification operation without following that page's browser flow.
## What card testing is
Card testing is the use of automation to find out which stolen payment cards still work. [Stripe's card-testing guide](https://docs.stripe.com/disputes/prevention/card-testing) describes scripts that submit many cards at once and read the issuer or 3D Secure responses to sort valid cards from dead ones. Stripe lists "carding" as another name for the same activity, along with "account testing," "enumeration," and "card checking."
OWASP splits the abuse into two automated threats, depending on what the attacker already holds:
| OWASP threat | Attacker starts with | What the automation does |
| --- | --- | --- |
| [OAT-001 Carding](https://github.com/OWASP/www-project-automated-threats-to-web-applications/blob/master/assets/oats/EN/OAT-001_Carding.md) | Bulk stolen card data | Makes many payment authorization attempts to see which cards are valid |
| [OAT-010 Card Cracking](https://github.com/OWASP/www-project-automated-threats-to-web-applications/blob/master/assets/oats/EN/OAT-010_Card_Cracking.md) | Card numbers without the expiry date or security code | Guesses the missing values by brute force |
Both reach you through the same operations. According to Stripe, testers use card setup and payments. They prefer card setup because validations and authorizations during setup don't usually appear on cardholder statements. Payment-based testing tends to use small amounts that cardholders are less likely to notice.
The cost continues after the attack stops. Stripe lists disputes on the test charges that succeed, extra authorization and dispute fees, and a higher decline rate that can affect legitimate payments even after the testing ends.
## Identify the payment operation being abused
Whichever name your team uses, look for the repeated verification workflow, not merely a large number of visits to checkout.
Inventory the operations that reach your provider: payment creation, payment-method setup, saved-card changes, and retries. Record which component calls the provider and which checks happen before that call.
Keep raw card details and security codes out of investigation logs. Use provider references and approved internal identifiers rather than making a second store of payment data. [OWASP's logging guidance](https://cheatsheetseries.owasp.org/cheatsheets/Logging_Cheat_Sheet.html) describes sensitive values that should be excluded or appropriately handled in logs.
## Confirm the pattern before imposing a broad block
Compare request counts with provider outcomes over the same time window. Look for the operation where the increase starts, repeated failed attempts, and whether successful customer journeys changed too.
Those patterns are investigation leads. Declines also happen to legitimate customers, and a provider incident or broken integration can produce retries. Check release history and provider status before treating every failure as abuse.
| Observation | Check alongside it | Possible next action |
| --- | --- | --- |
| Verification attempts rise without completed purchases | Setup-card and saved-card routes, not only checkout | Review controls on the operation creating those attempts |
| Failures rise immediately after a release | Request payloads and retry behavior | Fix the integration before tightening traffic policy |
| One session produces repeated attempts | User retries and the provider's transaction references | Bound retries without duplicating the payment operation |
| Attempts arrive through many client sources | Route, timing, and available session evidence | Review a combined pattern rather than one IP threshold |
| Legitimate completions fall after a rule change | Challenges, blocks, and payment errors separately | Revert the new rule if it exceeds the agreed customer-impact limit |
## Use the provider's supported protections
[Stripe's card-testing guidance](https://docs.stripe.com/disputes/prevention/card-testing) recommends current Payment Element or Checkout integrations and explains that protection depends partly on the integration and information it supplies. If you use another provider, review that provider's corresponding controls instead of copying Stripe-specific settings.
Check that your live integration still sends the supported risk information and handles the provider's responses correctly. A customized form can have a different protection path from a hosted checkout. Treat migration or integration changes as a reason to recheck this path.
Do not promise that a provider control catches every attempt. Request-side protection and payment-side decisions observe different things and need to be investigated together.
## Bound retries before expensive operations
Determine whether a request is creating a new payment operation or retrying an existing one. Follow the provider's documented retry and idempotency behavior; do not turn a timeout into an uncontrolled loop of fresh payment attempts.
Use separate budgets for operations with different costs. A page refresh, a payment-method setup, and a payment confirmation are not interchangeable units of work. [OWASP's API resource-consumption guidance](https://owasp.org/API-Security/editions/2023/en/0xa4-unrestricted-resource-consumption/) supports endpoint-specific rate limits and limits on repeated operations.
Review the identity behind each budget. A network address can be shared by many shoppers. An account or session can also make several legitimate attempts. Decide what happens when a required identifier is missing, and test that case explicitly.
Enforce a rejection before the unwanted provider call. Returning an error after completing the expensive operation changes the response without preventing that operation.
## Keep the response useful without leaking unnecessary detail
Give a legitimate customer a clear next step appropriate to the provider result. Avoid displaying internal risk rules or diagnostic detail that the user does not need. Keep detailed provider information in the approved operational tooling.
Measure challenges separately from payment declines. A customer who cannot complete a browser challenge never reaches the same stage as a customer whose payment was declined. Combining those outcomes into one failure count hides where the problem is.
## Validate changes without using real cards
Use the provider's test mode, documented test payment methods, and a staging flow you control. Do not test with real or stolen card details, and do not run an abuse simulation against live payment processing.
Rehearse normal checkout, an ordinary retry, a provider timeout, an interrupted browser challenge, and a request missing the expected session. Check whether the provider was called, whether an operation was duplicated, and whether the customer can recover.
No result is implied by this checklist. It is a method for checking your integration. Choose acceptable customer-impact limits and a rollback owner before the rule reaches production.
## Where Centinel fits
The [checkout and carding use case](https://www.centinelanalytica.com/use-cases/checkout-and-carding-fraud) covers automated abuse around payment workflows. [Bot management](https://www.centinelanalytica.com/bot-management) can supply a request-policy layer at a supported integration, while your provider evaluates the payment operation.
A bot classification is not proof that a card is stolen, and a human-looking browser is not proof that a transaction is safe. Keep those decisions separate. If browser automation is part of the observed pattern, review the [browser evidence model](https://www.centinelanalytica.com/blog/detect-browser-automation) and begin with the specific endpoint producing the unwanted attempts.
---
Page: https://www.centinelanalytica.com/blog/prevent-password-reset-abuse
# How to prevent password-reset flooding and abuse
Protect recovery requests from automated flooding. Bound email delivery, retries, and token handling without letting attackers lock legitimate users out.
By Frederick Jahn. Published 2026-09-05, updated 2026-10-11.
Password-reset flooding abuses the request that starts recovery. The attacker does not need to reset a password successfully to create trouble: repeated requests can burden the delivery system and interrupt the account owner's use of their inbox.
Protect the request stage before it queues a message. Keep recovery usable for the person who actually owns the account, and review token redemption separately from message sending.
[OWASP's forgot-password guidance](https://cheatsheetseries.owasp.org/cheatsheets/Forgot_Password_Cheat_Sheet.html) recommends controls against excessive submissions and warns against changing or locking an account before a valid reset token is presented.
## Follow the recovery timeline
For one synthetic account, identify the events from submission to completion:
- request received;
- request accepted or suppressed;
- delivery job queued;
- provider accepts or rejects the message;
- token presented;
- token accepted, expired, or already used;
- password changed and recovery completed.
Record an opaque correlation reference across those events. Keep the token and full recovery link out of ordinary logs.
A count of successful HTTP responses does not tell you how many messages were sent. Many applications intentionally return the same acknowledgment for requests that produce different internal actions. Monitor the internal stages without exposing their differences through the public response.
## Separate request limits from delivery limits
A request budget controls how frequently clients reach the recovery endpoint. A destination or account budget controls how much delivery work targets one recipient. Both can matter when traffic changes source addresses.
Use a server-side resend policy. A disabled button is helpful interface feedback, but it cannot enforce a delivery allowance on direct requests. Explain when the user can try again without revealing whether an arbitrary account exists.
If the delivery provider is slow, avoid treating each frontend retry as a fresh need for another message. [OWASP's business-logic guidance](https://cheatsheetseries.owasp.org/cheatsheets/Business_Logic_Security_Cheat_Sheet.html) covers retry-safe external actions and explicit workflow state. Apply those principles to the actual semantics of your recovery service.
## Investigate the failure at the right stage
| Symptom | First question |
| --- | --- |
| Requests rise, queued messages do not | Are controls suppressing excess work as intended? |
| Queued messages rise, deliveries stall | Is the provider or queue failing? |
| One recipient gets repeated messages | Is the destination allowance enforced server-side? |
| Users receive links that are already unusable | Are resend and invalidation rules conflicting? |
| Password changes occur without expected confirmation | Is token redemption or identity handling faulty? |
The final row is more serious than delivery flooding and requires its own investigation. Do not describe every reset anomaly as the same attack.
## Make token and resend rules agree
Use unpredictable reset tokens, a bounded lifetime, and single-use redemption, as described in the OWASP recovery guidance. Then decide what happens when another reset is requested.
If every request invalidates the previous token, repeated requests could interfere with a legitimate recovery attempt. If several tokens remain valid, define the allowed overlap and invalidate appropriate outstanding state when recovery succeeds. Review that tradeoff with the identity owner.
Also check a slow user, a delayed email, a link opened on another device, and two browser tabs. Those are ordinary situations a recovery design must handle.
## Respond without handing the attacker a lockout switch
For a suspected flood, preserve a small redacted timeline and narrow the controls to the affected request or delivery path. Monitor recovery completion, delivery backlog, and support complaints together.
Do not ask users to keep requesting more messages as a generic workaround. Give them a clear fallback and let support investigate using the correlation reference rather than forwarding a live reset link.
For unintended account disclosure, use [account-enumeration prevention](https://www.centinelanalytica.com/blog/prevent-account-enumeration). For automated attacks across login and recovery, see the [account takeover use case](https://www.centinelanalytica.com/use-cases/account-takeover). Keep the recovery timeline with the change so the next on-call engineer can tell suppression, delivery failure, and token failure apart.
---
Page: https://www.centinelanalytica.com/blog/prevent-otp-request-abuse
# How to prevent OTP request abuse and SMS pumping
Control automated one-time-code requests before delivery. Review resend budgets, destination patterns, and outcomes without blocking legitimate recovery.
By Frederick Jahn. Published 2026-09-05, updated 2026-10-11.
Prevent OTP request abuse by making an explicit decision before sending each one-time code. Bound resend behavior, monitor which destinations receive messages, and compare accepted sends with completed verification attempts.
A sudden SMS bill or a falling verification rate deserves investigation. Neither, on its own, proves SMS pumping: delivery failures, a broken resend button, and changed signup traffic can produce similar symptoms.
[Twilio's Verify fraud-prevention guidance](https://www.twilio.com/docs/verify/preventing-toll-fraud) describes delayed retries, rate controls, destination restrictions, and monitoring completed verifications relative to sends. Apply the mechanisms your provider supports rather than assuming every service has the same features.
## Map issuance and verification separately
An OTP workflow has at least two decisions: whether to issue a code and whether a submitted code is valid. Protecting the second does not control the cost of the first.
Inventory each issuance route, including signup, login, phone changes, and recovery. Include alternative channels such as voice calls if the interface offers them. A restricted SMS path is incomplete if an unrestricted “call me” action reaches another provider operation.
Keep a correlation ID for the verification attempt. Record delivery state and verification outcome without retaining the code itself. Treat phone numbers as sensitive operational data, with restricted access and a retention limit.
## Read the funnel before blocking destinations
Use counts from the same cohort and time window. A person who receives a code late may verify after the reporting interval ends.
| Stage | Useful question |
| --- | --- |
| Issuance requested | Did one flow suddenly begin requesting more codes? |
| Send allowed | Which application controls accepted the work? |
| Provider accepted | Was a billable or queued operation created? |
| Delivery reported | Is there a channel or destination delivery problem? |
| Verification completed | Did recipients finish the intended workflow? |
Choose and label your denominator. “Verified attempts divided by accepted sends” differs from “verified users divided by signup sessions.” Do not call both numbers conversion without qualification.
## Put resend policy on the server
Show the user when another code can be requested, but enforce the decision server-side. Group related requests under the verification attempt where possible. Also consider account, destination, and source budgets so rotating one identifier does not reset every allowance.
Document what the application does when a send has an unknown outcome. Blindly retrying a provider request can create duplicate delivery work; blindly refusing can strand the user. Use the provider's supported retry semantics and keep an operator-visible state for unresolved delivery.
[OWASP API6](https://owasp.org/API-Security/editions/2023/en/0xa6-unrestricted-access-to-sensitive-business-flows/) explains why a legitimate workflow needs limits when excessive automated use creates harm. OTP sending is a useful place to apply that review.
## Keep a legitimate path through the controls
Before restricting a destination region, compare it with the countries the application actually serves. A new marketing campaign or expansion may change the expected population. Escalate unexplained concentration to the delivery provider with a bounded time window and relevant references.
Check a traveler, a user on a shared network, a delayed delivery, and a retry after reopening the app. These cases should have intentional outcomes. Do not let repeated requests made by somebody else permanently prevent an account owner from recovering access.
Bot classification can inform the pre-send decision in a supported integration. It does not prove who controls a telephone number or whether a carrier relationship is fraudulent.
## Close the investigation with a stage-level result
After a change, inspect accepted sends, actual delivery, completed verification, and support contacts. A reduction in messages is not a success if legitimate verification also stops.
For password-link delivery problems, see [password-reset flooding](https://www.centinelanalytica.com/blog/prevent-password-reset-abuse). The [account takeover use case](https://www.centinelanalytica.com/use-cases/account-takeover) is the relevant broader protection discussion. Bring the issuance path and provider behavior to that review so traffic controls do not become a substitute for correct verification logic.
---
Page: https://www.centinelanalytica.com/blog/prevent-account-enumeration
# How to prevent account enumeration
Find account-existence leaks in login, signup, and password recovery. Review messages, status codes, and timing while keeping recovery usable.
By Frederick Jahn. Published 2026-09-05, updated 2026-10-11.
Account enumeration occurs when someone can determine which accounts exist from the application's responses. Prevent it by reviewing the whole response contract across login, signup, and recovery, then removing disclosures the product does not intend to make public.
A generic error message is useful, but it is only one part of that contract. Status codes, redirects, response size, and processing time can still separate an existing account from a nonexistent one.
[OWASP's account-enumeration testing guide](https://owasp.org/www-project-web-security-testing-guide/latest/4-Web_Application_Security_Testing/03-Identity_Management_Testing/04-Testing_for_Account_Enumeration_and_Guessable_User_Account) also notes an important scope decision: some products deliberately make usernames public. Define what should remain private before calling every discoverable identifier a vulnerability.
## Decide which facts the interface may disclose
Write down the intended behavior for an existing account, a nonexistent account, and an account that cannot currently sign in. Include email aliases and federated identities if they are supported.
A public profile directory may intentionally disclose a username while keeping its associated email private. A support-assisted workflow may disclose information after identity verification that the public recovery endpoint should not disclose.
Those are different policies. Do not attempt to solve them with one error string copied across every screen.
## Compare the complete response
Use synthetic accounts in an environment you are authorized to test. Keep the request count low and avoid sending recovery messages to real users.
| Surface | Compare | Common review gap |
| --- | --- | --- |
| Login | Body, status, and redirect | “Wrong password” differs from “unknown user” |
| Password reset | Acknowledgment and processing path | One response waits for a delivery provider |
| Signup | Duplicate-account handling | Email existence is disclosed without a product decision |
| Identity lookup | Public fields and response shape | A helper endpoint bypasses the main form's policy |
| Throttling | Behavior before and after limits | Limits reveal account state indirectly |
The table is a review method, not a checklist whose completion proves the absence of every side channel. Record what you actually compared.
## Make timing less dependent on account existence
If one branch performs expensive work and the other immediately returns, identical visible text may conceal a predictable difference. Review the server-side path before adding arbitrary sleep delays.
For password recovery, [OWASP recommends consistent responses and response times](https://cheatsheetseries.owasp.org/cheatsheets/Forgot_Password_Cheat_Sheet.html), with asynchronous processing among the available approaches. Keep that processing bounded: a uniform acknowledgment must not create an unlimited delivery queue.
Measure several synthetic requests under comparable conditions. One slow request is not a timing channel, and one fast test is not proof that the paths are equivalent. Capture the application version and observation point so another engineer can repeat the comparison.
## Keep useful feedback behind a trusted boundary
People still need to recover access. A neutral acknowledgment can explain that instructions will be sent if the account is eligible, while providing a safe support route for people who cannot receive them.
Avoid changing account state simply to make public responses look consistent. The recovery service should not lock an account because someone requested a reset for its address.
Where the product intentionally discloses an existing account during signup, document that decision and the reason. Then bound automated probing rather than pretending the disclosure has been eliminated.
## Use bot controls as a supporting layer
Rate limits and automation controls can reduce repeated probing, but they do not repair a response that reveals account existence in one request. Fix the disclosure first, then check whether public endpoints can be queried excessively.
Keep controls consistent across web and API paths. A protective frontend does not help when a second endpoint exposes the same lookup directly.
Review [password-reset abuse](https://www.centinelanalytica.com/blog/prevent-password-reset-abuse) for delivery flooding, which is a related but different problem. The [account takeover use case](https://www.centinelanalytica.com/use-cases/account-takeover) provides the product context for automated account attacks. Bring the response comparison to the implementation review so disclosure fixes and traffic controls remain separate work items.
---
Page: https://www.centinelanalytica.com/blog/how-to-prevent-account-takeover
# How to prevent account takeover
Protect login, password recovery, and account changes with layered controls. Learn where bot protection helps and which identity checks remain essential.
By Frederick Jahn. Published 2026-09-05, updated 2026-10-11.
Account-takeover prevention needs controls across the account lifecycle: authentication, recovery, active sessions, and sensitive changes. Start with stronger authentication and abuse controls on login, then check whether password resets or existing sessions provide another route into the account.
Bot protection helps with automated attempts. It cannot establish that a password was stolen or that the person using a valid session is its rightful owner.
## Map the account flows before changing login rules
List every entry point that can create or restore account access: password login, federated login, recovery links, one-time codes, and support-assisted recovery. Include APIs and mobile flows that reach the same account service.
Then list the actions that make a compromise harder to reverse, such as changing an email address, enrolling a new authenticator, or replacing recovery details. A successful login should not silently grant a permanent right to perform all of them.
For each flow, record its existing limits, what gets logged, the owner, and how a legitimate user recovers from a wrong decision. This inventory is a planning tool, not a claim that all takeover incidents follow one pattern.
## Make automated credential attacks harder
Credential stuffing reuses compromised username/password pairs; password spraying tries a small set of passwords across many accounts. [OWASP's credential-stuffing guidance](https://cheatsheetseries.owasp.org/cheatsheets/Credential_Stuffing_Prevention_Cheat_Sheet.html) recommends layered defenses, including multifactor authentication, rather than relying on one request property.
Consider the unit each limit protects. An account-level budget can identify repeated attempts against one account. An IP-level budget covers a network source. Neither describes the whole problem when many accounts or shared networks are involved. Review both with the identity provider's existing controls before adding another layer.
Avoid a permanent lockout that an unauthenticated requester can trigger against any account. The protection should slow abuse without becoming an easy way to deny legitimate access. Define retry behavior and a recovery route the account owner can actually use.
## Protect recovery with the same care as login
A strong login flow is less useful if recovery bypasses its protection. [OWASP's recovery guidance](https://cheatsheetseries.owasp.org/cheatsheets/Forgot_Password_Cheat_Sheet.html) calls for consistent account-existence responses, rate-limited requests, and reset tokens that are unpredictable, expire, and can be used only once.
Test a reset link after it has been used and after it has expired. Check what happens to existing sessions after a password reset, and make the intended behavior explicit. Protect code-sending endpoints as well as code-verification endpoints; they do different work and may have different abuse costs.
Keep support-assisted recovery in the inventory. An application control does not verify the identity of someone persuading a support agent to change an account.
## Recheck identity for sensitive changes
[OWASP's authentication guidance](https://cheatsheetseries.owasp.org/cheatsheets/Authentication_Cheat_Sheet.html) recommends reauthentication for sensitive actions and after risk events, with session handling appropriate to the result. Review email changes and authenticator enrollment as distinct operations, not incidental profile edits.
Prefer the stronger authentication options your identity system supports, including phishing-resistant options where available. Keep the enrollment and recovery path in the same review: adding a stronger factor does not help if an attacker can replace it through a weaker flow.
Notify the existing account contact about consequential changes through an appropriate channel. Make the response to an unexpected change clear. Notifications are a detection and recovery measure; they do not undo a change by themselves.
## Use an account-flow control matrix
This worksheet is a suggested review method. Adapt the controls to the identity system and application you operate.
| Flow | Evidence worth reviewing | Control to check | Legitimate-user check |
| --- | --- | --- | --- |
| Login | Account attempt history, available client evidence, authentication result | Identity-provider protections and bounded retries | Shared networks and ordinary mistyped passwords |
| Recovery request | Account-target frequency and sender workload | Consistent responses and bounded delivery attempts | A user who requests the message again |
| Recovery completion | Token validity, expiry, and prior use | Single-use token handling and intended session invalidation | Expired links and lost sessions |
| Email or factor change | Current authentication strength and recent risk events | Reauthentication and change notification | A legitimate user moving devices |
| Existing session | Session state and sensitive actions | Revocation and reauthentication where required | Multiple valid devices and interrupted flows |
## Test the full journey, not only the blocked request
Use test accounts and synthetic credentials in an authorized staging environment. Rehearse an ordinary login, a failed attempt followed by success, an expired reset, a reused reset, and a sensitive change. Separately simulate bounded automated attempts against those test accounts.
Record the client response, authentication result, session state, and whether the user can finish. A request classified as automated may still be an approved test or integration. An allowed browser request may still contain invalid credentials.
Choose rollback criteria before enforcement. If a new bot rule breaks login, revert that rule while retaining password, recovery, and session protections. Do not use a broad authentication bypass as the rollback plan.
## Where bot protection belongs
The [account-takeover use case](https://www.centinelanalytica.com/use-cases/account-takeover) covers the automated-traffic side of the problem. Centinel's [bot-management overview](https://www.centinelanalytica.com/bot-management) explains the evidence and policy layer; the [browser-automation guide](https://www.centinelanalytica.com/blog/detect-browser-automation) goes deeper into client observations.
Place that layer around the supported requests your application can enforce. Keep password validation, authenticator verification, account permissions, and session revocation with the systems responsible for identity. Begin with the weakest account flow found in the inventory, then verify that ordinary users can still complete it.
---
Page: https://www.centinelanalytica.com/blog/detect-credential-stuffing
# Credential stuffing: what it is, how to detect it
Credential stuffing is automated login testing with username and password pairs stolen from other sites. How it differs from brute force and how to detect it.
By Frederick Jahn. Published 2026-09-05, updated 2026-10-11.
Detect credential stuffing by examining how login attempts spread across accounts and what happens after authentication. A rise in failures is a starting point. It is not enough to establish that stolen credentials are being tested.
## What credential stuffing is
Credential stuffing is automated login testing with username-and-password pairs stolen from another site. OWASP catalogs it as [OAT-008 Credential Stuffing](https://github.com/OWASP/www-project-automated-threats-to-web-applications/blob/master/assets/oats/EN/OAT-008_Credential_Stuffing.md): "mass log in attempts used to verify the validity of stolen username/password pairs." It works because people reuse passwords, so a pair leaked by one service can open an account on another.
It is often confused with two other login attacks. [OWASP's credential-stuffing cheat sheet](https://cheatsheetseries.owasp.org/cheatsheets/Credential_Stuffing_Prevention_Cheat_Sheet.html) separates them by what the attacker tests. The last column is our reading of what each one tends to look like in login records:
| Attack | What is tested | Shape to expect in login records |
| --- | --- | --- |
| Credential stuffing | Pairs from a breach of another site | Many accounts, few attempts on each |
| Brute force | Many passwords against one account | One account, many attempts |
| Password spraying | One weak password against many accounts | Many accounts, few attempts on each |
Brute force and spraying guess. OWASP files both under [OAT-007 Credential Cracking](https://github.com/OWASP/www-project-automated-threats-to-web-applications/blob/master/assets/oats/EN/OAT-007_Credential_Cracking.md), while stuffing "does not involve any brute-forcing or guessing of values." Stuffing and spraying can look alike from the account side, because neither needs many attempts per account.
That matters for detection. [NIST SP 800-63B](https://pages.nist.gov/800-63-4/sp800-63b.html#throttle) requires a verifier to cap consecutive failed attempts on one account at no more than 100. A per-account cap like that does little against an attack that tries each account once or twice, so the source and outcome views below matter more. Your logs may support “suspected automated credential attack” before they support a more precise label.
## Start with outcomes, not a list of bad IPs
Choose a bounded time window and a known login route. Include password login on the web, equivalent mobile endpoints, and older API routes still in use. Keep federated login failures separate if a different identity provider owns their meaning.
Collect timestamps, pseudonymous account references, the trusted network observation, outcome categories, challenge decisions, and a request correlation ID. Do not record passwords, reset tokens, or session cookies. You do not need the attempted password to begin an account-spread investigation.
Separate invalid credentials from account locks, missing second factors, dependency failures, and application errors. Combining those outcomes into “failed login” can turn an identity-provider outage into an apparent attack.
## Compare the account and source views
An account view asks whether one account is receiving attempts from many sources. A source view asks whether one client or source is touching many accounts. Neither should disappear inside a combined account-and-IP counter.
[OWASP's bot-management guidance](https://cheatsheetseries.owasp.org/cheatsheets/Bot_Management_and_Anti-Automation_Cheat_Sheet.html) describes independent account and source budgets. That distinction also helps investigation: one source can spread attempts across accounts, while a distributed attack can concentrate on one account.
| Observation | Investigate | Alternative explanation |
| --- | --- | --- |
| Many accounts, similar failed outcomes | Automated credential attempts | A broken enterprise integration |
| One account, many sources | Targeted distributed attempts | Shared account or unusual client behavior |
| Failures followed by a new successful session | Whether the session is legitimate | User corrected a saved password |
| More second-factor failures | Authentication-stage abuse | Delivery or authenticator problem |
| Failures after a deployment | Changed request handling | Client or identity-service regression |
Treat the alternatives seriously. They determine whether the response should target an attacker, an integration, or your own release.
## Inspect the successful sessions
A detection process that reviews only failures misses the event that matters most: a suspicious attempt that obtains usable access. Follow the successful authentication into the resulting session where your retention policy and investigation authority permit.
Check whether the session performs sensitive actions such as changing recovery details or requesting unusual bulk data. Those actions warrant review; they do not independently prove compromise.
Keep the classification and the evidence together. “Known stolen credentials” requires evidence of that fact. “Automated attempts followed by a disputed session” may be the more accurate incident description.
## Contain narrowly and preserve recovery
Start with the affected authentication path and the available account/source controls. Consider stronger authentication for risky transitions, and coordinate session handling with the identity team when compromise is suspected. OWASP calls multi-factor authentication ["by far the best defense"](https://cheatsheetseries.owasp.org/cheatsheets/Credential_Stuffing_Prevention_Cheat_Sheet.html) against credential stuffing and password spraying, and [NIST SP 800-63B](https://pages.nist.gov/800-63-4/sp800-63b.html#passwordver) requires checking new passwords against a blocklist that includes passwords from previous breaches.
Do not turn attacker-generated failures into permanent account lockouts without a recovery design. A response that makes real users unable to sign in may accomplish the attacker's disruption goal.
Track ordinary login completion alongside rejected attempts. Check password managers, mobile clients, a user correcting a typo, and a shared-network population. A falling failure count can also mean the route is broken for everybody.
The [account takeover use case](https://www.centinelanalytica.com/use-cases/account-takeover) frames the automation-control discussion. For the broader account lifecycle, use [account-takeover prevention](https://www.centinelanalytica.com/blog/how-to-prevent-account-takeover); this investigation should feed that plan rather than replace it.
---
Page: https://www.centinelanalytica.com/blog/investigate-bot-detection-false-positives
# How to investigate bot-detection false positives
Trace a legitimate user's block to the decision that caused it. Separate classification errors, policy mistakes, and integration failures.
By Frederick Jahn. Published 2026-09-05, updated 2026-10-11.
Investigate a suspected bot-detection false positive by connecting the user's failed action to the exact decision that stopped it. Then determine whether the problem is classification, policy, or integration.
Those are different failures. A legitimate person classified as automated may be a detection error. An approved integration correctly classified as automated but blocked anyway is a policy problem. A challenge that succeeds but loses its state is an integration problem.
Calling all three “bad bot scores” makes the repair less precise.
## Collect enough evidence to find the request
Ask for the affected URL or action, approximate time with timezone, browser or client version, and a non-secret request reference if the product exposes one. Record whether the failure is reproducible.
Do not ask a customer to send a password, session cookie, challenge token, or unredacted network archive through ordinary support channels. Start with the minimum information needed to locate the event in restricted operational logs.
Also record what the user expected to happen. “The website blocked me” is less actionable than “the page loaded, but submitting the recovery form returned a challenge.”
## Identify the decision source
Trace the request through the edge, bot-control integration, application, and any downstream service relevant to the action. Determine which component produced the blocking response.
| Finding | Likely work item |
| --- | --- |
| Legitimate user receives an incorrect automation label | Detection evidence review |
| Approved automation is blocked under a general rule | Policy scope correction |
| User exceeds an intended resource allowance | Allowance or user-experience review |
| Challenge succeeds but the action still fails | Integration-state diagnosis |
| Application returns its own error | Application investigation |
The table is a triage aid, not a substitute for evidence. Several failures can coexist.
[Cloudflare's traffic-review guidance](https://developers.cloudflare.com/use-cases/solutions/stop-malicious-bots/) illustrates the importance of inspecting traffic and legitimate-user impact when tuning controls. Use the equivalent decision records in your actual stack.
## Keep the label separate from the action
Record the observed evidence, classification, policy rule, and enforced response separately. A response such as “block” does not explain why the system chose it.
Check missing observations as well as suspicious ones. If expected browser evidence never arrived, investigate whether the browser, network, consent flow, or integration prevented collection. Absence of an observation should not be silently described as a positive detection signal.
If a rule uses source-network context, inspect shared networks and trusted proxy handling. Avoid treating one address as a stable person or assuming a user-agent proves client identity.
## Correct the smallest supported scope
If the issue is a specific rule, route, or client integration, change that scope first. Give temporary exceptions an owner, expiry, and review condition. Keep an audit record of the evidence that justified them.
Do not introduce a blanket exception based only on a user-controlled header. A workaround should not create a new unverified path around protection.
For repeat verification failures, follow the [challenge-loop guide](https://www.centinelanalytica.com/blog/fix-bot-challenge-loops). For a browser signal dispute, use the [browser-automation detection guide](https://www.centinelanalytica.com/blog/detect-browser-automation).
## Report impact with an honest denominator
A support ticket count is not a false-positive rate. It measures reported problems, and many affected users may never contact support.
[NIST's measurement guidance](https://airc.nist.gov/airmf-resources/playbook/measure/) calls for evaluating errors against ground truth in the deployment context. For your review, define the labeled sample and the unit being counted before calculating a rate.
Keep ordinary users and approved automation visible as separate groups. Check that the correction restores their intended actions without quietly allowing the disputed abusive behavior.
The [bot-management overview](https://www.centinelanalytica.com/bot-management) frames the product evaluation. A useful handoff contains one redacted request trace, the responsible decision, the narrow correction, and the observed recovery—not a larger allowlist with no explanation.
---
Page: https://www.centinelanalytica.com/blog/how-to-identify-bot-traffic-in-google-analytics
# How to identify bot traffic in Google Analytics 4
Spot bot traffic in GA4: rule out collection mistakes, build a slice of suspicious sessions, and confirm it against request logs before you block anything.
By Frederick Jahn. Published 2026-09-05, updated 2026-10-11.
Use Google Analytics to find suspicious traffic patterns, then check those patterns against your site's request logs. A sudden jump in sessions, unusual geography, or low engagement can justify an investigation. None of those observations proves that the visitors are bots.
The useful question is narrower: what changed, which routes were affected, and does the server-side evidence support the same explanation?
## Know what GA4 already excludes
Google says GA4 automatically removes traffic identified as known bots and spiders. You cannot disable that exclusion or see how much traffic it removed. The remaining traffic is therefore not a verified count of human visits. See Google's [known bot-traffic exclusion documentation](https://support.google.com/analytics/answer/9888366).
GA4 is also not an access log. Its reports depend on collection and processing. Google's [measurement overview](https://support.google.com/analytics/answer/12159447?hl=en) explains how website tags send interaction data for reporting. An HTTP request that never produces an analytics event may still reach your application and consume resources.
Keep that distinction in mind when investigating [web scraping](https://www.centinelanalytica.com/blog/what-is-web-scraping): a quiet analytics report does not establish that your content is receiving little automated traffic.
## Rule out collection and data-hygiene causes
Before assigning a bot label, check what shipped near the start of the anomaly. Look for a tag added twice, a changed consent flow, a new campaign, a monitoring job, or an event sent from both the browser and the server.
Google's [Measurement Protocol](https://developers.google.com/analytics/devguides/collection/protocol/ga4) can send events over HTTP for server-side and offline interactions. It supplements automatic collection. If your team uses it, inspect those senders as well as the browser tag; an event does not necessarily describe a page load.
Ask who owns internal and test traffic. GA4 supports internal-traffic filters, but an active exclusion permanently removes matching incoming data from processing, including BigQuery export. Test the rule before activating it, following [Google's internal-traffic filter guidance](https://support.google.com/analytics/answer/10104470?hl=en). A broad exclusion can hide the evidence you were trying to understand.
## Build a diagnostic slice
Choose a fixed time window around the change and compare it with a comparable earlier period. Start with the affected landing routes and traffic sources. Add device or geography only when the dimension helps explain the difference.
Use a worksheet like this rather than a universal bot threshold:
| Observation | Plausible alternative | Next check |
| --- | --- | --- |
| Sessions rose after a release | Duplicate collection or a changed event definition | Compare tag versions and event senders |
| One route has many short visits | A campaign or a page that answers a quick question | Check referrals, campaign changes, and request activity |
| Events arrive at regular intervals | A scheduled internal job | Compare the schedule with monitoring and integration owners |
| Unfamiliar geography dominates | A legitimate audience change or proxy use | Check campaign scope and network evidence; do not block a country from this alone |
| Many events have no plausible page sequence | Direct event submission or instrumentation gaps | Inspect Measurement Protocol sources and collection coverage |
Record what would disprove your first explanation. If the anomaly disappears when duplicate events are removed, you have a measurement issue to fix before considering enforcement.
[OWASP's skewing taxonomy](https://github.com/OWASP/www-project-automated-threats-to-web-applications/blob/master/assets/oats/EN/OAT-016_Skewing.md) describes automation used to manipulate metrics. It is one possible cause, not the default explanation for an unusual chart.
## Correlate the slice with request evidence
Compare the same routes and time window in your edge or application logs. Record response codes, request volume, and available session or crawler evidence. Establish whether the requests actually reached the protected application or were answered earlier by a cache or enforcement layer.
Do not match people by geography and timestamp. Those fields can describe many unrelated visits. If you lack an approved shared identifier between the two systems, compare aggregates and call the result a cohort-level comparison. Keep unmatched events and unmatched requests separate.
For a supported identifier already deployed by your team, check how it is created, whether it survives retries, and whether one identifier maps to several events or requests. A successful join still does not establish human identity. Do not add personal data to Analytics to make a join easier.
The [browser-automation guide](https://www.centinelanalytica.com/blog/detect-browser-automation) explains the request-side evidence layer. Also distinguish declared crawlers from browser agents using the [AI traffic taxonomy](https://www.centinelanalytica.com/blog/what-is-ai-agent-traffic); their collection behavior need not be the same.
## Run a controlled analytics-to-request comparison
If the collection path remains unclear, test it on a site and analytics property you control. The following is a reader exercise, not a completed experiment or a requirement to accept this diagnostic method.
The optional plan `ga4-request-correlation-v1`, dated `2026-09-04`, uses 20 sessions per cohort: `ordinary-browser`, `known-approved-bot`, `direct-measurement-protocol`, and `browser-automation`. That is 80 planned sessions, with no measured results reported here.
Use ordinary test browsers, an authorized crawler, your own synthetic event sender, and test-owned browser automation. For each, record whether the request exists, whether the event appears after processing, whether a valid join exists, and what remains unknown. Do not assume an approved crawler appears on Google's undisclosed exclusion inputs, and do not assume every accepted event will appear in every report.
The purpose is to find differences in your measurement paths, not to calculate a universal bot-detection rate.
## Separate reporting cleanup from request prevention
Fix a collection error in the measurement setup. Use a reporting comparison or tested filter for a reporting problem. If request evidence shows unwanted automation, apply the relevant access policy where the application receives or serves the request.
An analytics filter cannot stop a client downloading a page. Conversely, a block at the edge can change what reaches both the application and the analytics tag. Keep the change log for those systems together so the next traffic drop has an explanation.
## Where Centinel fits
Centinel's [bot-management overview](https://www.centinelanalytica.com/bot-management) describes the request and browser evidence used to evaluate automated traffic. This guide does not assume a built-in GA4 join or a one-click bot cleanup feature. Correlation depends on your instrumentation and the observations available at the integration point.
Once the investigation identifies unwanted collection, use the [scraping and crawler use case](https://www.centinelanalytica.com/use-cases/scraping-and-ai-crawlers) to review response options. Start with the affected route and retain the evidence behind the decision.
---
Page: https://www.centinelanalytica.com/blog/fix-bot-challenge-loops
# CAPTCHA loop: why it repeats and how to fix it
A CAPTCHA challenge is a check a site runs before it lets a request through. Why visitors get stuck in repeat challenges, and how to find the failing step.
By Frederick Jahn. Published 2026-09-05, updated 2026-10-13.
A CAPTCHA challenge is a check a website runs before it lets a request through. It can be a visible puzzle, a checkbox, or an invisible test that runs in the browser. A visitor who passes should get a token or cookie and continue without seeing the check again.
A bot-challenge loop happens when the user repeatedly reaches verification without obtaining usable access to the intended page or action. Diagnose the transition that fails: challenge loading, completion, server validation, state persistence, or the next request's policy decision.
Do not begin by disabling protection for every visitor. First capture one reproducible failure with its time, route, browser version, and request reference.
[Cloudflare's challenge troubleshooting documentation](https://developers.cloudflare.com/cloudflare-challenges/troubleshooting/challenge-solve-issues/) lists browser and network conditions that can interrupt completion. A failed challenge is therefore not proof that the visitor is automated.
## Follow one attempt all the way through
Use an authorized test session and the normal public entry point. Record whether the challenge loads, whether the completion callback runs, whether the server receives evidence, and which response follows.
Keep tokens and cookies out of support tickets. If a network trace is necessary, redact it and share it only through the approved investigation channel. A screenshot of a successful widget may help describe the symptom, but it does not show server validation.
| Last successful transition | Inspect next |
| --- | --- |
| Page displayed | Script loading and browser execution |
| Challenge completed | Evidence submission to the application |
| Evidence received | Server verification result and timeout handling |
| Verification accepted | State set for the correct browser context |
| State sent on next request | Policy ordering and scope |
| Intended page loaded | Whether a later action starts verification again |
This sequence narrows the search. Without it, client and server teams can both report that their part “worked” while the user remains stuck.
## Check expiry and duplicate submission
If the integration uses a one-time token, inspect when it was issued and where it was consumed. A long form-completion time, double submit, or retry can create a different state from the simple happy path.
[Turnstile's documented tokens expire and can be validated only once](https://developers.cloudflare.com/turnstile/get-started/server-side-validation/). Check the actual provider's semantics; do not reuse a consumed token indefinitely or treat every duplicate as an attack.
Separate a failed verification from an unknown network outcome. Review the provider's supported retry mechanism and the application's handling of a response that never arrives.
## Check where browser state is stored and sent
For cookie-based clearance or session state, inspect the attributes and the host that receives the next request. [MDN's Set-Cookie reference](https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/Set-Cookie) explains that a host-only cookie is sent to the host that set it, not automatically to sibling subdomains.
Also review path, secure transport, expiry, and cross-site behavior as applicable. Do not broaden cookie scope or remove security attributes simply to make the loop disappear. Establish which origin needs the state and why.
If a reverse proxy changes hosts or redirects between entry points, follow that path explicitly. State that exists in the browser may still be absent from the request the enforcement component evaluates.
## Inspect the next rule, not only the first check
A completed challenge can be followed by a separate rule that triggers another challenge. Capture the decision source and configuration version for both requests.
Verify whether the successful state applies to the intended route and action. A token for one action should not silently become permission for every protected operation.
Be careful with console errors. Cloudflare documents diagnostic requests that can fail without blocking challenge completion. Correlate an error with the failed transition rather than assuming the first red line is the cause.
## Verify the repair under ordinary failures
Check a slow form submission, a refresh, a duplicate click, and the supported cross-origin path. Confirm the user reaches the original action without losing necessary form state.
If a temporary exception is necessary, make it narrow, expiring, and owned. Preserve unrelated authentication and authorization controls.
## Check what your current protection misses
Repeated challenges deserve investigation alongside the automated traffic you want to stop. Centinel works alongside your existing CDN or WAF and applies allow, challenge or block policy through your existing edge.
[Request a free site audit](https://www.centinelanalytica.com/) to see which scraping tools get through your current setup. We run real scraping tools against your site, review the results, and email you the report.
Use [false-positive investigation](https://www.centinelanalytica.com/blog/investigate-bot-detection-false-positives) when the underlying classification is disputed. For broader product behavior, see the [bot-management overview](https://www.centinelanalytica.com/bot-management). Close the issue with the failed transition and the repaired behavior, so a future loop does not restart the investigation from scratch.
---
Page: https://www.centinelanalytica.com/blog/captcha-alternatives
# CAPTCHA alternatives for your workflow
Compare non-interactive checks, proof of work, authentication, and workflow controls. Choose a CAPTCHA alternative by what it protects and how users recover.
By Frederick Jahn. Published 2026-09-05, updated 2026-10-11.
The best CAPTCHA alternative depends on what the protected action requires. A public form needs different controls from a signed-in account change or a machine-to-machine API.
Start by naming the abuse: spam submissions, automated account creation, credential attempts, or excessive data access. Then choose a control that changes that outcome and preserves a usable path for legitimate users.
The [W3C's Group Draft Note on CAPTCHA accessibility](https://www.w3.org/TR/turingtest/) discusses how visual, audio, and cognitive tasks can exclude people. Treat accessibility as part of the workflow evaluation, not a label inferred from the absence of a picture puzzle.
## Compare mechanisms, not slogans
| Approach | Useful question it can help answer | What remains separate |
| --- | --- | --- |
| Non-interactive browser checks | Does this request have evidence consistent with automation? | Identity and permission to act |
| Proof of work | Has a client performed required computation? | Whether the client is human or the action legitimate |
| Stronger authentication | Does the requester control the required authenticator? | Whether authenticated use is abusive |
| Rate and resource limits | Is the client within its allowance? | Whether low-volume misuse is occurring |
| Workflow validation | Is this action valid in the application's current state? | How to manage excessive valid attempts |
These approaches can coexist. Avoid asking one signal to answer every question.
A honeypot field may be useful against simple form automation, for example, but should not become the only reason to reject a user without checking autofill and accessibility behavior. Its usefulness depends on the form and the clients that reach it.
## Check what happens on the server
A widget that reports success in the browser is not a complete enforcement design. The protected endpoint must validate the relevant evidence and decide what action it permits.
For a documented example, [Cloudflare requires server-side validation of Turnstile tokens](https://developers.cloudflare.com/turnstile/get-started/server-side-validation/). Those tokens expire and are single-use. This illustrates an integration requirement, not a ranking of CAPTCHA products.
Ask equivalent questions of any candidate: what evidence reaches the server, what can be replayed, what expires, and what happens if verification is unavailable? Keep the answer tied to the supported deployment.
## Evaluate the entire user journey
Choose one real workflow and define what successful completion means. Include the initial page, form submission, errors, retries, and the return path after verification.
Review keyboard-only interaction, screen-reader announcements, a slow connection, a user who takes time to complete the form, and a supported mobile browser. Also include approved automation if it legitimately uses the endpoint.
Do not measure only the challenge's completion screen. A person can finish a check and still fail the business action because the token expires, the form loses state, or the response format is wrong.
For an interstitial placed before content, see the [interstitial protection guide](https://www.centinelanalytica.com/blog/interstitial-page-content-protection). That placement has different consequences from checking one form submission.
## Decide what a failure should do
Different actions justify different recovery paths. A refused public search may be retried later. A blocked account-recovery attempt needs a usable support route. An API integration needs a documented machine-readable response.
Write the failure behavior before rollout. Avoid an endless sequence of “try again” screens or an unrestricted fallback that quietly removes the protection.
Measure the behavior of the actual integration under an expired token, duplicate submission, and verification timeout in a controlled environment. Do not assume the vendor handles application state that your server owns.
## Make the decision with evidence from your workflow
Compare the candidates using permitted action completion, unwanted action completion, integration effort, and operator visibility. Keep data collection and retention questions in the evaluation even when no interactive puzzle appears.
Centinel's [bot-management overview](https://www.centinelanalytica.com/bot-management) is a relevant starting point when the problem requires broader automated-traffic assessment. Use the [proof-of-concept guide](https://www.centinelanalytica.com/blog/bot-protection-proof-of-concept) to turn the workflow into evaluation criteria.
Choose the mechanism that addresses the identified abuse and gives legitimate users a clear way through. A quieter screen is useful; it is not the entire acceptance test.
---
Page: https://www.centinelanalytica.com/blog/bot-protection-proof-of-concept
# How to run a bot-protection proof of concept
Build a buyer-owned bot-protection evaluation with clear workflows, labeled traffic, and enforcement checks. Includes a practical scorecard.
By Frederick Jahn. Published 2026-09-05, updated 2026-10-11.
A bot-protection proof of concept should answer a decision: can this deployment control the unwanted automation affecting your service while keeping intended workflows usable?
Choose the workflows before choosing the traffic sample. A tool that identifies automation in a dashboard still needs an enforcement path that changes what the client can do. An effective block also needs to happen before the protected work or data is delivered.
This guide is an evaluation plan to adapt, not a report of a completed test.
## Write acceptance criteria before the demonstration
Select a small set of important journeys: a public page, a login or signup action, and any API or bulk-access path relevant to the problem. Do not include an endpoint merely because it is easy to test.
For each journey, name the unwanted outcome, the permitted users and automation, the enforcement location, and the behavior during a verification failure or timeout.
[NIST's AI measurement guidance](https://airc.nist.gov/airmf-resources/playbook/measure/) recommends context-specific evaluation against ground truth and documentation of what is not measured. That is a useful discipline whether a proposed bot-control system uses rules, statistical models, or both.
## Keep detection and enforcement separate
Use a scorecard like this for each deployment under consideration.
| Question | Evidence to retain |
| --- | --- |
| Was automation identified? | Decision record linked to the labeled attempt |
| Was the intended response enforced? | Response and protected action outcome |
| When was it enforced? | Position relative to data delivery or expensive work |
| Could an approved client complete its task? | End-to-end workflow result |
| What happened when evidence was unavailable? | Controlled failure observation |
| Could the operator explain and reverse a decision? | Investigation and rollback record |
Do not combine every row into one “accuracy” number. A classification failure and an enforcement integration failure require different fixes.
## Choose traffic whose purpose is known
Include ordinary user journeys, approved automation, and authorized test automation that represents the problem being evaluated. Label the test case independently of the product's output.
Record client version, configuration, observation point, and relevant environment details. Use synthetic accounts and data. Obtain permission for the endpoints and any external services involved, and keep traffic bounded.
A sample of known automation tells you something about that sample. It cannot establish legitimate-user impact if no legitimate workflows were included. Likewise, approved automation must not disappear into a category labeled simply “bad bots.”
Keep a separate set of cases for checking a configuration after tuning it. Repeating only the exact cases used to choose the rule can make a fragile configuration look more convincing.
## Test the failure paths that affect users
Include a slow response, an expired verification state where applicable, a client retry, and the supported behavior when a dependency is unavailable. Agree on the intended fallback before running those checks.
For interactive or browser-based verification, inspect keyboard use and supported assistive-technology workflows. The [W3C CAPTCHA accessibility discussion](https://www.w3.org/TR/turingtest/) explains why alternative presentation alone does not settle usability for every user.
For APIs, verify response formats and retry behavior. A browser-oriented success path does not establish a usable machine-client integration.
## Make the result auditable
Preserve a bounded aggregate report plus restricted references to the underlying attempts. Include failed and indeterminate cases. Do not silently drop a run because a client, network, or integration error made the outcome inconvenient.
Define the unit for each calculation: requests, sessions, or completed actions. Keep numerator and denominator visible. Record configuration changes between runs so results do not mix different policies.
End with an operational decision: proceed on the evaluated routes, repair an integration and repeat selected checks, or reject the fit. Give any unresolved item an owner and a specific next observation.
Use the [bot-management overview](https://www.centinelanalytica.com/bot-management) to scope a Centinel evaluation. Keep the scorecard under your team's control and resolve the open workflow failures before making a rollout decision.
---
Page: https://www.centinelanalytica.com/blog/protect-site-search-from-bots
# How to protect site search from bot abuse
Investigate automated search queries, deep pagination, and expensive filters. Set search-specific budgets while preserving useful search and autocomplete.
By Frederick Jahn. Published 2026-09-05, updated 2026-10-11.
Site-search bot protection starts with the work a query creates. A small stream of expensive searches can matter more than a large stream of cached autocomplete requests. Measure those paths separately before applying one limit to everything called search.
The operator's task is to keep useful discovery available while restricting automated collection and unnecessary backend work. That requires visibility into query shape, pagination, and the results actually returned.
## Separate the search surfaces
A website may have several search systems hiding behind one box: suggestions while typing, a full result page, filtered searches, related-record lookups, and an API used by mobile clients. Inventory them before investigating traffic.
For each surface, record the normalized route, permitted query options, largest response, backend owner, and whether results are public or account-specific. Confirm which controls execute before the search engine receives the request.
Do not log raw search terms by default. Queries may contain names, email addresses, or other sensitive input. Query length, approved filter names, pagination depth, and a restricted correlation reference can often answer the first operational questions.
## Compare request volume with search cost
[Elasticsearch's pagination documentation](https://www.elastic.co/docs/reference/elasticsearch/rest-apis/paginate-search-results) explains why deep offset pagination can increase memory and CPU use: shards must account for earlier hits as well as the requested page. That is one documented mechanism, not a claim that every search backend behaves identically.
Build a comparison using the actual backend's cost indicators.
| Query family | Record | Investigate when |
| --- | --- | --- |
| Autocomplete | Requests per interaction and cancellation | Old queries continue after the user has moved on |
| Filtered results | Allowed filters and execution time | A narrow option repeatedly triggers expensive work |
| Deep pagination | Page depth and records returned | Clients systematically traverse large result sets |
| Zero-result searches | Work spent and result count | High backend work produces little useful output |
| Account search | Tenant scope and response fields | Search exposes a wider set than detail permissions |
These are diagnostic leads. A poorly debounced frontend or an integration retry bug can also create repetitive searches.
## Choose controls for the expensive operation
Start with explicit limits on page size, allowed filter combinations, and execution time where the search platform supports them. Validate those limits server-side. Removing an option from the interface does not remove it from the endpoint.
Use separate budgets for lightweight suggestions and full searches. If account-specific search is available, include an account or tenant allowance rather than relying entirely on source IP.
A bulk retrieval job may need a different interface with explicit authorization and scheduling. Treat it as an approved export, not an unusually patient user clicking “next.”
[OWASP's sensitive business-flow guidance](https://owasp.org/API-Security/editions/2023/en/0xa6-unrestricted-access-to-sensitive-business-flows/) supports reviewing what repeated use does to the service. The fact that a query is syntactically valid does not settle whether its use is acceptable.
## Avoid breaking the search experience
Observe a normal typing sequence, pasted text, a filter change, and a slow connection. A user should not consume an entire allowance merely because the frontend sends requests for every character.
If a search is refused, give the interface a controlled response that it can explain. Do not return an HTML challenge to a client that expects JSON without a deliberate integration path. Stop automatic retries from repeatedly submitting the same rejected query.
After a change, compare search completion and backend work by query family. Also check whether extraction moved from search to detail pages. The [broader scraping prevention guide](https://www.centinelanalytica.com/blog/how-to-prevent-web-scraping) helps map those adjacent paths.
Use the [API abuse use case](https://www.centinelanalytica.com/use-cases/api-abuse) to review protection options for the selected search path. Keep the query-cost comparison with the configuration so future changes to search can be checked against it.
---
Page: https://www.centinelanalytica.com/blog/protect-data-exports-from-abuse
# Protect data-export endpoints from abuse
Review export permissions, queue admission, result delivery, and signed downloads. Bound automated bulk extraction without disrupting approved data exports.
By Frederick Jahn. Published 2026-09-05, updated 2026-10-11.
An export is a separate data-access workflow. Protecting the page that contains the download button does not automatically protect job creation, background processing, status polling, or the finished file.
Review the entire lifecycle: who may request the export, how much work can enter the queue, what data the worker reads, and who can retrieve the result. Put a bound on each stage before investigating sophisticated automation.
## Follow one export from request to file
Choose a normal export and trace its server-side identifiers. The following is a review checklist, not a prescribed architecture.
1. The application authenticates the requester and checks the requested scope.
2. A job is admitted under an account or tenant allowance.
3. A worker generates only the permitted result.
4. The requester checks the job's status.
5. The file is delivered through an authenticated response or a controlled link.
6. The result expires or is removed under the application's retention policy.
For each transition, name the component that enforces it. Also decide what happens if permissions change while a job is waiting. A role revoked after submission should not disappear from the security design merely because processing is asynchronous.
[OWASP's authorization guidance](https://cheatsheetseries.owasp.org/cheatsheets/Authorization_Cheat_Sheet.html) calls for permission checks on requests and protected resources. Apply that principle to status and download endpoints as well as job creation.
## Bound queued work before it becomes expensive
Consider a hypothetical user who presses “export” again because the interface still says “preparing.” A bot could repeat the same action, but so could a confused legitimate user. If each click creates a complete new job, request blocking is only part of the problem.
Give the interface a stable job reference. Where business semantics permit, let a retry retrieve the existing operation instead of admitting duplicate work. Define how that behavior changes when export parameters differ.
Set explicit limits for concurrent jobs, queued jobs, maximum result size, and total generation work per account or tenant. A worker pool limit protects execution capacity but may still leave an unbounded backlog; inspect both.
## Treat download links as access credentials
[Amazon S3 documents that presigned URLs can be used more than once before they expire](https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-presigned-url.html). A signed link is therefore not inherently a one-time download. Its effective lifetime can also end when the signing credentials expire.
If an export requires one-time retrieval or immediate revocation, verify how your actual delivery design provides that property. Do not promise it merely because the URL contains a signature.
Keep signed query strings out of analytics events, screenshots, support tickets, and broadly accessible logs. Record an opaque export reference instead. Review whether referrers or redirects could expose a usable link.
## Match the response to the stage
| Problem | Useful first control |
| --- | --- |
| Repeated identical submissions | Retry-safe job admission |
| Many distinct large exports | Account or tenant work budget |
| Unauthorized status lookup | Ownership check on the job resource |
| Shared active download link | Delivery-policy review and bounded lifetime |
| Abandoned completed files | Scheduled retention enforcement |
These controls address different failures. Adding another challenge to the starting page cannot repair a missing ownership check on a finished object.
## Verify approved bulk access
Include a scheduled integration, an ordinary interactive export, a cancelled job, and a retry after a network interruption in the review. Check the expected file permissions and the cleanup behavior, not just whether the initial request receives a success response.
Then inspect how bulk extraction relates to ordinary detail and API access. An export allowance is easy to misunderstand if the same collection remains unlimited elsewhere. Continue with [authenticated scraping controls](https://www.centinelanalytica.com/blog/prevent-authenticated-scraping) for that boundary.
The [scraping and crawler use case](https://www.centinelanalytica.com/use-cases/scraping-and-ai-crawlers) can frame a protection evaluation. Bring the job and delivery flow so the team can identify where automation controls would actually apply.
---
Page: https://www.centinelanalytica.com/blog/prevent-authenticated-scraping
# How to prevent authenticated scraping
Control automated extraction by signed-in users. Review account permissions, resource budgets, exports, and approved integrations.
By Frederick Jahn. Published 2026-09-05, updated 2026-10-11.
A valid login does not settle whether a client should copy everything it can reach. Preventing authenticated scraping means controlling both the resources an account may access and the amount or pattern of access the application permits.
Begin with the data boundary. If an account can read another customer's records, fix authorization first. If it can copy its permitted collection far beyond its intended workflow, investigate automated misuse and access policy.
[OWASP distinguishes authentication from authorization](https://cheatsheetseries.owasp.org/cheatsheets/Authorization_Cheat_Sheet.html) and recommends validating permissions on each request. A successful login is therefore the beginning of this review, not its conclusion.
## Inventory every way an account receives data
List the detail pages, list endpoints, mobile APIs, search results, file downloads, and export jobs available to the same account. Record which backend decides access for each one.
This inventory often changes the proposed fix. A browser challenge on a detail page is irrelevant to an export file served through a separate path. A limit on one API version leaves the question open if another version provides the same collection.
Use a worksheet like this; the rows describe review questions rather than a product configuration.
| Surface | Permission question | Usage question |
| --- | --- | --- |
| Detail response | May this account read this record? | How much new material is returned over time? |
| List response | Are every row and field within scope? | Can pagination enumerate the whole collection? |
| Search | Are result counts and snippets filtered? | Does search expose data missing from detail access? |
| Export job | May this role create this export? | Are concurrent jobs and result size bounded? |
| Integration | Is the credential scoped to its owner? | Is its bulk workload explicitly approved? |
## Describe the allowed workflow before judging it
An account used for an approved nightly sync should not be compared with an ordinary interactive session. Document its expected resource scope, cadence, and maximum work. Give it a named owner and an expiry or review date.
For interactive accounts, compare cumulative access with the task the interface supports. A user examining a short list behaves differently from a tool traversing every available result, but the distinction is evidence to investigate. It is not proof that an account belongs to a particular person.
[OWASP API6](https://owasp.org/API-Security/editions/2023/en/0xa6-unrestricted-access-to-sensitive-business-flows/) treats excessive access to legitimate business flows as a security problem in its own right. Authentication can be working correctly while that problem exists.
## Put limits next to the data decision
An application can make useful distinctions that a global IP rule cannot. It knows which account requested the material, which permission was checked, and whether the response contains one record or a large export.
Consider independent budgets for an account, a resource family, and expensive bulk operations. Choose their values from legitimate usage and service capacity. Do not infer a safe threshold from a generic requests-per-second example.
Keep authorization mandatory even when the automation assessment is favorable. Conversely, passing authorization should not automatically waive an approved usage allowance.
## Handle a suspicious account carefully
Preserve a redacted request timeline before changing policy. Separate evidence of extraction from evidence of account compromise. Automated activity under an account does not establish stolen credentials; an owner may have written a script or shared a permitted integration.
If compromise is suspected, the [account-takeover prevention guide](https://www.centinelanalytica.com/blog/how-to-prevent-account-takeover) covers the separate authentication and recovery response. If the issue is excessive extraction, start with a bounded restriction on the affected data path and a clear support route.
Recheck mobile clients, normal pagination, and sanctioned integrations after the change. Your completion criterion is that the protected collection follows its intended access policy while approved work remains possible.
Review the [scraping and crawler use case](https://www.centinelanalytica.com/use-cases/scraping-and-ai-crawlers) against the resource inventory. Identify which component can stop each response before the data leaves it.
---
Page: https://www.centinelanalytica.com/blog/how-to-prevent-web-scraping
# How to prevent web scraping
Choose scraping controls by route: access checks, rate limits, crawler rules, and bot detection. Use a control matrix to test failures and roll back safely.
By Frederick Jahn. Published 2026-09-05, updated 2026-10-11.
To reduce unwanted web scraping, control access at the point where your server returns the data. Enforce permissions for private resources, limit expensive or high-volume operations, and use automation evidence to decide which clients need a different response. Keep a separate path for approved crawlers and integrations.
Start with one route. A blanket rule across the whole site makes it harder to see what improved and which legitimate workflow broke.
This guide is for the engineer choosing those controls. For the mechanics of collection and extraction, start with [what web scraping is](https://www.centinelanalytica.com/blog/what-is-web-scraping).
## Define what you need to prevent
Write down the unwanted outcome before selecting a tool. Someone collecting public records presents a different problem from an account reading private records it should not see. An expensive search endpoint has another concern: how much work each request triggers. A search budget cannot fix a private-record permission bug.
[OWASP's scraping taxonomy](https://github.com/OWASP/www-project-automated-threats-to-web-applications/blob/master/assets/oats/EN/OAT-011_Scraping.md) covers collection through web pages and APIs, including resources behind accounts. A login screen alone therefore does not settle the scraping question.
For a public page, the practical goal is to restrict unwanted automated collection while preserving the access you intend to offer. Once you deliver readable data to a client, a bot policy cannot take that copy back. For private data, the first requirement is stricter: a requester without the necessary permissions must not receive it, whether that requester is a person or a script.
Consider a hypothetical application with public detail pages, a search endpoint, and an account export. Protecting the detail-page HTML would leave a gap if the same data remained freely available through search or an export response. Inventory the responses that contain the data, including alternate formats and cached copies.
## Separate crawler instructions from enforcement
Use `robots.txt` to state where cooperative crawlers should go. [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309) defines that protocol and explicitly separates its rules from access authorization. Your server still needs to enforce any restriction that must hold for a client that ignores the file.
[Google's robots.txt guidance](https://developers.google.com/search/docs/crawling-indexing/robots/intro) also distinguishes crawling from indexing: a disallowed URL can appear in search results when other pages link to it. Neither excluding a route from crawling nor hiding it from search is a substitute for securing its response.
Keep private paths protected by server-side permissions. Use the [robots.txt guide](https://www.centinelanalytica.com/blog/robots-txt-ai-bots-guide) for crawler-policy syntax, and check approved crawler access separately when changing enforcement rules.
## Build a route-specific control matrix
The following matrix is a proposed planning method, not a tested Centinel configuration. Use it to review one route with its application owner.
| Resource | Intended access and useful evidence | First control to evaluate | Wrong-decision cost and rollback |
| --- | --- | --- | --- |
| Public detail page | Visitors and approved crawlers; verified operator identity and available session evidence | Keep intended access; review repeated collection before adding selective restrictions | Blocked visitors or crawlers; revert the new restriction and recheck access |
| Public search endpoint | Interactive users and explicitly supported integrations; request cost, result count, and request history | Bound result size and execution cost; evaluate a route-specific request budget | Broken searches or legitimate bursts; restore the previous budget without removing resource caps |
| Private record endpoint | Accounts entitled to the requested record; server-validated identity and object permissions | Enforce authorization on every request, then assess automated misuse within valid permissions | Data disclosure or denied account access; preserve authorization while reverting the new bot rule |
| Account export | Roles entitled to export; account identity, export size, and concurrent jobs | Check export permissions and bound job size and concurrency | Failed legitimate exports; restore the prior job policy while retaining permission checks |
| Partner API | Approved clients within their assigned scope; validated credentials and usage history | Apply client-specific permissions and budgets | Broken integration jobs; restore the prior approved policy, not an unrestricted bypass |
Separate the evidence from the decision. A crawler name is a claim to verify. A valid account establishes an authenticated identity, not permission to read every object. Missing browser telemetry tells you that an observation is absent; it does not, by itself, explain why.
For each row, name the enforcement component and the person responsible for reverting the policy. If nobody can point to where a denied request stops, the matrix is unfinished.
## Layer controls in a safe order
### Enforce permissions before returning private data
Apply the check on the server, including API and download routes. [OWASP's authorization guidance](https://cheatsheetseries.owasp.org/cheatsheets/Authorization_Cheat_Sheet.html) recommends checking permissions on every request and protecting static resources where access is restricted. Hiding a button or requiring the browser to visit another page first does not perform that check.
A bot decision must not override an account's lack of permission. Keep authorization independent so a rollback of a new scraping rule cannot expose private records.
### Bound the work a request can trigger
A request counter treats a small detail response and a large export as one request each. Review the work behind the endpoint as well as its frequency. [OWASP's API resource-consumption guidance](https://owasp.org/API-Security/editions/2023/en/0xa4-unrestricted-resource-consumption/) recommends limits on operations and data sizes, including server-side validation of parameters that select how many records a response returns.
In the hypothetical application, start by reviewing search result limits and export concurrency. Then choose request budgets that fit the intended clients. Do not copy a threshold from an unrelated site's configuration.
### Decide what the rate limit counts
Record the counter's key, window, scope, and behavior when the key is unavailable. An IP budget groups all requests using that address. If several legitimate users share it, they share the budget too. A credential-based budget groups traffic differently and needs a trustworthy credential binding.
[RFC 6585](https://www.rfc-editor.org/rfc/rfc6585#section-4) defines `429 Too Many Requests` and optional `Retry-After` guidance. It leaves client identification and request counting to the implementation. A 429 response communicates the limit; it is not evidence that the client is a scraper.
Test an approved integration's retry behavior. Confirm that your enforcement layer actually stops the expensive operation rather than doing the work and rejecting the response afterward.
### Add automation evidence where it changes the response
For browser routes, review available browser and session observations alongside the request history. For direct API clients, use the evidence that exists on that path; a browser challenge is not a sensible default for a client that cannot render it.
The [browser-automation detection guide](https://www.centinelanalytica.com/blog/detect-browser-automation) explains how to interpret the evidence layers. Use the evidence to investigate a client and apply the route's policy. Avoid turning one unusual property into a site-wide block.
## Run a control-and-failure test
This is a proposed reader exercise. No run or measured outcome is reported here.
The versioned plan is `web-scraping-control-matrix-v1`, dated `2026-09-04`. It calls for 20 sessions in each of four cohorts, 80 sessions in total. Those counts are a planned functional sample, not a production accuracy estimate.
Use synthetic records on a staging route you own or are authorized to test. Every client must be test-owned or explicitly authorized. In the cohort names, "unapproved" means the test route's policy does not approve that automated collection; it does not mean the operator lacks permission to conduct the test.
| Cohort ID | Known client setup | Question for the test |
| --- | --- | --- |
| `ordinary-browser` | A person using an ordinary browser to complete the intended route | Can the user finish, including after an ordinary retry? |
| `approved-crawler` | A test-owned crawler with an explicitly approved identity and route scope | Does approved collection still work within its budget? |
| `unapproved-direct-client` | A test-owned HTTP client assigned to the restricted collection case | Does the route enforce its policy without depending on browser JavaScript? |
| `unapproved-browser-automation` | A test-owned automated browser assigned to the same restricted collection case | Is the policy enforced when the client can execute browser code? |
Pin the client versions and policy revision. Keep the target data and intended task consistent, and retain a known-good policy for comparison. The operator should record each session's assigned ground truth before inspecting the detector's output.
Record the following separately:
- The observations available at the integration point, with absent observations marked unknown.
- The automation classification and the policy decision. An approved crawler can correctly be classified as automated and still be allowed.
- The actual response sent to the client, including retries or challenges.
- Whether the task completed, how much synthetic data was returned, and whether origin work occurred before rejection.
Keep timeout and incomplete sessions visible. A missing result must not silently become a successful block. Retain per-session evidence privately; use aggregate counts and a traceable evidence reference for review.
Also rehearse failure cases in isolation: a missing session, unavailable browser telemetry, a detector timeout, legitimate users sharing an IP budget, and a rule rollback. Record these checks separately from the 80-session plan; do not mix them into its cohort counts or describe them as completed until they have run.
## Measure the policy and keep a rollback path
Before enforcement, write down the outcome that would make you revert. The application owner should set tolerances for legitimate task failures and response time based on that route's requirements.
Compare the old and proposed policy using the same tasks. Then review unwanted extraction completion alongside legitimate-user completion, approved-crawler access, challenge or retry frequency, and origin workload. Include counts and denominators. "More blocked requests" alone cannot tell you whether the policy is useful.
Treat an unavailable detector as a separate failure mode. For a public route, the team may choose continued access under existing resource budgets. Private routes must keep their authorization checks. Decide the timeout and fallback before rollout; do not discover the default during an incident.
Keep rollback narrow: restore the previous bot-policy revision, leave access controls in place, and rerun the legitimate-user and approved-crawler tasks. Confirm the integration actually uses the restored revision.
## Where Centinel fits
Centinel's [bot-management overview](https://www.centinelanalytica.com/bot-management) describes the evidence model and evaluation questions. For this method, the integration boundary matters: the customer's server calls the validation API and enforces its response. Browser-to-collector observations and the protected-request metadata sent by that server are different inputs. A direct HTTP client does not supply a browser payload.
Take particular care with an observation-only rollout. Centinel's [policy documentation](https://docs.centinelanalytica.com/admin/policy) says monitor mode returns terminal blocks as allows, while challenges still return normally. It affects the request beyond the rule that enabled it. Do not treat that switch as a promise that test traffic will encounter no intervention; verify the complete policy on an isolated staging route.
For the broader product use case, see [scraping and AI crawler protection](https://www.centinelanalytica.com/use-cases/scraping-and-ai-crawlers). The next practical step is to fill in the matrix for one route, identify the enforcement point, and review it with the team that owns the application before changing production traffic.
---
Page: https://www.centinelanalytica.com/blog/detect-low-and-slow-scraping
# How to detect low-and-slow web scraping
Find scraping that stays under short rate limits by comparing resource coverage, account activity, and extraction over a longer window than a rate chart.
By Frederick Jahn. Published 2026-09-05, updated 2026-10-11.
Low-and-slow scraping is automated collection spread over enough time that a short rate-limit window may never look busy. To investigate it, compare what a client collects across a longer period: distinct resources, repeated traversal, returned data, and the permissions that make those responses available.
A quiet request graph can still describe substantial extraction. The useful question is whether access fits the client's purpose and allowance.
[OWASP's scraping threat description](https://github.com/OWASP/www-project-automated-threats-to-web-applications/blob/master/assets/oats/EN/OAT-011_Scraping.md) includes recurring collection of accessible pages and API responses. It does not give a universal request rate that separates scraping from normal use.
## Change the unit of investigation
Start with one valuable resource family, such as a directory or a collection of detailed records. Choose a bounded review window that covers its normal usage cycle. A minute, a session, and a working day answer different questions.
For authenticated access, group by a server-validated account or API credential. For anonymous access, keep the uncertainty visible: a cookie may disappear, and an IP address may represent many visitors. Do not silently turn a shared fingerprint into a person.
Record the number of distinct resources returned, total response bytes, and whether requests revisit earlier material or steadily cover new material. Use normalized routes for grouping, but preserve a restricted reference to the original request if an investigator needs it.
## A small example of what a rate chart misses
The following is hypothetical. Two clients each make six requests in every ten-minute window. Neither looks unusual by rate alone.
| Observation over the review period | Client A | Client B |
| --- | --- | --- |
| Resource selection | Revisits a small working set | Continues through new records |
| Navigation | Alternates reading and editing | Repeats list-to-detail traversal |
| Successful data returned | Mostly repeated material | Mostly previously unseen material |
| Known purpose | Documented work session | No documented bulk-access need |
Client B deserves investigation, not an automatic guilty verdict. A migration, accessibility tool, or approved synchronizer could produce a similar record. Conversely, Client A could still misuse the material it receives.
The example's value is the next question it creates: who is entitled to retrieve this collection at that scale?
## Look for continuity without inventing identity
A source IP changing halfway through a collection does not prove a new visitor. Equally, similar navigation from several IPs does not prove coordination.
Keep strong and weak grouping keys separate in the investigation. An authenticated account is useful for account-level policy; a repeated route sequence is a lead. If you combine weak signals into a suspected group, record which signals were used and what alternative explanation remains.
[OWASP's bot-management guidance](https://cheatsheetseries.owasp.org/cheatsheets/Bot_Management_and_Anti-Automation_Cheat_Sheet.html) describes controls at several observation layers. Use that layered approach to corroborate the sequence instead of making one browser property carry the decision.
## Respond to extraction, not just speed
For the selected resource family, consider a cumulative allowance for unique records or delivered data, alongside a short burst limit. Put approved bulk jobs on an explicit, bounded access path. A long-window allowance should have a reset rule, an owner, and a way for legitimate users to request more access.
Begin by recording which requests the proposed policy would affect. Check a representative busy workflow and an approved automation job. Then enforce a narrow change where authorization and response handling are understood.
Watch whether successful extraction falls, whether ordinary work still completes, and whether activity moves to another route. A higher block count alone does not answer any of those questions.
Use the [scraping and crawler use case](https://www.centinelanalytica.com/use-cases/scraping-and-ai-crawlers) to scope the protection boundary. For the broader route-by-route control plan, continue with [web scraping prevention](https://www.centinelanalytica.com/blog/how-to-prevent-web-scraping).
---
Page: https://www.centinelanalytica.com/blog/audit-ai-crawler-traffic
# How to audit AI crawler traffic on your website
Build a crawler inventory from request logs. Separate claimed names, verified operators, resource use, and policy outcomes before changing crawler access.
By Frederick Jahn. Published 2026-09-05, updated 2026-10-11.
An AI crawler audit should tell you which clients reach your site, which identities you can verify, what resources they use, and whether your access policy is working. A list of user-agent strings answers only the first part, and even then it records claims rather than verified operators.
Start with a bounded log window and the routes you care about. Keep unknown clients in the inventory so the audit does not become a report about only the bots that announce themselves.
## Choose the observation point and time window
Decide whether you are auditing edge requests, origin requests, or both. A cache can answer requests without forwarding them to the origin. A control at the edge can also reject traffic before an application log exists. Comparing those totals without accounting for the boundary leads to misleading conclusions.
Use a recent complete window that covers the activity you want to understand. Record its start, end, time zone, and known collection gaps. Preserve sampling information if the logging system does not retain every request.
Collect only the fields needed for the investigation. Useful fields include a normalized route, time bucket, response status, cache outcome, bytes returned, claimed crawler name, verification result, and policy revision. Keep sensitive query parameters, cookies, and credentials out of the audit export. [OWASP's logging guidance](https://cheatsheetseries.owasp.org/cheatsheets/Logging_Cheat_Sheet.html) is a useful reference for deciding what to exclude.
## Separate crawler purpose from its name
Different clients from the same operator can perform different jobs. [OpenAI's crawler documentation](https://developers.openai.com/api/docs/bots) distinguishes [GPTBot](https://www.centinelanalytica.com/bots/gptbot)'s training-related role, [OAI-SearchBot](https://www.centinelanalytica.com/bots/oai-searchbot)'s search role, and [ChatGPT-User](https://www.centinelanalytica.com/bots/chatgpt-user)'s user-initiated visits. It also describes their separate controls and published network references.
[Anthropic's crawler guidance](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) distinguishes [ClaudeBot](https://www.centinelanalytica.com/bots/claudebot), [Claude-SearchBot](https://www.centinelanalytica.com/bots/claude-searchbot), and [Claude-User](https://www.centinelanalytica.com/bots/claude-user). Do not apply one operator's verification or robots behavior to another by analogy.
Record the purpose documented by the operator, the source, and when you checked it. Keep the raw observed name alongside the normalized label; names and versions change. The [AI traffic taxonomy](https://www.centinelanalytica.com/blog/what-is-ai-agent-traffic) helps distinguish background crawling from other automated visits.
## Verify identity before adding a trusted rule
Use the operator's documented method. For Google, the [verification guide](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests) describes published IP ranges and reverse DNS followed by a forward lookup back to the original address. Use the correct crawler category and exact hostname boundaries, not a loose substring match.
Record separate outcomes: verified, contradicted, and unknown. A lookup timeout belongs in unknown. An address absent from a reference does not automatically establish malicious intent; check which client the reference actually covers.
Store the verification source and retrieval time. A request with a copied user-agent string must not inherit trust from another request with the same name. For a fuller procedure, see [how to verify crawler and agent identities](https://www.centinelanalytica.com/blog/how-to-verify-ai-agents).
## Build the audit table around decisions
Group by verified operator or unknown client category, normalized resource, and response. Avoid one hostname-wide total that hides which route is doing the work.
The following rows are hypothetical examples of decisions an audit could support. They are not observations from Centinel traffic.
| Finding to investigate | Evidence to retain | Decision to review |
| --- | --- | --- |
| A verified search crawler reaches public pages successfully | Verification reference, response distribution, affected routes | Preserve intended search access |
| A verified training crawler reaches a route meant to be restricted | Route policy, robots rules, actual response, policy revision | Reconcile cooperative rules with enforced access |
| An unverified client claims a trusted crawler name | Original claim, trusted source address, verification outcome | Investigate before granting an exception |
| An unknown client repeatedly reaches expensive search routes | Bounded request history, response cost, available client evidence | Review route budgets and automation evidence |
| An approved crawler begins receiving failures after a release | Change time, rule revision, response details | Check for a policy or integration regression |
Measure workload as well as request count. A cached detail page and an origin-generated search response can impose different work even when each appears once in a counter. Keep response bytes, cache results, and available origin-work indicators separate rather than inventing a single cost score.
## Compare intended policy with actual responses
[RFC 9309](https://www.rfc-editor.org/rfc/rfc9309) defines cooperative crawler instructions, not an authorization boundary. Your audit should therefore compare three records: the robots rule, the enforced route policy, and what the requester actually received.
For an approved client, check successful access to its intended routes. For a restricted client, check whether protected data was returned. A block counter alone cannot answer either question if another endpoint returns the same data.
Use the [crawler access-policy guide](https://www.centinelanalytica.com/blog/how-to-control-ai-agent-access) to document the decision. If you need to change enforcement, the [crawler-blocking guide](https://www.centinelanalytica.com/blog/how-to-block-ai-crawlers) covers the rollout questions.
## Leave a repeatable handoff
Save the window, scope, aggregation method, verification references, and policy revision with the result. Separate confirmed findings from questions requiring another observation. Assign one owner and a next action to each proposed change.
Repeat the same audit after a policy change and compare equivalent windows. Retain missing-data and unknown categories; a smaller unknown count is meaningful only if the evidence improved, not if the query stopped including those requests.
The [scraping and AI crawler use case](https://www.centinelanalytica.com/use-cases/scraping-and-ai-crawlers) connects this inventory to response options. Start with the route where intended access and observed behavior disagree, and keep its evidence with the policy change.
---
Page: https://www.centinelanalytica.com/blog/what-is-an-antidetect-browser
# What is an antidetect browser?
An antidetect browser changes the browser characteristics a site can observe. Learn how profiles work, which inconsistencies matter, and how to respond.
By Frederick Jahn. Published 2026-09-04, updated 2026-09-05.
An antidetect browser changes the characteristics that a website can observe
about a browser and its device. The operator can use separate profiles so that
several sessions appear to come from different browser environments, or make
one environment resemble another.
The name is informal. There is no web standard that defines an "antidetect
browser," and implementations do not all work the same way. Some modify values
with injected JavaScript. Others change browser code or run a prepared virtual
environment. That distinction matters when you investigate one.
## What the browser is trying to change
A browser fingerprint is assembled from observable characteristics rather
than one identifier. The [W3C's fingerprinting guidance](https://www.w3.org/TR/fingerprinting-guidance/) describes a surface
that can include browser and operating-system characteristics, device details,
configuration, environmental readings, and behavior. A site may collect some
values directly and infer others from rendering or timing.
Common inputs include HTTP and JavaScript user-agent values, language,
timezone, screen properties, supported APIs, fonts, and graphics output from
Canvas or WebGL. None of these values is automatically an identity. Their
combination can still make one browser easier to recognize across requests.
An antidetect profile changes part of that combination. The desired result is
a plausible environment that does not link back to the operator's unmodified
browser.
## Three implementation approaches
Researchers who evaluated anti-fingerprinting browsers grouped the designs
they studied into three approaches:
1. JavaScript injection replaces properties or methods before page code reads
them.
2. Native spoofing changes the browser implementation so it returns modified
values itself.
3. Environment recreation runs a chosen browser and operating environment in
a virtualized setup.
The distinction comes from the 2020 paper [Taming the Shape Shifter: Detecting Anti-fingerprinting Browsers](https://pmc.ncbi.nlm.nih.gov/articles/PMC7338203/). It is a useful
model, not a guarantee that every current product fits neatly into one box.
Injection is comparatively easy to update, but an incomplete override can
leave JavaScript objects in an impossible state. Native changes can remove
some of those obvious seams, while increasing the work required to track new
browser releases. A complete environment can produce genuine values for that
environment, but it needs more resources than changing a property in place.
## Why a changed fingerprint is not a verdict
Changing a fingerprint does not reveal why it was changed. Privacy
countermeasures alter fingerprints to resist cross-site tracking. The 2020
study also documents anti-fingerprinting browsers built for fraud involving
stolen credentials. A site still has to evaluate the action taking place.
That is why "fingerprint changed" should not become "malicious bot" in one
step. It is evidence about the client environment. The application still needs
to consider the requested action, account history, request sequence, and any
approved identity attached to the session.
The opposite shortcut is risky too. A browser profile that looks ordinary on
one check has not proved that a person is operating it.
## How consistency checks find altered profiles
Spoofing becomes difficult when several values must agree. The
[FP-Scanner study](https://www.usenix.org/conference/usenixsecurity18/presentation/vastel)
tested fingerprinting countermeasures by looking for contradictions between
their reported attributes. Its checks included agreement between the HTTP and
JavaScript user agent, consistency between the claimed operating system and
WebGL renderer, and browser features that should or should not exist in a
given configuration.
This is more useful than looking for one suspicious string. For example, a
session may claim one browser family in its request headers while exposing a
different family through JavaScript. A reported operating system may conflict
with the graphics stack. A supposedly native function may have been replaced.
Even a real inconsistency needs context. Browser extensions and privacy
defenses can produce unusual combinations. Treat the mismatch as one
observation and compare it with independent request, network, session, and
workflow evidence. [How to detect browser automation](https://www.centinelanalytica.com/blog/detect-browser-automation) explains how to build that
evidence model without turning one signal into a verdict.
## A practical response
Start by recording the raw values and where each one was observed. Preserve
the declared browser separately from any derived classification. Then ask:
- Do the request headers and browser APIs describe a possible configuration?
- Does the environment remain coherent during the session?
- Is the client following a reachable application workflow?
- Is the action expected for this account or approved identity?
- Which response is proportionate to the evidence and the route at risk?
An uncertain profile can be observed or challenged on a sensitive workflow
without blocking every visitor who uses a privacy feature. Stronger responses
should depend on corroborating evidence and an explicit access policy.
## Sources
- [W3C: Mitigating Browser Fingerprinting in Web Specifications](https://www.w3.org/TR/fingerprinting-guidance/)
- [USENIX Security 2018: Fp-Scanner: The Privacy Implications of Browser Fingerprint Inconsistencies](https://www.usenix.org/conference/usenixsecurity18/presentation/vastel)
- [DIMVA 2020: Taming the Shape Shifter: Detecting Anti-fingerprinting Browsers](https://pmc.ncbi.nlm.nih.gov/articles/PMC7338203/)
---
Page: https://www.centinelanalytica.com/blog/detect-bots-using-residential-proxies
# How to detect bots using residential proxies
Detect bots behind residential proxies using trusted source data, client consistency, and workflow evidence. Choose scoped controls without blocking whole ISPs.
By Frederick Jahn. Published 2026-09-04, updated 2026-10-11.
Detect bots using residential proxies by checking what the client does, not
only where its IP address is registered. Establish the trusted source address,
then compare independent client, session, and application evidence. A
residential-network label proves neither a human visitor nor abusive automation.
If IP rules keep missing repeated retrieval, investigate the protected action
across first-party sessions or authenticated accounts where available. Do not
assume changing addresses means one actor, or that everyone behind one address
is the same person. Scope controls to the workflow and evidence you can defend.
## The short answer
Start with these five checks:
1. Establish which component observed the source address.
2. Check whether that component is inside a trusted proxy boundary.
3. Record the address classification and the lookup time.
4. Compare independent client and session evidence.
5. Apply a response for the protected route, not for an entire ISP range.
Keep an explicit unknown state when the evidence supports neither automation
nor an approved client. Otherwise, missing evidence quietly becomes a
human-or-bot verdict.
For example, consider a test account that exceeds an explicit export budget
while using several residential addresses. Enforce the account's export limit;
you do not need to prove who owns each endpoint. If unrelated public-page
visitors share only a proxy label, that evidence does not justify treating them
as one abusive session. These are decision examples, not measured findings.
For a commercial evaluation, the [bot management overview](https://www.centinelanalytica.com/bot-management)
sets out the evidence and deployment questions to answer before adopting a
new detector.
## What a residential proxy changes
A residential proxy relays traffic through an address assigned to a consumer
or small-business connection. The simplest request path looks like this:
```
requesting software
|
v
residential proxy service
|
v
consumer-network endpoint
|
v
protected application
```
The protected application observes the endpoint used for the final hop. That
address says where the request emerged onto the internet. It does not identify
the software that selected the URL, the person who owns the connection, or the
party paying for the proxy service.
The [FBI's residential-proxy safety alert](https://www.fbi.gov/investigate/cyber/alerts/2026/evading-residential-proxy-networks-protecting-your-devices-from-becoming-a-tool-for-criminals)
describes several ways consumer devices can become proxy endpoints. Some
owners consent to sharing bandwidth. Others may not know their device or
connection is being used. This is one reason an address-level block can affect
someone who did not initiate the activity you are investigating.
There is another proxy boundary closer to the application. A CDN, load
balancer, or ingress proxy may be the direct peer your server sees. RFC 7239
defines the [Forwarded HTTP field](https://www.rfc-editor.org/rfc/rfc7239.html) for cooperating proxies to
pass information that would otherwise be lost, including a preceding client
address.
Do not trust a forwarding field merely because it exists. Configure the first
trusted ingress to remove requester-supplied `Forwarded` or equivalent
client-IP fields before writing its own value. If trusted proxies form a chain,
document which hop appends each element and parse only the trusted portion.
RFC 7239 warns that the field cannot be relied on without a trustworthy chain.
Write the trust boundary down:
| Question | Record |
| --- | --- |
| Where is the direct peer address observed? | Named CDN, proxy, load balancer, or application socket |
| Which intermediaries may append forwarding data? | Exact services and network ranges |
| Who removes untrusted incoming fields? | Named component and configuration owner |
| Which field does the application use? | Exact field and parsing rule |
| What happens when the chain is malformed? | Reject, ignore, or mark unknown |
If those answers are missing, fix the boundary before tuning a proxy rule.
## Why IP reputation is incomplete
An IP intelligence result is a lookup against a dataset. It has a provider, a
collection method, a timestamp, and blind spots. Keep those details beside the
label.
In a 2022 study centered on China, Yang and colleagues collected residential
proxy addresses that were absent from the public datasets they compared. The
[authors' paper](https://arxiv.org/abs/2209.06056) is useful evidence that list
coverage can be incomplete. It is not a current global benchmark. The market,
collection period, and proxy services in your traffic may be different.
For each lookup, preserve the raw operational facts:
Record the provider and dataset version when available. Otherwise, record
`unknown` rather than inferring a version.
| Field | Example value |
| --- | --- |
| Observed address | Stored in the restricted request record |
| Observer | Edge integration |
| Network owner and type | Consumer ISP, hosting, mobile, unknown |
| Proxy label | Residential proxy, not listed, unknown |
| Dataset and version | Provider plus feed or model version |
| Lookup time | UTC timestamp |
| Confidence supplied by provider | Raw value, if available |
| Cache age | Duration since the last provider result |
"Not listed" and "not a proxy" are different results. The first means the
dataset has no matching label. The second is a claim that needs a documented
method and an expiry time. Neither result establishes who operated the client.
## Build an evidence matrix
HTTP gives the server a method, target, fields, and optional content. [RFC 9110](https://www.rfc-editor.org/rfc/rfc9110.html#section-6) specifies that
message structure. It does not add a human identity to the request. Detection
comes from comparing observations that fail in different ways.
| Evidence family | Question | Useful observation | Benign counterexample |
| --- | --- | --- | --- |
| Network | What path reached the trusted boundary? | Address classification changes repeatedly within one task | Mobile networks, VPNs, privacy relays, and corporate egress can also change the visible path |
| Client consistency | Do the declared client and observed environment agree? | Browser headers conflict with browser capabilities or the session's earlier values | Browser updates, extensions, enterprise policy, and embedded webviews create unusual combinations |
| Session sequence | Does the request follow the application's reachable workflow? | High-cost actions repeat without the expected navigation or state transition | API clients, assistive technology, retries, and restored sessions may skip ordinary page order |
| Application outcome | Which protected action is being repeated? | One workflow consumes an explicit action budget across rotating addresses | Shared offices, classrooms, carrier NAT, and popular events can group unrelated users |
| Known identity | Is there a first-party account, service credential, or verified crawler identity? | The claimed client lacks its required credential or verification result | A new legitimate visitor may have no prior identity or history |
Each suspicious observation has a normal counterexample. Before acting, a
reviewer needs to explain why the observations belong together and why the
benign explanation is less likely.
Keep raw observations separate from derived labels. For example:
```
{
"network": {
"classification": "residential-proxy",
"datasetVersion": "record-the-version",
"observedAt": "record-the-time"
},
"client": {
"declaredFamily": "record-the-claim",
"consistency": "unknown"
},
"session": {
"workflow": "record-the-route-sequence",
"history": "insufficient"
},
"decision": {
"confidence": "unknown",
"response": "observe"
}
}
```
An analyst can tell which input changed,
which dataset supplied a label, and why the decision remained unknown.
For a deeper treatment of the browser layer, read [How to detect browser automation](https://www.centinelanalytica.com/blog/detect-browser-automation). The broader scraping policy is
covered in [What is web scraping?](https://www.centinelanalytica.com/blog/what-is-web-scraping).
## Run a controlled test
The test below is a protocol, not a result. Run it on a staging route with
accounts, endpoints, and networks you are authorized to use. Do not enroll a
third party's device or connection in a proxy test.
Experiment ID: residential-proxy-evidence-matrix-v1.
Use a four-by-two design. Cross four client types with two network paths, and
keep the remaining test conditions fixed inside each comparison.
| Cohort ID | Client control | Network path |
| --- | --- | --- |
| human-ordinary-network | Manually operated browser | Ordinary approved test connection |
| human-residential-network | Manually operated browser | Authorized residential-proxy test path |
| approved-automation-ordinary-network | Approved monitoring or QA automation | Ordinary approved test connection |
| approved-automation-residential-network | Approved monitoring or QA automation | Authorized residential-proxy test path |
| browser-automation-ordinary-network | Test-owned current Playwright with default settings | Ordinary approved test connection |
| browser-automation-residential-network | Test-owned current Playwright with default settings | Authorized residential-proxy test path |
| direct-client-ordinary-network | Test-owned direct HTTP client | Ordinary approved test connection |
| direct-client-residential-network | Test-owned direct HTTP client | Authorized residential-proxy test path |
All clients and network paths must be authorized for the experiment. The
Playwright and direct-client cohorts represent automation that the staging
route's policy has not approved for access; they are not unauthorized tests.
Before the run, record the browser, automation library, direct-client version,
operating system, proxy contract, IP intelligence dataset, staging build, and
rule versions. Pre-register the same session count for each cohort. Twenty per
cell means 160 sessions in the first run. It is a debugging exercise, not a
representative population study.
Then run the protocol:
1. Create a staging workflow with a landing page, a reachable protected route,
and one bounded action such as a test search or test form submission.
2. On the isolated staging route, record validation responses without applying
their block or redirect actions to real users. If you use a Centinel monitor
[policy](https://docs.centinelanalytica.com/admin/policy), record both the actual and returned decisions: monitor converts a
block to a returned allow, but it does not change allow or redirect.
3. Use a fresh browser profile for browser cohorts and a fresh application
cookie or test credential for every session.
4. Randomize the cohort order so a deployment or dataset refresh does not
affect only one group.
5. Attempt the same server-recognized request sequence in every session. Record
which browser-only steps the direct client cannot perform.
6. Save one row per session with its known cohort, raw evidence, derived
labels, decision, and any collection failure.
7. Review misclassifications by cohort and evidence family. Do not discard
unknown or missing values.
8. Repeat after changing one variable, such as the browser version or network
provider. Do not change several variables in the same comparison.
The useful output is a table of broken assumptions. Did the network dataset
label every authorized proxy session? Did it label an ordinary session? Did a
browser consistency rule react to the network path even though the browser
did not change? Did missing data silently become low risk?
Do not publish an accuracy rate from this staging exercise. The cohorts are
small and selected, and the labels describe your test setup rather than the
traffic population on a live application. Production evaluation needs
reviewed outcome labels, a representative sample, and a stated cost for both
false positives and false negatives.
## Choose a route-specific response
The same evidence should not produce the same response everywhere. A public
page, login, inventory check, and account export have different failure costs.
| Route condition | Evidence state | Possible response | Guardrail |
| --- | --- | --- | --- |
| Public, inexpensive read | One weak network label | Allow and observe | Expire the label and retain the unknown state |
| Interactive route | Several independent inconsistencies | Bounded challenge or step-up check | Prevent loops and provide an accessible fallback |
| Expensive action | Repeated action exceeds a documented session or account budget | Rate-limit the action | Return a defined retry path and do not rely on IP alone |
| Sensitive action | Strong automation evidence and missing required identity | Deny under the route policy | Log the rule version and provide a review path |
| Approved service or crawler | Required identity verifies | Apply its explicit allow policy | Scope access to the named routes and review the trust rule |
Begin in observation mode. Compare the candidate rule with known monitoring,
QA automation, accessibility flows, mobile clients, and confirmed abuse. A
rule that only separates the four client types across these two test paths is
not ready to protect a live route.
The [scraping and AI crawler use case](https://www.centinelanalytica.com/use-cases/scraping-and-ai-crawlers)
shows how this response ladder fits a broader resource policy.
## Where Centinel fits
Centinel is a sidecar service, not the site's reverse proxy. In a supported
browser flow, the collector receives browser observations and network
fingerprints for the browser-to-collector connection. The customer's
server-side integration sends the protected request's URL, method, IP address,
referrer, headers, and session cookie to `/validate`.
Keep those observation points distinct when interpreting network evidence.
Direct API clients do not execute the browser collector, so calling a protected
endpoint does not give them a browser payload.
That boundary matters for residential-proxy work. Centinel should contribute
the evidence it actually observes, preserve missing values, and leave the
route response to the configured policy. It does not identify the subscriber
behind an address, prove that a residential endpoint is compromised, or
replace API authentication and network-layer availability controls.
Review the [integration options](https://www.centinelanalytica.com/integrations) before enabling a rule. Confirm
which connection each signal describes, which client types supply browser
evidence, and where the returned decision is enforced. Evaluate changes using
legitimate workflow completion as well as reviewed abuse, not a rising block
count alone.
Need help separating network labels from the automated traffic reaching your
pages? [Request a site audit](https://www.centinelanalytica.com/free-audit). Describe the affected workflow and
current controls so the review starts with the decision you need to make.
## Sources
- [FBI: Residential proxy networks and consumer devices](https://www.fbi.gov/investigate/cyber/alerts/2026/evading-residential-proxy-networks-protecting-your-devices-from-becoming-a-tool-for-criminals)
- [Yang and colleagues: An Extensive Study of Residential Proxies in China](https://arxiv.org/abs/2209.06056)
- [RFC 7239: Forwarded HTTP Extension](https://www.rfc-editor.org/rfc/rfc7239.html)
- [RFC 9110: HTTP Semantics](https://www.rfc-editor.org/rfc/rfc9110.html#section-6)
---
Page: https://www.centinelanalytica.com/blog/what-is-zyte
# What is Zyte API? Scraping modes and detection
Zyte API fetches raw HTTP, runs managed browsers and extracts structured data. What each mode looks like to a target site, and how to assess it.
By Frederick Jahn. Published 2026-09-03, updated 2026-09-30.
Zyte API is not one kind of scraper request. It can retrieve a raw HTTP response, run a managed browser, expose that browser to Playwright or Puppeteer, or extract structured data from a page. Those modes look different to the website being accessed.
## What Zyte API does
A customer sends a target URL to one API endpoint and specifies the output it needs. Zyte API can make an HTTP request, render the page in a managed browser, perform browser actions, or run automatic extraction for supported page types. Its [usage documentation](https://docs.zyte.com/zyte-api/usage/index.html) also describes geolocation, IP selection, cookies, and client-managed or server-managed sessions.
For workflows that need direct browser control, Zyte exposes a managed headless browser over the Chrome DevTools Protocol (CDP). Existing Playwright or Puppeteer code can connect to that browser while Zyte handles the browser infrastructure and proxy routing. This differs from a normal browser request, where Zyte performs the requested actions and returns the result.
Zyte's history is closely connected to Scrapy, the open-source Python crawling framework. Current Zyte documentation provides a `scrapy-zyte-api` integration, but Zyte API can also be called from Python, other HTTP clients, or proxy-compatible software.
## How it is used for web data collection
The API covers several jobs that otherwise require separate components. A simple page can be fetched as an HTTP response. A JavaScript-heavy page can be rendered in a browser. Automatic extraction can return structured records for products, articles, jobs, search results, and other documented page types.
Zyte also supports a proxy mode for existing clients. The [proxy-mode documentation](https://docs.zyte.com/zyte-api/usage/proxy-mode.html) makes an important distinction: proxy mode has lower integration overhead, but it does not expose every feature of the HTTP API. Automatic extraction and server-managed sessions, for example, belong to the HTTP API.
## What a target website sees
There is no single request shape for all Zyte usage. An HTTP request, a browser-rendered request, a live CDP session, and proxy mode expose different behavior to the target. The selected proxy type, location, and session settings can change the network path and continuity between requests. Any rule that assumes "Zyte always looks like this" starts from a false premise.
This is why "block one IP" is not a complete response. It may stop the address observed today without identifying the automation or the work it is carrying out. A browser-only rule has the opposite gap when the retrieval path does not use a browser.
The stronger approach is to combine available connection, request, identity, browser, and behavior evidence. Then apply policy according to the requested resource and the activity around it. For the browser-rendered and CDP modes, the guide on [how sites detect automated browsers](https://www.centinelanalytica.com/blog/detect-browser-automation) covers the browser layer of that evidence.
## Centinel benchmark result
In our September 2026 benchmark, Centinel detected Zyte.
The useful follow-up is specific: determine which pages or workflows the automation reached, whether it was authorized, and which response fits the site's policy. The [scraping and AI crawler use case](https://www.centinelanalytica.com/use-cases/scraping-and-ai-crawlers) shows how Centinel applies that kind of per-route policy to scraping services such as Zyte.
## What to check in your own traffic
When investigating possible Zyte-backed automation, examine:
- whether the traffic behaves like single-page retrieval, a crawl, or a live browser workflow;
- which public, expensive, or authenticated resources it requests;
- whether request sequences persist when IP addresses or browser properties change;
- whether a known partner, internal team, or approved data job accounts for it.
This avoids treating a network address as permanent identity and keeps approved automation separate from unwanted extraction.
Read [What is web scraping?](https://www.centinelanalytica.com/blog/what-is-web-scraping) for a broader way to separate technical collection, site policy, and legal review.
## Sources
- [Zyte API usage documentation](https://docs.zyte.com/zyte-api/usage/index.html)
- [Zyte API browser automation](https://docs.zyte.com/zyte-api/usage/browser.html)
- [Zyte API CDP headless browser](https://docs.zyte.com/zyte-api/usage/cdp.html)
- [Zyte API proxy mode](https://docs.zyte.com/zyte-api/usage/proxy-mode.html)
---
Page: https://www.centinelanalytica.com/blog/what-is-ulixee-hero
# Ulixee Hero: the scraping browser and detection
Ulixee Hero is a Chrome-based scraping browser that emulates an ordinary browser. What its emulation covers and how Centinel detected it in a 2026 test.
By Frederick Jahn. Published 2026-09-03, updated 2026-09-30.
Ulixee Hero is a Chrome-based browser automation project built for web
scraping. It gives Node.js code access to a browser and its DOM, while its
emulation layer tries to make the automated environment resemble an ordinary
browser more closely.
That second part is what separates Hero from a bare browser-control API. The
project treats browser emulation and automation masking as part of the tool,
not as a patch the user must assemble afterward.
## What Hero does
Hero scripts can navigate to a page and read or interact with its DOM through
Node.js. The [official Hero repository](https://github.com/ulixee/hero)
describes the tool as powered by Chrome and designed specifically for
scraping. It also describes emulators intended to disguise an automated script
as a modern browser and account for detection beyond page-level JavaScript,
including network and TLS characteristics.
The repository documents supporting profile data collected from real browser
environments. An emulator builder uses that data to patch headless Chrome
toward the characteristics of a headed browser on a home operating system.
Project plugins address markers associated with the browser, network,
operating system, and user interaction.
Ulixee's [troubleshooting documentation](https://github.com/ulixee/hero/blob/main/docs/help/troubleshooting.md)
adds an operational detail: installing Hero downloads Chrome and emulator data
files used to mask automated and headless operation. It also says each Hero
instance records a session database, which can help a developer inspect a
scraping run.
## Where Hero fits
Hero is useful when a scraper needs a real browser engine to execute
JavaScript, render a page, and interact with an application instead of making
plain HTTP requests. Its Node.js DOM interface lets a script query page content
without moving every operation into an evaluate callback.
Hero is still an automated client. It does not become a declared crawler or an
approved integration merely because its browser characteristics look
ordinary. A site should decide whether to allow it based on the requested
resource and the site's access policy.
## Why single-signal checks are a poor fit
Hero's own architecture is meant to address several automation markers. A
check for a default headless user agent therefore asks too little. So does a
decision based only on an IP address or one browser property.
A more useful investigation compares independent observations. Do the request
and browser APIs agree about the client? Does protocol behavior fit the
declared browser? Does the sequence of actions make sense for the protected
workflow? Does the session carry an account, credential, or crawler identity
that the site already trusts?
No one answer settles the question. Together, they can show whether the
session is coherent and whether its behavior fits the site's policy. For the
full method, see [How to detect browser automation](https://www.centinelanalytica.com/blog/detect-browser-automation) and [TLS fingerprinting explained](https://www.centinelanalytica.com/blog/tls-fingerprinting-explained).
## What Centinel observed
Centinel detected Ulixee Hero in our September 2026 benchmark.
Hero is built for scraping. The [scraping and AI crawler use case](https://www.centinelanalytica.com/use-cases/scraping-and-ai-crawlers) shows how Centinel turns that
detection into a per-route policy on your own site.
## What to record during an evaluation
If Hero appears in an authorized test, keep the evidence reviewable:
- Preserve the raw request and browser observations separately from the final
classification.
- Record which protected route and workflow the session exercised.
- Compare Hero traffic with authorized manually operated sessions under the
same test conditions.
- Keep the response policy separate from detection so the team can observe,
challenge, rate-limit, or block according to the route's risk.
This makes the result useful after the test ends. A reviewer can see which
observations supported the classification and change the policy without
rewriting the detector.
## Sources
- [Ulixee Hero: official repository and README](https://github.com/ulixee/hero)
- [Ulixee Hero: official troubleshooting documentation](https://github.com/ulixee/hero/blob/main/docs/help/troubleshooting.md)
---
Page: https://www.centinelanalytica.com/blog/what-is-playwright-stealth
# Playwright Stealth: what it patches and misses
playwright-stealth hides navigator.webdriver and other simple automation tells. What the Python package patches, what it cannot hide, how sites spot it.
By Frederick Jahn. Published 2026-09-03, updated 2026-10-13.
Playwright Stealth is a Python package that applies browser-side evasions to pages or browser contexts controlled through Playwright. It is meant to hide simple signs of automation, not to turn an automated session into a person.
## What the package changes
The current [playwright-stealth package](https://github.com/Mattwmaster58/playwright_stealth) is a maintained fork of an earlier Python port of `puppeteer-extra-plugin-stealth`. Its recommended setup wraps Playwright so new pages receive the stealth changes. An operator can also apply selected evasions to a browser context.
Those evasions target values that page JavaScript can inspect. The package's own examples check `navigator.webdriver`, and its configuration supports changes such as navigator language overrides. The [WebDriver specification](https://w3c.github.io/webdriver/#dfn-webdriver) defines `navigator.webdriver` as the browser property that reports when a user agent is controlled through automation. Hiding it defeats a very simple test, but the missing value does not prove that a visitor is human.
The package is easy to confuse with related projects that share a similar name. This article concerns the Python package published on PyPI as `playwright-stealth`, whose current release is 2.0.3, published on April 4, 2026.
## How it gets used
A developer can add Playwright Stealth to an existing browser script without maintaining a custom browser build such as [Camoufox](https://www.centinelanalytica.com/blog/what-is-camoufox). That makes it a practical first step when a plain automated browser is rejected by basic client-side checks.
The package's maintainer is candid about its reach: the project describes it as a proof-of-concept starting point and says not to expect it to bypass more than simple bot detection. This is a useful boundary. A successful run against a public fingerprint test does not establish that the same configuration will pass a production bot-management system.
## Why single-signal rules miss it
A rule that equates `navigator.webdriver === true` with automation has an obvious blind spot once software changes that property. The same applies to any short, public checklist of JavaScript values.
Defenders should look for agreement across layers. Browser claims can be compared with request and connection properties. Session activity can be assessed in the context of the resource being accessed. Repeated navigation or action patterns may add evidence even when a well-known property has been rewritten. None of those signals should carry the decision alone. The guide on [how sites detect automated browsers](https://www.centinelanalytica.com/blog/detect-browser-automation) explains how these layers combine.
## Centinel benchmark result
Centinel detected Playwright Stealth in our September 2026 benchmark.
Keep a stealth-assisted Playwright case in regression testing. It checks whether detection has accidentally collapsed back to a single property that the package already knows how to change.
## Is your protection catching stealth automation?
Hiding an automation flag can expose a gap in a simple bot rule. Check your own site with a free audit. We run real scraping tools against your site, review the results, and email you the report.
[Request a free site audit](https://www.centinelanalytica.com/free-audit)
## Treat concealment as evidence, not intent
Stealth software tells you that a session may be trying to conceal automation. It does not tell you the operator's intent. Apply policy to the action and resource: approved testing, monitoring, or partner automation can be allowed, while abusive extraction or account activity may require rate limits, a challenge, or a block. [Centinel bot management](https://www.centinelanalytica.com/bot-management), the product tested in that benchmark, applies that policy per route. Test it against your own traffic.
Log the evidence behind the decision. That makes a false positive reversible and gives the detection team something concrete to retest when the package changes.
## Sources
- [playwright-stealth 2.0.3 on PyPI](https://pypi.org/project/playwright-stealth/)
- [Current Playwright Stealth source repository](https://github.com/Mattwmaster58/playwright_stealth)
- [WebDriver specification for the navigator.webdriver flag](https://w3c.github.io/webdriver/#dfn-webdriver)
- [Playwright's official browser documentation](https://playwright.dev/python/docs/browsers)
---
Page: https://www.centinelanalytica.com/blog/what-is-kernel-sh
# Kernel browser (kernel.sh): cloud Chromium
Kernel runs on-demand cloud Chromium for web agents. How it works, what its stealth mode changes, and how Centinel detected it in a September 2026 test.
By Frederick Jahn. Published 2026-09-03, updated 2026-09-30.
Kernel supplies cloud-hosted Chromium sessions for web agents and automation. Developers create a browser on demand, connect their code, and delete the session when the task is done.
## What Kernel provides
A developer can create a browser through Kernel's API and connect over the Chrome DevTools Protocol (CDP) or WebDriver BiDi. Kernel also has a Playwright execution API, computer controls for screenshot-driven agents, and a hosted Model Context Protocol server.
The browser service supports headless and headful sessions. Profiles preserve cookies and browser storage between runs, while Live View and replays help people inspect what an automation did. Browser pools keep configured browsers ready for repeated work. Kernel's [browser creation guide](https://www.kernel.sh/docs/introduction/create) explains when to choose these options.
Kernel works much like [Browserbase](https://www.centinelanalytica.com/blog/what-is-browserbase), another managed cloud-browser service. Its features serve more than scraping. Teams can use remote browsers for testing, account workflows, repetitive web operations, or agents that need to work through a visual interface.
## What stealth mode changes
The word "stealth" needs careful reading here. Kernel applies anti-detection settings to its browsers by default. Its current [stealth mode documentation](https://www.kernel.sh/docs/browsers/bot-detection/stealth) says the optional mode adds a managed static ISP proxy and an automatic CAPTCHA solver. A customer can replace or disable those managed services.
Profiles affect browser state. Proxies affect network identity. Computer controls affect how input reaches the browser. These are separate variables, and they should not be collapsed into one generic "stealth browser" label.
For a target website, this means that a Kernel session can look different depending on the chosen control method, profile, and network route. A rule built around one fixed IP range or one JavaScript property is unlikely to describe every session.
## How Kernel can reach a website
An agent may drive a Kernel browser with Playwright, send computer-control commands, or invoke browser tools over MCP. The resulting browser then makes requests to the target site. With a profile, a later run can reuse cookies and authenticated state from an earlier session.
That persistence is operationally useful, but it changes the risk for a protected site. Automation that reaches an authenticated workflow can do more than an anonymous page fetch. The policy should account for the resource, account state, request sequence, and declared purpose.
## Why layered detection matters
Kernel documents browser configuration, proxy options, persistent profiles, and input methods designed to reduce common automation indicators. None of those features provides a universal signature that a website can safely treat as "Kernel."
Detection therefore has to look beyond one property. Connection and request evidence can be combined with browser observations and behavior across the session. This also reduces the chance of blocking an ordinary visitor merely because one signal resembles automation.
## Centinel benchmark result
In our September 2026 benchmark, Centinel detected Kernel.
Identification is the start of the policy decision. Teams can observe an approved automation, limit it to specific resources, rate-limit a costly route, or block activity that violates the site's rules. [Centinel bot management](https://www.centinelanalytica.com/bot-management), the product tested in that benchmark, lets you make each of those choices per route. Test it against your own traffic.
## What to check in your own traffic
For suspected Kernel traffic, review:
- whether the session is anonymous or reusing authenticated state;
- which application routes and browser actions repeat;
- whether IP, browser identity, or timing changes while the task stays the same;
- whether the activity belongs to an approved agent, test, or integration.
That session-level view is more useful than chasing one cloud-browser fingerprint.
[How to detect browser automation](https://www.centinelanalytica.com/blog/detect-browser-automation) provides a practical evidence matrix for this review.
## Sources
- [Kernel: Create a cloud browser](https://www.kernel.sh/docs/introduction/create)
- [Kernel: Bot anti-detection overview](https://www.kernel.sh/docs/browsers/bot-detection/overview)
- [Kernel: Stealth mode](https://www.kernel.sh/docs/browsers/bot-detection/stealth)
- [Kernel: MCP server](https://www.kernel.sh/docs/reference/mcp-server)
---
Page: https://www.centinelanalytica.com/blog/what-is-camoufox
# Camoufox in 2026: does it beat bot detection?
Camoufox is a Firefox fork that spoofs fingerprints in C++. What it changes, the leaks its docs admit, and how Centinel detected it in a 2026 test.
By Frederick Jahn. Published 2026-09-03, updated 2026-10-13.
Camoufox is an open-source, Firefox-based anti-detect browser built for browser agents and other automated work. It has a Python interface designed to work with existing Playwright code.
## What Camoufox changes
Camoufox modifies browser behavior below the usual page-script layer. The [official feature reference](https://camoufox.com/) says fingerprint values are intercepted in the C++ implementation rather than overwritten with injected JavaScript. Supported changes include navigator properties, screen and viewport values, WebGL, fonts, locale, timezone, WebRTC data, and related request headers.
The project also isolates Playwright's internal page-agent code and uses BrowserForge to generate device characteristics. Its aim is consistency: a reported operating system, graphics device, screen, fonts, locale, and network identity should make sense together.
This is different from a small stealth script, such as [Playwright Stealth](https://www.centinelanalytica.com/blog/what-is-playwright-stealth), that rewrites a few JavaScript properties after launch. It is also more operationally involved because the browser fork and its fingerprint data have to keep pace with Firefox and with new detection checks.
## Where it gets used
Camoufox is intended for AI agents and browser automation that need Playwright compatibility while presenting varied Firefox identities. Its documentation includes fingerprint rotation, proxy-aware geolocation and locale settings, remote hosting, and human-like cursor movement.
Those features can support legitimate automated testing and agents. They can also be used for scraping or other activity a site has not approved. The browser name does not settle that question; the requested resource and observed behavior do.
## What Camoufox's documentation says about detection
The project's [2026 status note](https://camoufox.com/stealth/) is unusually direct. After a maintenance gap, the maintainers reported that an older Firefox base and newly found fingerprint inconsistencies had reduced performance, and that active development had resumed.
Camoufox also documents two remaining detection paths. A generated fingerprint can contain values that do not agree with one another, and interaction analysis may expose scripted mouse movement or timing. Moving spoofing into C++ removes some JavaScript-patching artifacts, but it does not remove the need for consistent browser, network, and behavioral evidence. The guide on [how sites detect automated browsers](https://www.centinelanalytica.com/blog/detect-browser-automation) covers those layers.
## Centinel benchmark result
Centinel detected Camoufox in our September 2026 benchmark.
Camoufox belongs in a regression suite precisely because its implementation differs from stock Playwright and JavaScript stealth plugins. Testing only one automation family leaves a detector with an easy blind spot.
## See what gets through your bot protection
Browser-like scraping can be hard to distinguish from ordinary visits. Request a free site audit to see which scraping tools reach your site. We run real scraping tools against your site, review the results, and email you the report.
[Request a free site audit](https://www.centinelanalytica.com/free-audit)
## Build a useful Camoufox test
Do not block a visitor merely because one Firefox value looks unusual. Compare the browser's claims with the connection and request, then assess what the session does. A public page, an account flow, and a protected data endpoint carry different risks.
Use the evidence to choose a policy: observe uncertain traffic, rate-limit expensive repetition, challenge when more confidence is needed, and block activity that violates the site's rules. Maintain an explicit allow path for approved agents and automation. [Centinel bot management](https://www.centinelanalytica.com/bot-management), the product tested in that benchmark, applies each of these responses per route. Test it against your own traffic.
When Camoufox is part of a regression suite, keep the browser build and test inputs with the result. Retest after either Camoufox or the protected application changes. The value of the exercise is not a permanent signature; it is proof that the detection path still works against a browser that was designed to alter the obvious signals.
## Sources
- [Camoufox official repository](https://github.com/daijro/camoufox)
- [Camoufox introduction and feature reference](https://camoufox.com/)
- [Camoufox stealth design and current status](https://camoufox.com/stealth/)
---
Page: https://www.centinelanalytica.com/blog/what-is-firecrawl
# What is Firecrawl? How it scrapes and is detected
Firecrawl turns web pages into Markdown or structured data for AI agents. How it fetches, what your site sees, and how Centinel detected it in a 2026 test.
By Frederick Jahn. Published 2026-09-02, updated 2026-09-30.
Firecrawl is a web data API for software and AI agents. It can search the web, fetch one page, crawl a site, and return cleaned content as Markdown or structured data. It is also open source, so "Firecrawl traffic" does not always mean traffic from Firecrawl's hosted service.
## What Firecrawl does
The basic operation is `scrape`: send a URL and choose an output format. Firecrawl's [scrape API](https://docs.firecrawl.dev/api-reference/endpoint/scrape) can return Markdown, HTML, links, screenshots, or structured output. It can also wait for a page, perform browser actions, parse PDFs, select a location, and choose a proxy strategy.
Other endpoints have different jobs. Search finds pages. Crawl follows links through a site. Map discovers URLs. These are useful when an agent needs current web material but should not spend its context window parsing navigation, banners, and raw markup.
Firecrawl is available as a hosted service and as [open-source software](https://github.com/firecrawl/firecrawl). A hosted request and a self-hosted deployment need not share the same network origin, browser setup, or operating pattern. Hosted-only is the wrong mental model.
## How agents use it
An application can ask Firecrawl for a page instead of visiting the page in its own browser. The API fetches the target, renders JavaScript when needed, cleans the result, and returns data that the application can use in retrieval, research, monitoring, or extraction workflows.
This creates two separate connections. The application talks to Firecrawl; the selected Firecrawl deployment talks to the target website. The target sees the latter request, not the end user's original browser or IP address.
That does not make every Firecrawl request harmful. It does mean that the target site's access decision has to be based on the request it receives and the surrounding behavior, not on an assumption about the person or application that initiated the job.
## Why simple checks are unreliable
Firecrawl supports multiple retrieval paths. A page may be handled through basic fetching, a browser-backed scrape, or an enhanced proxy strategy. In `auto` mode, its [proxy documentation](https://docs.firecrawl.dev/features/stealth-mode) says the service can retry with enhanced infrastructure when a basic request fails.
There is no single Firecrawl browser fingerprint that safely describes every deployment. Self-hosting adds another source of variation. An IP blocklist may catch one exit network and miss another, while a browser-only check has little to inspect when the retrieval path does not use a browser.
Defenders get a more reliable answer by combining connection, request, browser, identity, and behavior evidence. Repeated navigation patterns, the resources requested, and the rate of activity matter alongside the properties of any one request. For browser-backed scrapes, the guide on [how sites detect automated browsers](https://www.centinelanalytica.com/blog/detect-browser-automation) covers the browser layer of that evidence.
## Centinel benchmark result
In our September 2026 benchmark, Centinel detected Firecrawl.
Once traffic is identified, policy should follow the resource and behavior. A public documentation page may tolerate retrieval that an authenticated account workflow, search endpoint, or expensive application route should not. The [scraping and AI crawler use case](https://www.centinelanalytica.com/use-cases/scraping-and-ai-crawlers) shows how Centinel applies that kind of per-route policy to scrapers such as Firecrawl.
## What to check in your own traffic
When investigating possible Firecrawl activity, ask:
- Which routes were requested, and were they public or access-controlled?
- Did the client fetch one page, enumerate a section, or traverse the whole site?
- Does the same sequence recur across changing IP addresses or browser identities?
- Is there a declared integration or approved automation job that explains it?
This turns bot detection into an access-control decision rather than a label applied to every web data tool.
For the wider policy question, [What is web scraping?](https://www.centinelanalytica.com/blog/what-is-web-scraping) separates collection mechanics from the decision to allow or restrict them.
## Sources
- [Firecrawl: Scrape API](https://docs.firecrawl.dev/api-reference/endpoint/scrape)
- [Firecrawl: Search API](https://docs.firecrawl.dev/api-reference/endpoint/search)
- [Firecrawl: Enhanced proxy mode](https://docs.firecrawl.dev/features/stealth-mode)
- [Firecrawl open-source repository and self-hosting guide](https://github.com/firecrawl/firecrawl)
---
Page: https://www.centinelanalytica.com/blog/what-is-browserbase
# What is Browserbase? Cloud browsers for AI agents
Browserbase rents remote Chromium sessions to AI agents over an API. How sessions work, what a target site sees, and how Centinel detected it in a 2026 test.
By Frederick Jahn. Published 2026-09-02, updated 2026-09-30.
Browserbase rents remote browser sessions to developers. Instead of running and maintaining Chromium on their own servers, they create a session through an API and connect their automation code to it.
## What Browserbase provides
A Browserbase session is an isolated browser instance in the cloud. The Sessions API returns a connection URL that works with browser-automation tools such as Playwright, Puppeteer, or Selenium. Browserbase's [session documentation](https://docs.browserbase.com/platform/browser/getting-started/create-browser-session) also covers recording, logging, timeouts, proxies, CAPTCHA handling, browser identity settings, and persistent contexts.
Those pieces solve different operational problems. Recordings and Live View help a developer inspect a failed run. A context can carry cookies, authentication state, local storage, and other browser data into a later session. Proxies can change the network path or geolocation. Browser identity settings alter what the remote browser presents to a website.
Browserbase also develops [Stagehand](https://www.centinelanalytica.com/blog/what-is-stagehand), an open-source SDK for browser agents. Stagehand adds higher-level actions and structured extraction while Browserbase supplies the remote browser. The two are related, but they are not the same product: developers can connect other automation frameworks to Browserbase, and Stagehand can also run locally.
## How it can be used for data retrieval
Cloud browsers are useful when a workflow needs JavaScript rendering or interaction rather than a simple HTTP request. Browserbase's own [web data retrieval guide](https://docs.browserbase.com/use-cases/web-data-retrieval) shows Stagehand and Playwright sessions extracting structured data, rotating proxies, handling CAPTCHAs, and running concurrent sessions.
The same infrastructure can support testing, authenticated workflows, form automation, or data extraction. A website therefore cannot infer intent just because a request came from an automated browser. The useful questions are narrower: is the client automated, what resources is it requesting, and does that behavior comply with the site's access policy?
## A product name does not tell you the intent
The target receives requests made by the remote browser, not a direct connection from the developer's laptop. The visible details depend on how the session was configured. A session may use Browserbase's network infrastructure or a configured proxy, and it may start fresh or reuse a saved context.
That makes one-field rules brittle. An IP rule can lose value when the exit network changes. A user-agent check only sees the declared browser identity. A JavaScript check may find an obvious automation artifact in one setup and nothing useful in another.
A stronger review combines identity, request, browser, connection, and behavior evidence. It should also keep the policy decision separate from classification. A known automation session may be approved, monitored, rate-limited, challenged, or blocked depending on what it is doing.
## Centinel benchmark result
In our September 2026 benchmark, Centinel detected Browserbase.
For a site operator, the next step is to decide what that traffic may access. Start with observation, record which routes and workflows it reaches, and apply a specific rule rather than treating every cloud-browser session as abusive. The product tested in that benchmark, [Centinel bot management](https://www.centinelanalytica.com/bot-management), sets rules like this per route. Try it on your own traffic.
## What to check in your own traffic
If Browserbase or another managed browser service such as [Kernel](https://www.centinelanalytica.com/blog/what-is-kernel-sh) appears in an investigation, review the whole session:
- Which pages, APIs, forms, or account workflows did it reach?
- Did it reuse an authenticated state or arrive as a fresh visitor?
- Does the request sequence repeat across identities or network paths?
- Is the automation declared, expected, and covered by an existing agreement or allow rule?
These checks produce a defensible policy decision. A copied browser string does not.
Use the evidence model in [How to detect browser automation](https://www.centinelanalytica.com/blog/detect-browser-automation) to turn those observations into a repeatable review.
## Sources
- [Browserbase: Create a browser session](https://docs.browserbase.com/platform/browser/getting-started/create-browser-session)
- [Browserbase: Core features](https://docs.browserbase.com/platform/browser/core-features/overview)
- [Browserbase: Web data retrieval](https://docs.browserbase.com/use-cases/web-data-retrieval)
- [Browserbase Stagehand repository](https://github.com/browserbase/stagehand)
---
Page: https://www.centinelanalytica.com/blog/what-is-browser-use
# Browser Use: how the AI browser agent gets caught
Browser Use lets a language model drive a real browser. How the agent works, what your logs see, and how Centinel detected it in a September 2026 test.
By Frederick Jahn. Published 2026-09-02, updated 2026-09-30.
Browser-use is an open-source Python framework for building agents that operate a web browser. Give the agent a task and a language model, and it can open pages, click controls, enter text, fill forms, and extract data.
## What Browser-use controls
Browser-use sits above the browser. It handles the agent loop: reading the page, asking the model what to do next, carrying out that action, and continuing until the task is complete. The [open-source project](https://github.com/browser-use/browser-use) supports its own hosted models as well as models from other providers and is available under the MIT license.
The browser underneath is configurable. The [current CLI](https://docs.browser-use.com/open-source/browser-use-cli) can launch managed headless Chromium, connect to an existing Chrome profile, or use a cloud browser. The Python library also accepts a local executable, a browser channel, proxy settings, profile data, or a Chrome DevTools Protocol (CDP) connection to a remote browser. That range matters to defenders because there is no single network or browser fingerprint that identifies every Browser-use job.
## Where its traffic appears
The project's own examples cover form filling and structured data extraction. Its documentation also recommends the Python library for repeatable work such as scraping, monitoring, and quality assurance.
These jobs can look ordinary one page at a time. An agent can use a normal Chrome profile with existing cookies, follow multi-step flows, and wait for the model between actions. It can also run headlessly in a fresh local or cloud browser. A rule written for one of those setups may say little about another.
Browser-use is the orchestration layer, not an identity sent in an HTTP header. A site usually observes the chosen browser, network path, requests, and interaction pattern rather than the name "Browser-use." It is one source of the [agentic traffic](https://www.centinelanalytica.com/blog/what-is-ai-agent-traffic) that sites now need to classify.
## Why one browser check is not enough
A visible automation property can be useful evidence, but it is not a durable identity. Browser-use lets an operator change browsers, profiles, launch arguments, and remote infrastructure. A separate stealth layer can alter some browser-facing properties again.
Detection therefore needs context. Check whether the connection, request headers, browser claims, and session behavior agree. Pay attention to what the session touches, how it moves through the flow, and whether the same task pattern recurs across different apparent identities. Treat each observation as evidence, not a verdict on its own.
For a practical model, see [how to detect browser automation](https://www.centinelanalytica.com/blog/detect-browser-automation).
## Centinel benchmark result
Centinel detected Browser-use in our September 2026 benchmark.
That result makes Browser-use a useful regression case for a bot-protection program. Retest it when the browser or detection logic changes.
## Set policy from the job
Start with the resource, not the tool name. Automation on a public status page deserves a different policy from automation attempting to log in, reserve inventory, or extract protected data.
Record enough evidence to explain the decision, then choose the least disruptive control that fits the flow. Observation or rate limits may be enough for low-risk traffic. Sensitive or abusive flows may justify a challenge or block. Preserve an allow path for automation the site has approved. [Centinel bot management](https://www.centinelanalytica.com/bot-management), the product tested in that benchmark, sets these controls per route. Test it against your own traffic.
## Sources
- [Browser-use open-source repository and usage guide](https://github.com/browser-use/browser-use)
- [Browser-use CLI browser modes](https://docs.browser-use.com/open-source/browser-use-cli)
- [Browser-use browser configuration reference](https://docs.browser-use.com/open-source/customize/browser/all-parameters)
---
Page: https://www.centinelanalytica.com/blog/why-your-waf-cant-see-them
# Why WAF rules can miss browser-like crawlers
Learn where request rules end, where session evidence can help, and how to test the bot controls enabled in your WAF without relying on category claims.
By Frederick Jahn. Published 2026-07-22, updated 2026-10-11.
A request can be valid HTTP and still belong to an unwanted automated workflow. This is not a universal WAF failure. It is a scope question: which evidence does the configured control receive, how much state does it keep, and which application decision must it make?
Modern CDN, WAF, and bot products vary widely. Some include browser challenges, managed identities, behavioral models, or session features. Evaluate the product and plan you run, not a simplified category description.
## What a WAF decision can see
The [OWASP WAF overview](https://owasp.org/www-community/Web_Application_Firewall) describes a web application firewall as an application firewall for HTTP applications, typically operating as a reverse proxy. That position can provide request and response context useful for application-security rules.
[RFC 9110](https://www.rfc-editor.org/rfc/rfc9110) defines the semantics of an HTTP request. A request may expose a method, target, fields, and message content, but those elements do not prove the operator or purpose behind it.
A configured WAF or bot module may also have access to reputation, verified crawler lists, rate state, browser signals, and prior requests. Confirm the actual scope:
- Does it cover every host, route, method, and origin path?
- Does it inspect only the current request, or correlate a session?
- Which client and browser signals are collected?
- How are identities and allowlists updated?
- Which response actions are available on each route?
- What data is unavailable because of encryption, architecture, or plan limits?
The answers determine the decision boundary more accurately than the product label.
## Where straightforward rules work well
Rules are effective when the policy can be expressed using evidence the control sees. Examples include a bounded request rate, a known malicious pattern, a route that should never be public, or a crawler identity that can be verified against current operator data.
[OpenAI's crawler documentation](https://developers.openai.com/api/docs/bots), for example, distinguishes [GPTBot](https://www.centinelanalytica.com/bots/gptbot), [OAI-SearchBot](https://www.centinelanalytica.com/bots/oai-searchbot), and [ChatGPT-User](https://www.centinelanalytica.com/bots/chatgpt-user) and publishes network data for verification. A rule can use that verified identity as input to a resource policy. The name in the user-agent field alone remains untrusted input.
Rules become less decisive when the same request shape is common to both legitimate and unwanted traffic. A Chromium request from a consumer network is not unusual. Blocking that combination broadly can create false positives, while allowing it broadly can leave the abusive workflow unchanged.
## Where a session view can add evidence
Some automation becomes clearer as a sequence. The useful observations may include route order, concurrency, retries, state reuse, navigation depth, interaction timing, and the application actions completed across several requests.
Session evidence can help when:
- the unwanted behavior is defined as a sequence rather than one request;
- the requests can be correlated without unsafe or unbounded state;
- the chosen integration observes the whole relevant path;
- legitimate automation and unusual users are represented in the baseline;
- the response can be applied before material harm occurs.
It does not make uncertainty disappear. Shared devices, changing networks, privacy controls, accessibility tools, and internal automation can produce uncommon sequences. Test each feature against labeled examples and preserve an unresolved state when the evidence conflicts.
Centinel is designed to contribute browser, network, crawler, and session evidence for these decisions. Available evidence and enforcement depend on your integration. A browser component has no authority over a direct request that never runs it, while a server or edge component needs the required application context supplied safely.
## Test the gap before adding a product
Begin with a concrete failure case. “Bots get through” is too broad. “This automated cohort completes this expensive route sequence while the current policy allows it” can be tested.
Build a small matrix:
| Cohort | Expected policy | Why it belongs in the test |
| --- | --- | --- |
| Ordinary supported browsers | Allow | Establish normal behavior and completion |
| Privacy and accessibility configurations | Allow or recover | Find legitimate edge cases |
| Approved monitoring or integration | Scoped allow | Prevent damage to required automation |
| Verified named crawler | Route-specific decision | Test identity and purpose separation |
| Stock browser automation | Defined by use case | Establish a reproducible automated baseline |
| Reproduced abusive workflow | Mitigate | Measure the actual gap |
Run the current configuration first. Record enabled controls, versions, expected decisions, reason codes, latency, and downstream work. Add or change one layer at a time. Use observation mode and a holdout where practical.
A successful test includes error review. Count false positives, false negatives, unresolved decisions, challenge failures, and lost legitimate completions. If ground truth is weak, report the uncertainty instead of converting the result into a detection percentage.
## Use both layers without creating policy conflicts
When several components participate, give each one a clear responsibility. A common arrangement is:
```
request and application-security controls
-> automation evidence and session policy
-> route-specific allow, observe, rate-limit, challenge, or deny
-> application
```
The actual order depends on the integration. Verify how allow rules, cached responses, challenges, retries, and direct-origin access interact. One layer should not silently undo another layer's decision.
Define bounded failure behavior. If classification is unavailable, decide what each resource should do. A low-risk public page and an account recovery operation may need different fallbacks.
## Do not buy a category label
A specialist bot product is useful only if its evidence closes a measured gap in the current stack. Ask every provider, including Centinel, to demonstrate:
- coverage of the relevant traffic path;
- the independent evidence families used for the decision;
- behavior on ordinary and approved automated cohorts;
- false-positive review and recovery;
- latency and failure behavior;
- reason codes, retention, and operator controls;
- a safe rollout and rollback plan.
If the current CDN or WAF bot controls already meet those requirements, another layer may add no value. If they do not, use the failed test to specify the missing capability instead of assuming every product in a new category supplies it.
See the [configured comparison method](https://www.centinelanalytica.com/compare/centinel-vs-cdn-bot-protection) for the full test plan and the [bot management overview](https://www.centinelanalytica.com/bot-management) for the broader architecture checklist.
---
Page: https://www.centinelanalytica.com/blog/what-is-ai-agent-traffic
# What is agentic traffic? AI agents explained
Agentic traffic comes from software that picks and performs steps toward a user's goal. How it differs from crawlers and fetchers, and how to measure it.
By Frederick Jahn. Published 2026-04-20, updated 2026-10-11.
Agentic traffic is web traffic created by software that chooses and performs
steps toward a goal. A user might ask an agent to compare products, research a
topic, or complete a workflow. The agent then decides which pages or APIs to
visit and what to do next.
A crawler collecting pages on a schedule is automated, but it is not acting on
one user's live task, and neither is a search bot building an index. Reports
often count all three as "AI traffic". They need different access rules: a
crawler reads public pages, while an agent may log in, fill a form, or place an
order.
## What agentic traffic means
There is no HTTP header that turns a request into agentic traffic. The useful
definition is operational:
> Agentic traffic is one or more web requests made by software that can
> select its next action while working toward a user or application goal.
The sequence may include page navigation, JavaScript execution, API calls, or
state-changing actions such as submitting a form. It may also stop after one
request if the task is simple. The test is whether the software chose the next
step. Page count and the presence of a large language model do not decide it.
Sites also receive requests from AI products that are not agentic in this
sense. OpenAI, Anthropic, and Perplexity each document separate clients for
model training, for search indexing, and for retrieval started by a user.
OpenAI states that `GPTBot` may collect training content, `OAI-SearchBot`
supports ChatGPT search, and `ChatGPT-User` performs certain user-triggered
actions ([OpenAI crawler documentation](https://developers.openai.com/api/docs/bots)).
Anthropic separates `ClaudeBot`, `Claude-SearchBot`, and `Claude-User` the same
way ([Anthropic crawler documentation](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)).
## Classify the trigger before the client
Start with why the request happened. The client name comes second.
```
Was the request triggered by a live user or application task?
|
+-- No or not known
| |
| +-- Building a search index? -> search crawler
| +-- Collecting model-development data? -> training crawler
| +-- Purpose cannot be established? -> unidentified automation
|
+-- Yes
|
+-- Fetching content for an answer? -> user-directed retrieval
+-- Choosing and performing steps? -> agentic session
```
If the trigger cannot be established, record it as unknown.
The same distinction works as a traffic matrix:
| Traffic class | Trigger | Typical request shape | Published examples | First policy question |
| --- | --- | --- | --- | --- |
| Search crawler | Operator's indexing system | Repeated discovery and refresh requests | `Googlebot`, `OAI-SearchBot`, `Claude-SearchBot`, `PerplexityBot` | Do we want this search product to discover and surface the content? |
| Training or model-development crawler | Operator's collection schedule | Broad or repeated content collection | `GPTBot`, `ClaudeBot` | Has this use of the content been approved? |
| User-directed retrieval | A user asks an AI product to fetch or answer something | One or more fetches tied to the request | `ChatGPT-User`, `Claude-User`, `Perplexity-User` | Should this user-triggered fetch reach the requested resource? |
| Agentic session | A user or application delegates a task | A sequence of chosen reads or actions, sometimes through a browser | No universal identifier | Which actions may this automated session perform? |
The examples were checked against operator documentation on October 6, 2026.
Names and stated purposes change, so recheck them before you rely on the table.
Perplexity says `PerplexityBot` supports search and indexing, and
`Perplexity-User` may fetch a page after a user asks a question
([Perplexity crawler documentation](https://docs.perplexity.ai/docs/resources/perplexity-crawlers)).
`Google-Extended` belongs in none of the rows. Google documents it as a
`robots.txt` product token that controls certain uses of content Google has
already crawled. It has no HTTP user agent of its own, so a log search for the
string finds no real Google traffic
([Google's crawler documentation](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers)).
## What the classes look like in logs
A single log row can tell you the claimed client, source address, route,
method, status, time, and response size. It usually cannot tell you whether a
human prompt, a scheduler, or another application caused the request.
Collect enough context to classify a sequence later:
- the source IP observed at a trusted network boundary;
- the full user-agent header and any verified operator identity;
- timestamp, method, host, path, status, and bytes transferred;
- a request or trace identifier;
- an authenticated account or first-party session identifier, when present;
- connection and browser signals available at your edge;
- the classification result, reason, and rule version.
Do not build sessions from IP addresses alone. Corporate gateways, mobile
networks, privacy relays, and proxy services can place unrelated users behind
one address. The reverse is also possible: one automated workflow may move
between addresses.
A declared search or training crawler is the simplest case to label, once its
identity has been verified. Named user fetchers provide evidence about the
trigger too, because their operators publish their purpose. An ordinary browser
session is harder. It might be a person, a browser agent, a test runner, or a
scraper using browser automation. Agent frameworks such as [Browser Use](https://www.centinelanalytica.com/blog/what-is-browser-use) and [Stagehand](https://www.centinelanalytica.com/blog/what-is-stagehand) drive
an ordinary Chromium browser, so the name of the framework never reaches your
logs.
## What a user agent cannot tell you
A user-agent header is a claim supplied by the client, not proof of who sent the
request. Checking that claim against operator evidence is covered in [How to verify AI agents](https://www.centinelanalytica.com/blog/how-to-verify-ai-agents). For classification, keep two
separate fields in the traffic record:
| Field | Question | Example values |
| --- | --- | --- |
| Identity | Who can we verify sent this? | OpenAI, Anthropic, unknown |
| Traffic class | What triggered the request and what is it doing? | search crawl, training crawl, user retrieval, agentic session, unknown |
A verified operator can run several types of client, so identity does not
settle the class. The absence of a named AI user agent proves even less: agents
can use direct HTTP clients or browser automation, and detection may establish
that a session is automated without revealing who operates it.
## How to recognize an agentic session
Look at the sequence and the actions it attempts. Agentic traffic often moves
through a task-shaped path: discover a page, inspect details, follow a relevant
link, then call an API or submit data. The exact path depends on the delegated
goal.
None of these signals is conclusive on its own:
- route transitions that follow content or application state;
- a switch from read-only requests to state-changing methods;
- browser execution followed by an API request;
- repeated attempts after validation or permission errors;
- timing and navigation patterns inconsistent with the site's human sessions.
Use connection, browser, request, account, and behavioral evidence to decide
whether the client is automated. Use the request sequence to describe what the
automation is doing. Leave the operator and trigger unknown when the evidence
does not establish them.
Unidentified automation may be a price scraper, uptime monitor, SEO tool,
fraud bot, test runner, or a custom agent. Label it agentic only when the
sequence shows chosen steps.
## Match the response to the traffic class
The class tells you which policy question to ask. The answer still depends on
the route.
| Traffic class | Sensible starting point |
| --- | --- |
| Search crawler | Verify the operator, then decide which content should be discoverable in that search product. |
| Training crawler | Apply an explicit model-development access rule. Do not inherit the search-crawler rule. |
| User-directed retrieval | Verify named clients where possible, protect restricted resources, and set bounded request limits. |
| Agentic session | Apply normal authorization to every action. Add abuse controls around login, checkout, forms, and other state changes. |
| Unidentified automation | Use the site's default automated-traffic policy until stronger evidence changes the classification. |
Verification does not override authorization. A genuine user-directed fetcher
does not gain access to a subscriber-only article without the required
credentials. An authenticated agent should not be able to submit an action its
account is not permitted to perform.
The policy may allow, observe, rate-limit, challenge, or block a request. Make
that decision using the resource, identity result, behavior, and business rule
together. [How to control AI agent access](https://www.centinelanalytica.com/blog/how-to-control-ai-agent-access) covers that layer in detail.
## How to measure agentic traffic
One chart for every AI-related request mixes scheduled crawls with delegated
tasks. Report traffic class and identity status separately. A report should
include:
- requests and sessions by traffic class;
- verified, failed, unknown, and unidentified identity counts;
- pages and bytes served by route group;
- cache hits, cache misses, response status, and origin work where available;
- state-changing attempts and their authorization result;
- policy actions and reason codes;
- the classifier version used for the report.
Keep raw observations separate from inferred labels. If a classification rule
changes, versioning lets you reprocess historical data or at least explain why
the chart moved. Otherwise a taxonomy update can look like a traffic surge.
For scheduled crawler mechanics, see [What is an AI crawler?](https://www.centinelanalytica.com/blog/what-is-an-ai-crawler). For browser-controlled automation, see
[How to detect browser automation](https://www.centinelanalytica.com/blog/detect-browser-automation).
## Finding the classes on your own site
Centinel helps teams identify and control automated traffic. When you evaluate
any product for this, test whether it keeps operator identity, traffic class,
and policy outcome separate, and what it reports when a request cannot be
classified or verified. The [bot management guide](https://www.centinelanalytica.com/bot-management) has the
full evaluation checklist.
To see which classes reach your current pages before you set policy,
[request a site audit](https://www.centinelanalytica.com/free-audit).
## Sources
- [OpenAI crawler documentation](https://developers.openai.com/api/docs/bots)
- [Anthropic crawler documentation](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)
- [Perplexity crawler documentation](https://docs.perplexity.ai/docs/resources/perplexity-crawlers)
- [Google's common crawler documentation](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers)
---
Page: https://www.centinelanalytica.com/blog/how-to-control-ai-agent-access
# How to control AI agent access
Build a practical access policy for AI crawlers and agents with a resource matrix, explicit identity states, bounded responses, and tested failure behavior.
By Frederick Jahn. Published 2026-04-20, updated 2026-10-11.
An AI-agent access policy should answer one question: what may this client do
with this resource, given the identity and behavior you can actually verify?
That produces better rules than a site-wide "allow AI" or "block AI" switch.
A search crawler fetching public stories, an assistant opening a page for a
reader, and an automated browser attempting account actions do not deserve the
same response.
Start with a resource matrix. Give every row an owner, review date, logging
requirement, and failure behavior. Then test the policy with verified,
unverified, spoofed, and unknown traffic before enforcing it.
## Start with purpose and resource
Classify the request by why it exists, not by whether somebody has attached the
label "AI" to it. Useful working classes include search indexing, model
development, retrieval for a current answer, user-directed work, approved
automation, and unidentified automation. [What is AI agent traffic?](https://www.centinelanalytica.com/blog/what-is-ai-agent-traffic) defines those boundaries in more
detail.
The operator documentation confirms that these purposes are not
interchangeable. OpenAI documents separate controls for `GPTBot`,
`OAI-SearchBot`, and `ChatGPT-User`. [OpenAI crawler documentation](https://platform.openai.com/docs/bots) Anthropic likewise
documents `ClaudeBot`, `Claude-SearchBot`, and `Claude-User` separately.
[Anthropic crawler documentation](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)
Purpose is still only one input. The same client may be acceptable on a public
article and unacceptable on an account endpoint. List the actual resource
groups before deciding what to do with any crawler name.
## Write the policy as a resource matrix
The example below leans conservative. Replace `YYYY-MM-DD`, the owners,
identity methods, limits, and failure choices with decisions from your own
team. One row often becomes several when traffic classes need different
responses on the same resource.
| Resource | Traffic class | Identity requirement | Response | Owner | Review | Logging | Failure behavior |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Public articles | Verified search or retrieval crawler; user-directed agent; unknown automation | Exact documented token plus the operator's current proof for a named allow; otherwise keep identity unknown | Allow approved classes; observe or apply a scoped limit to unknown traffic when behavior warrants it | Editorial and platform | YYYY-MM-DD | Class, identity state, route, bytes, response, rule reason | Keep the previous known-good public policy when verification is temporarily unavailable |
| Subscription archives | Subscriber or agent acting with subscriber authority | Valid first-party session or delegated credential with archive permission | Enforce the same authorization as the subscriber; limit bulk export separately | Subscription product and security | YYYY-MM-DD | Account or credential ID, scope, resource, result, reason; never log secrets | Deny protected access when authorization cannot be checked |
| Site search | Reader, user-directed agent, approved integration, or unknown automation | Session or account for higher budgets; explicit unknown state for anonymous use | Allow a bounded query budget; return `429` when the selected key exceeds it | Search or platform owner | YYYY-MM-DD | Limit key type, route cost, count, response, retry value | Fall back to the documented anonymous budget, not an unlimited query path |
| Feeds | Feed reader, search crawler, retrieval client, or unknown fetcher | Verification for named exceptions; no invented identity for an unidentified client | Serve the public feed under its cache and request budget; deny restricted feed variants without authorization | Editorial platform | YYYY-MM-DD | Feed, identity state, bytes, response, cache outcome | Serve a safe cached public response or the chosen unavailable response |
| Login and account | Reader, delegated agent, monitoring, or unidentified automation | Valid authentication and request protections; bot classification never grants account permission | Authenticate, authorize, rate-limit abusive actions, and use a reviewed step-up check when appropriate | Identity and security | YYYY-MM-DD | Attempt outcome, account-safe identifier, action, rule reason; exclude credentials | Fail closed for state-changing actions when authorization or policy dependencies are unavailable |
| APIs | Partner integration, user-delegated agent, internal job, or unknown client | Credential with the required audience and scope; bind delegated work to the user's permissions | Allow the scoped operation; use `401`, `403`, or `429` according to the actual failure | API owner and security | YYYY-MM-DD | Credential identifier, scope, endpoint, result, limit state; exclude tokens and bodies by default | Deny protected operations when authentication, authorization, or policy checks cannot complete |
Use the matrix to record actual decisions. If the platform cannot produce an
identity state or response named in a row, change the row or the implementation
before launch. A control that exists only in the policy document will fail its
first real test.
Precedence needs to be written down too. Resource authorization wins over a
crawler allow rule. A valid crawler identity must never unlock a subscription
archive, account action, or API operation that lacks the required credentials.
## Decide identity before response
A user-agent string is a claim. It is useful because it tells you what a client
wants to be called, but another HTTP client can copy the same value. Copying is
common: on the median site in our [crawler verification data](https://www.centinelanalytica.com/blog/fake-crawler-report), most requests using several AI crawler names
were fake.
Record claim and verification separately.
Use distinct states:
| Identity state | Meaning | Safe policy use |
| --- | --- | --- |
| Verified named client | The request made an exact configured claim and passed that operator's current verification method | Apply the rule for that named client and resource |
| Authenticated user or delegated client | The request has valid first-party or delegated credentials with known permissions | Authorize only the operations covered by those permissions |
| Claimed, verification unavailable | The verifier timed out, its evidence source failed, or required metadata is missing | Use the row's temporary-failure behavior; do not silently convert it to verified |
| Claimed, verification failed | The request made a named claim but did not match the configured proof | Apply the explicit failed-claim policy and record the reason |
| Unknown | The request has no useful claim or no supported proof | Use resource and behavior policy without assigning an operator identity |
Verification methods differ by operator. Google documents two methods for its
crawlers: matching against its published IP ranges or performing reverse and
forward DNS checks. [Google crawler verification](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests)
Other operators publish their own tokens, ranges, or instructions. Use the
method attached to the exact client instead of combining every range from one
company into a company-wide identity.
Keep verification evidence outside editorial copy. Store the source URL,
retrieval time, parser version, last known-good data, and refresh result. Bound
network calls, accepted response size, redirects, and parsing. A failed refresh
should not erase current evidence or widen access by accident.
The implementation details and current operator table are in [How to verify AI agents](https://www.centinelanalytica.com/blog/how-to-verify-ai-agents).
## Choose the smallest useful response
Most policies need five response families: allow, observe, rate-limit,
challenge, and deny. They are not an escalation ladder that every request must
climb.
**Allow** serves the requested resource. Keep the identity state and rule
reason in the log so approved traffic remains measurable.
**Observe** serves the normal response while recording evidence for a decision
with an owner and end date. Observation without an end date becomes accidental
policy.
**Rate-limit** constrains a defined key, route, window, and burst. The `429 Too Many Requests` status means the client sent too many requests in a given
period. [RFC 6585](https://www.rfc-editor.org/rfc/rfc6585.html#section-4) allows
a `Retry-After` header and leaves the choice of counting method and client
identity to the server. An IP-only key can combine unrelated users and miss a
distributed client, so record why the selected key fits the route.
**Challenge** adds evidence where a browser is expected and a wrong block
would harm a reader. Define accessibility, cookie, timeout, repeat, and loop
behavior. Do not place a browser challenge in front of feed readers, APIs, or
known crawlers that cannot complete it.
**Deny** refuses the request. Use authentication and authorization semantics
for protected resources rather than disguising every denial as bot detection.
[RFC 9110](https://www.rfc-editor.org/rfc/rfc9110.html#section-15.5.2) defines
`401` for a request that lacks valid authentication credentials and requires a
`WWW-Authenticate` challenge. The RFC defines `403` for a request the server
understood but refuses to fulfill. Choose status and response content based on
the real condition and avoid leaking sensitive policy details.
For implementation options and rollback controls, see [How to block AI crawlers](https://www.centinelanalytica.com/blog/how-to-block-ai-crawlers).
## Keep `robots.txt` in its proper role
`robots.txt` publishes rules for crawlers that follow the Robots Exclusion
Protocol. It does not authenticate a client or authorize access to protected
content.
[RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html) specifies how crawlers
select and apply groups. Its security section says the rules are not a form of
access authorization and warns against exposing private-looking paths in the
file. Use authentication and application authorization for resources that
must remain restricted.
The declared rule and enforced behavior should still agree. If the file asks a
named crawler not to fetch a path while the request boundary deliberately
allows it, the organization has two policies. Give somebody ownership of that
conflict rather than assuming the crawler list will settle it.
## Choose failure behavior before rollout
The uncomfortable policy decisions appear when a dependency fails. Make them
before launch.
For each matrix row, decide what happens when:
- the crawler range or verification record cannot refresh;
- DNS verification times out;
- the policy service is unavailable;
- connection or browser evidence is missing;
- a rate-limit store cannot be reached;
- a challenge cannot load or repeats;
- logging is degraded;
- the client matches more than one rule.
Public and protected resources usually need different answers. Continuing with
a previous known-good rule may be reasonable for a public article during a
short verifier outage. An account change or protected API call should not gain
permission because authorization is unavailable.
Avoid one global `fail_open` switch. Store the chosen fallback with the
resource rule, then test it. Define a maximum age for last known-good evidence
and what happens when that age is exceeded.
## Test and review the policy
Build a staging matrix before enabling enforcement:
| Test case | Expected result |
| --- | --- |
| Verified named crawler on an allowed public route | Receives the intended public response and a verified reason code |
| Same crawler on a protected route without credentials | Remains unauthorized |
| Copied crawler user agent from an unrelated source | Does not inherit verified identity |
| Verification source temporarily unavailable | Uses the configured temporary-failure path |
| Verification fails | Uses the failed-claim path, which remains distinct from an outage |
| Unknown automation within the public budget | Receives the normal public response unless another scoped rule applies |
| Search client exceeds its chosen key and window | Receives the documented limit response and retry behavior |
| Browser cannot complete a challenge | Does not enter a loop and has the chosen fallback |
| Policy dependency fails during an account action | Cannot bypass authentication or authorization |
| Previous policy version is restored | Returns the tested pre-change behavior |
Start in observation mode where the route permits it. Compare rule results with
approved crawlers, internal monitoring, normal readers, and confirmed abuse.
Review response counts, bytes, origin work, challenge completion, reader
errors, and unknown-state volume by resource.
Every rule needs an owner, version, review date, reason code, and one-step
rollback. Review sooner when an operator changes its documentation, a browser
or proxy changes the evidence you collect, or an incident exposes a policy
gap. Remove expired exceptions. A forgotten allow rule is still an allow rule.
## Evaluate the deployment, not the diagram
Before relying on Centinel or another control layer, confirm which observations
and responses are available in the deployment you will operate. Ask where the
decision runs, how identity evidence refreshes, what unknown means, which
routes can be observed or limited, what happens during dependency failure, and
how a rule is rolled back.
Use the resource matrix as the acceptance test. A demo that identifies a
crawler but cannot reproduce the row's response and fallback has not yet
implemented the policy.
Use the [bot management evaluation checklist](https://www.centinelanalytica.com/bot-management) to carry those
tests into a product review without collapsing every resource into one global
bot rule.
To inventory the traffic and routes before writing enforcement rules, [request a site audit](https://www.centinelanalytica.com/free-audit).
## Sources
- [OpenAI crawler documentation](https://platform.openai.com/docs/bots)
- [Anthropic crawler documentation](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)
- [Google crawler verification](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests)
- [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html)
- [RFC 6585: 429 Too Many Requests](https://www.rfc-editor.org/rfc/rfc6585.html#section-4)
- [RFC 9110: HTTP Semantics](https://www.rfc-editor.org/rfc/rfc9110.html)
---
Page: https://www.centinelanalytica.com/blog/how-to-verify-ai-agents
# Verify AI crawlers and agents: spot fake GPTBot
Check whether a request claiming to be GPTBot, ClaudeBot or another AI agent is real, using published IP ranges, reverse DNS or signed requests.
By Frederick Jahn. Published 2026-04-20, updated 2026-10-11.
To verify an AI agent, check its claimed identity against the operator's
published IP ranges, forward-confirmed reverse DNS, or a supported signature
scheme with trusted keys. A user-agent header alone is not proof. If no
supported evidence is available, keep the identity unknown rather than granting
trusted access.
First choose what you need to verify. A crawler's operator, a browser's use of
automation, and a user's permission to delegate an action are different questions:
- **Named crawler or fetcher:** use its operator's documented verification method.
- **Browser agent with no operator proof:** evaluate automation and behavior; do
not invent a verified identity from a plausible fingerprint.
- **Agent accessing an account:** require normal authentication and authorization.
Operator verification does not establish the account owner's consent.
A verified crawler can still exceed a retrieval budget or request a restricted
route. Verification establishes only the identity supported by the evidence;
your resource policy decides what that client may do.
## Start with the identity claim
Record four things before building a rule:
1. the user-agent token and full header;
2. the source IP observed at your trusted network boundary;
3. the resource being requested;
4. the verification method published for that client.
Do not take the source IP from `X-Forwarded-For` unless it was set or sanitized
by infrastructure you control. A client can supply that header too.
Use a small set of identity results. It is easier to operate than a vague
confidence score.
| Result | Meaning |
| --- | --- |
| Verified | The claim matches the operator's published verification method. |
| Failed | The client made a named claim, but the required evidence did not match. |
| Unknown | The client made a claim, but no supported proof was available or the check could not finish. |
| Unidentified | The request made no useful operator claim. |
An `Unknown` result is not a soft pass. It means the request has not earned the
privileges attached to a verified identity.
## Verification methods published by operators
Operator methods differ by client and purpose. Store the official source,
verification scope, and last successful review with every catalog entry.
An operator-wide network list does not necessarily verify a specific product.
| Client | Documented purpose | Published verification method |
| --- | --- | --- |
| [Googlebot](https://www.centinelanalytica.com/bots/googlebot) and other Google crawlers | Search and other Google products | Match the source IP against the published ranges for the crawler's category (common crawlers, special-case crawlers, or user-triggered fetchers), or use forward-confirmed reverse DNS with the documented hostname suffix. [Google verification guide](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests) |
| [OAI-SearchBot](https://www.centinelanalytica.com/bots/oai-searchbot), [GPTBot](https://www.centinelanalytica.com/bots/gptbot), OAI-AdsBot, and [ChatGPT-User](https://www.centinelanalytica.com/bots/chatgpt-user) | Search, ads validation, model development, and user-triggered actions are documented separately | Match each user agent to its corresponding published JSON IP list. Do not treat one OpenAI list as proof for every OpenAI client. [OpenAI crawler documentation](https://developers.openai.com/api/docs/bots) |
| [ClaudeBot](https://www.centinelanalytica.com/bots/claudebot), [Claude-SearchBot](https://www.centinelanalytica.com/bots/claude-searchbot), and [Claude-User](https://www.centinelanalytica.com/bots/claude-user) | Model development, search, and user-directed retrieval are documented separately | Anthropic's linked IP list establishes Anthropic origin. Keep the claimed bot purpose separate; the shared list alone does not prove which of the three products sent the request. [Anthropic crawler documentation](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) |
| Applebot | Apple search experiences | Use forward-confirmed reverse DNS under `applebot.apple.com`, or match the source address to Apple's published Applebot CIDR list. [Applebot documentation](https://support.apple.com/en-us/119829) |
| Bingbot | Bing Search | Use forward-confirmed reverse DNS under `search.msn.com`, or use Microsoft's published IP list and refresh it as directed. [Bingbot verification guide](https://www.bing.com/webmasters/help/how-to-verify-bingbot-3905dc26) |
| PerplexityBot and Perplexity-User | Search/indexing and user-triggered retrieval are documented separately | Match each client to the product-specific JSON IP list published by Perplexity. [Perplexity crawler documentation](https://docs.perplexity.ai/docs/resources/perplexity-crawlers) |
Notice what the table does not say. A user-agent match by itself never produces
`Verified`. A request that says `Googlebot` but fails Google's published check
is not verified Googlebot. A request that says `Claude-User` should not be
checked against an unrelated vendor's network list.
Spoofed claims are common. On the median news site in our verification data,
[84% of requests claiming to be GPTBot failed OpenAI's published check](https://www.centinelanalytica.com/blog/fake-crawler-report).
## Verify a published IP range
IP-list verification is a membership check, but the maintenance around it is
where implementations usually go wrong.
Map the exact user-agent token to a configured operator record. Fetch the range
file from the URL in that record, not from a URL supplied by the request. Parse
only the expected schema and valid CIDR values. Put limits on response size,
redirects, request time, and the number of entries accepted.
Keep the last known-good list when a refresh fails. A network timeout should
produce `Unknown`, not turn every matching user agent into either verified
traffic or a spoof. Log the list version and the range that matched so an
operator can explain the decision later.
The request check is then straightforward:
```
claim = identify the user-agent token
record = load the configured record for that exact client
if record has no supported verification method:
result = unknown
else if source IP is inside the client's current published ranges:
result = verified
else:
result = failed
```
Do not combine all ranges from one company into a company-wide allowlist unless
the company documents them that way. OpenAI, for example, publishes separate
range files for different clients and purposes.
## Verify with forward-confirmed reverse DNS
Google, Apple, and Bing document a two-way DNS check. The forward lookup is not
optional.
1. Run a PTR lookup on the source IP.
2. Check the returned hostname against the operator's exact documented suffix.
3. Resolve that hostname to its A and AAAA addresses.
4. Require the original source IP to appear in the forward result.
Check the hostname on a label boundary. `crawl.example.googlebot.com` may match
an allowed suffix; `googlebot.com.attacker.example` must not. Normalize a final
DNS dot before comparison, and do not accept a substring match.
The forward lookup closes a simple hole. An attacker may control the PTR record
for an IP and give it a convincing name. They should not pass unless the named
operator's hostname resolves back to the source address as its documentation
requires.
DNS checks need bounds too. Set short timeouts, cap CNAME traversal, cache
successful results according to DNS TTLs, and distinguish a mismatch from a
resolver failure. A mismatch is `Failed`. A timeout or temporary DNS error is
`Unknown`.
## Signed requests are useful, but support is not universal
[RFC 9421](https://www.rfc-editor.org/rfc/rfc9421.html) defines HTTP Message
Signatures. A client can sign selected request components, and a verifier can
check the signature with trusted key material. The standard supplies the
mechanism. An application still has to decide which components must be signed,
where keys come from, which algorithms are accepted, and how replay is handled.
Web Bot Auth applies that mechanism to automated traffic. The protocol is still
an [IETF working-group draft](https://datatracker.ietf.org/doc/draft-ietf-webbotauth-httpsig-protocol/),
not a finished RFC. Cloudflare has a deployed implementation with a
`Signature-Agent` key directory and documents its current requirements and
limitations in the [Web Bot Auth guide](https://developers.cloudflare.com/bots/reference/bot-verification/web-bot-auth/).
A valid signature proves that a holder of the accepted key signed the covered
parts of the request. It does not prove that the operator's behavior is safe or
that every unsigned part of the request can be trusted. The verifier also needs
a rule that connects the key to an identity it recognizes.
Treat key discovery as untrusted input. Do not fetch an arbitrary internal or
private address because a client placed it in `Signature-Agent`. Bound response
size, redirects, fetch time, and key count. Require HTTPS and an explicit trust
policy. These checks matter just as much as the signature math.
## Fingerprints can challenge a claim, not authenticate an operator
TLS, HTTP/2, browser-environment, and behavioral signals help detect
contradictions. A client may claim to be a browser while its connection looks
like a common HTTP library. It may claim to be a scheduled crawler while its
requests behave like an interactive browser session.
Those observations are useful, but they answer a different question. A TLS
fingerprint does not prove that OpenAI, Anthropic, Google, or another named
operator sent a request unless the operator has made that fingerprint part of
its published verification method. Fingerprints change. They can also be
imitated.
Use these signals to classify automation, find spoofing indicators, and decide
how much scrutiny an unknown client needs. Do not silently upgrade
`Unidentified` to `Verified` because its behavior looks tidy.
For the connection-layer mechanics, see [TLS fingerprinting explained](https://www.centinelanalytica.com/blog/tls-fingerprinting-explained). For browser-controlled traffic,
see [How to detect browser automation](https://www.centinelanalytica.com/blog/detect-browser-automation).
## Identity and behavior are separate decisions
Verification should feed a policy decision, not replace one.
| Identity result | Policy context | Sensible starting response |
| --- | --- | --- |
| Verified | Client and requested resource are approved | Allow or observe; apply a documented rate limit if needed. |
| Verified | Identity is known, but the route or purpose is not approved | Enforce the resource policy. A real identity does not override it. |
| Failed | A trusted name was claimed without matching evidence | Do not apply the trusted allow rule. Block or challenge according to the route's risk. |
| Unknown | Proof is unavailable or the check had a temporary failure | Use the preselected fallback for that resource. Observe, limit, or challenge rather than guessing. |
| Unidentified | No operator identity can be established | Apply the site's normal automated-traffic policy. |
The fallback should vary by resource. A public article, a subscription archive,
a login endpoint, and an internal API do not need the same failure behavior.
Choose fail-open or fail-closed behavior before the verifier has an outage.
The full policy model is in [How to control AI agent access](https://www.centinelanalytica.com/blog/how-to-control-ai-agent-access). For the enforcement context these
identities feed, see the [scraping and AI crawler use case](https://www.centinelanalytica.com/use-cases/scraping-and-ai-crawlers).
## A practical verification runbook
Start with a small set of clients that already matter to your site.
1. Inventory the claimed user agents in edge or origin logs.
2. Create one catalog record per client and purpose. Store the official source,
verification method, refresh schedule, owner, and last successful update.
3. Verify the source address at the trusted network boundary.
4. Return one of the four explicit identity results with a reason code.
5. Keep identity policy separate from route and behavior policy.
6. Test a genuine request, a copied user agent, a DNS forward mismatch, a stale
range file, and a verifier timeout.
7. Review operator documentation on a schedule and when traffic changes.
Also test the boring failure paths. What happens when DNS is slow? What happens
when an operator publishes malformed JSON? Does a failed refresh erase the last
known-good ranges? Can a request make the verifier fetch an internal URL? These
are the cases that decide whether the verifier remains useful under pressure.
Before building any of this, you can run one address through the
[crawler verification tool](https://www.centinelanalytica.com/verify-crawler). It forward-confirms reverse DNS
where the operator documents that method, and falls back to the address list
where the operator publishes one instead.
## Where Centinel fits
Centinel is built to help teams identify and control automated traffic.
During an evaluation, ask which operator records are maintained, which checks
run in your [supported integration](https://www.centinelanalytica.com/integrations), where the result is enforced,
and what happens when verification evidence is unavailable. Test those answers
against your own traffic rather than relying on a crawler-count headline.
The broader [bot management solution checklist](https://www.centinelanalytica.com/bot-management) shows how to
test crawler verification alongside browser automation, residential proxy
traffic, reader impact, and rollback.
If you want to compare the identities declared in your logs with the automated
traffic actually reaching your pages, [request a site audit](https://www.centinelanalytica.com/free-audit).
## Sources
- [Verify requests from Google crawlers and fetchers](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests)
- [OpenAI crawler documentation](https://developers.openai.com/api/docs/bots)
- [Anthropic crawler documentation](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)
- [Applebot documentation](https://support.apple.com/en-us/119829)
- [Verify Bingbot](https://www.bing.com/webmasters/help/how-to-verify-bingbot-3905dc26)
- [Perplexity crawler documentation](https://docs.perplexity.ai/docs/resources/perplexity-crawlers)
- [RFC 9421: HTTP Message Signatures](https://www.rfc-editor.org/rfc/rfc9421.html)
- [IETF Web Bot Auth protocol draft](https://datatracker.ietf.org/doc/draft-ietf-webbotauth-httpsig-protocol/)
- [Cloudflare Web Bot Auth implementation](https://developers.cloudflare.com/bots/reference/bot-verification/web-bot-auth/)
---
Page: https://www.centinelanalytica.com/blog/interstitial-page-content-protection
# Interstitial challenge pages for protection
Decide when an interstitial challenge fits a protected route, which failure modes to handle, and how to measure the result before broad rollout.
By Frederick Jahn. Published 2026-04-01, updated 2026-10-11.
An interstitial challenge places a verification step between a request and the requested resource. It can add useful evidence or impose a cost on automation, but it also adds latency and a new failure point for legitimate visitors.
The right question is not whether every site needs a challenge. It is whether a particular challenge improves the outcome for a particular route at an acceptable user cost.
## What an interstitial challenge does
A full-page interstitial temporarily replaces the requested page. The client must complete a browser check, interactive task, or other proof before the protected response is served. A successful result is usually represented by a short-lived token or cookie that reduces repeated challenges.
[Cloudflare's challenge documentation](https://developers.cloudflare.com/cloudflare-challenges/challenge-types/challenge-pages/) is one vendor example of this pattern: a full-page screen verifies the visitor before access to the site continues. Implementations differ, so confirm token lifetime, supported content types, accessibility behavior, and failure handling for the system you evaluate.
Passing a challenge confirms only that the client completed that challenge under those conditions. It cannot prove the requester is human. Browser automation can execute JavaScript, retain cookies, and sometimes complete interactive flows. A challenge is one response in a larger policy, not an identity oracle.
## When an interstitial is appropriate
An interstitial is worth evaluating when all of these are true:
- the route has a defined automated-abuse problem;
- serving the resource has meaningful cost or consequence;
- weaker responses such as observation or rate limiting are insufficient;
- the client is expected to support the challenge mechanism;
- a recovery path exists for legitimate users who cannot pass.
Common candidates include high-cost search, repeated account actions, scarce inventory operations, and routes returning data that is valuable in bulk. The choice is not industry-specific. It depends on the resource, the abusive workflow, and the integration point.
Use a decision table before writing a rule:
| Request state | Resource sensitivity | Initial response |
| --- | --- | --- |
| Low risk, expected behavior | Low | Allow and observe |
| Uncertain evidence | Low | Observe or apply a bounded rate limit |
| Uncertain evidence | High | Consider a non-interactive challenge |
| Corroborated automation | High | Challenge, rate-limit, or deny according to policy |
| Verified trusted automation | Any | Apply its explicit resource policy |
The response should follow from evidence and resource sensitivity. A new visitor, unfamiliar network, or privacy setting is not abuse by itself.
## Choose the least disruptive challenge
Challenge mechanisms impose different costs:
- **Non-interactive browser checks** collect or validate browser evidence without asking the user to solve a task. They are less visible but still depend on compatible script execution, storage, and network behavior.
- **Interactive CAPTCHA-style checks** require an action. They can provide another signal for sensitive flows, but they demand accessible alternatives and can interrupt conversion.
- **Proof-of-work** asks the client to perform computation. It may change automation economics, but it also consumes user CPU, battery, time, and data.
- **Step-up authentication** asks an already known user for stronger account proof. It can fit account actions better than a generic browser puzzle.
Begin with the lowest-friction response that meets the route's goal. Escalate only when the added evidence or cost changes the decision.
The [OWASP Automated Threats project](https://owasp.org/www-project-automated-threats-to-web-applications/) provides a useful vocabulary for defining the automated action first. Naming the threat prevents “bot” from becoming a catch-all justification for friction.
## Failure modes to design for
### Capable automation passes
An automated browser may run the challenge correctly. Treat a pass as scoped evidence with an expiry, not permanent trust. Keep request rate, session sequence, identity evidence, and application behavior in the decision.
### Legitimate clients cannot run it
Script blockers, strict privacy settings, older browsers, embedded webviews, assistive technology, and network filtering can interrupt a challenge. Provide a bounded retry and a different recovery path. Do not create an endless refresh or challenge loop.
### The interaction excludes users
The W3C's [WCAG guidance on CAPTCHA](https://www.w3.org/WAI/WCAG22/Understanding/captcha.html) requires alternatives that use different output modes for users with disabilities. Accessibility review must cover keyboard operation, focus order, screen-reader announcements, motion, contrast, time limits, and the recovery path.
### Crawlers receive the wrong response
A search crawler that receives only a challenge page may fail to access the content you intended to index. Keep verified crawler policy separate from unverified browser traffic, and test the rendered and HTTP responses with the search tools relevant to the site.
Google also warns that intrusive interstitials can make page content harder to access. Its [search guidance on interstitials](https://developers.google.com/search/docs/appearance/avoid-intrusive-interstitials) should be part of QA for indexable routes.
### State is lost or replayed
Blocked cookies, multiple subdomains, CDN cache keys, and clock differences can cause repeated challenges. Conversely, an overly broad or long-lived pass token may be shared or replayed. Bind the token only to evidence the deployment can validate, keep its lifetime bounded, and define revocation behavior.
### The challenge service fails
Decide whether each route fails open, fails closed, or falls back to another control. The right answer can differ for a public article, an account recovery action, and an expensive API. Bound upstream calls, retries, token size, and verification time.
## Place the challenge on the right traffic path
A challenge can protect only requests that pass through its decision and verification points. Map the complete path:
```
request
-> route policy
-> risk decision
-> allow response
-> challenge response
-> verify result
-> issue bounded pass token
-> retry original resource
```
Check alternate hosts, direct origin access, APIs, cached copies, prefetches, feeds, and asset URLs. If valuable content is available through a path that never evaluates the token, the interstitial changes the main page without protecting the resource.
Browser-side checks also do not govern clients that never load the page. Server or edge enforcement must cover the request that actually returns the protected data.
## Roll out with a holdout
Start in observation mode. Record which sessions would have been challenged and why. Review ordinary browser coverage, assistive flows, search crawler handling, and internal monitoring before serving the interstitial.
Then use a limited rollout with an unchanged holdout where practical. Compare:
- protected-action completion;
- challenge presentation, pass, failure, and abandonment;
- repeat challenges and challenge loops;
- origin work and automated-abuse outcomes;
- performance by device and network class;
- support and accessibility reports;
- false positives found through review or appeal.
Define rollback thresholds before exposure increases. Keep the challenge version in telemetry so a regression can be tied to a concrete change.
Re-test when the browser, challenge code, policy model, or protected flow changes. A launch result is not a permanent benchmark.
## Where Centinel fits
Centinel can use browser, network, session, and crawler evidence to decide when a supported route should allow, observe, challenge, rate-limit, or deny a request. The exact options depend on the integration and protected traffic path.
For the broader problem, see [scraping and AI crawler protection](https://www.centinelanalytica.com/use-cases/scraping-and-ai-crawlers). For evaluation questions about coverage, failure behavior, and rollout, see the [bot management overview](https://www.centinelanalytica.com/bot-management).
An interstitial is useful when it makes a measured route-level decision better. It should not become a universal toll that every user pays because the underlying policy was never defined.
---
Page: https://www.centinelanalytica.com/blog/detect-browser-automation
# Automated browser detection: evidence and actions
Detect Playwright, Selenium and other automated browsers with request, browser and session evidence, then choose when to observe, challenge or block.
By Frederick Jahn. Published 2026-03-28, updated 2026-10-13.
Detect browser automation by combining independent request, browser, and
session evidence, then checking it against the action you protect.
`navigator.webdriver === true` reports WebDriver control; `false` does not
prove a human visitor. Neither result establishes abuse.
Start with the decision your team needs to make: stop repeated extraction,
protect an interactive workflow, or preserve approved monitoring. Choose one
route and record both suspicious activity and legitimate completions before
enabling enforcement. Blocking more sessions is not success if the intended
users can no longer finish that workflow.
For a vendor evaluation, use the [bot management checklist](https://www.centinelanalytica.com/bot-management)
to compare evidence availability, false-positive handling, and rollback. For
content extraction, the [scraping and AI crawler use case](https://www.centinelanalytica.com/use-cases/scraping-and-ai-crawlers) connects these checks to resource policy.
## Start with `navigator.webdriver`
`navigator.webdriver` is the web platform's standard automation disclosure.
The [W3C WebDriver specification](https://w3c.github.io/webdriver/#interface)
says the property returns `true` when the browser's WebDriver-active flag is
set. The spec describes it as a way for a cooperating browser to tell a page
that WebDriver controls it.
You can inspect it with one expression:
```
const webdriverDeclared = navigator.webdriver === true;
```
Interpret the result narrowly:
- `true` means this browser reports that it is under WebDriver control;
- `false` does not prove that a human controls the browser;
- either result says nothing about whether the activity is approved or abusive.
A positive value can come from your own regression tests, uptime monitoring,
or another legitimate automated workflow. Automation can also use a different
control path or alter browser-visible state. Keep the raw value as one named
observation. Do not turn it into a universal block rule.
## Build a layered evidence model
Use layers that fail differently. Each layer should record an observation, its
source, and its confidence rather than writing a final verdict into the field.
| Signal layer | What it observes | Common evasion or drift | False-positive risk | Pair it with |
| --- | --- | --- | --- | --- |
| Declared identity | User agent, Client Hints, language, referrer, crawler token | Headers can be copied, removed, or made internally inconsistent | Privacy tools, embedded browsers, unusual clients | Connection, environment, published crawler verification |
| Connection and protocol | Source path, TLS handshake, negotiated protocol, HTTP/2 behavior | Proxies and impersonation libraries change what the server sees | CDN changes, browser updates, shared networks | Declared identity, session continuity, traffic history |
| Browser environment | Exposed APIs, viewport, platform, locale, time zone, feature support | Profiles and injected scripts can modify values | Enterprise policy, extensions, accessibility tools, privacy protections | Declared identity, control artifacts, repeated observations |
| Automation and control artifacts | WebDriver disclosure, injected globals, execution contexts, protocol side effects | Framework and browser changes remove or reshape artifacts | Developer tools, QA harnesses, monitoring, extensions | Environment, session behavior, approved-tool inventory |
| Session behavior | Navigation order, resource graph, interaction sequence, dwell and input timing | Automation can wait, replay, or drive browser input APIs | Power users, assistive technology, prefetching, poor networks | Route risk, first-party session, traffic history |
| Traffic history | Repetition across sessions, accounts, networks, routes, and time windows | Distributed clients rotate identifiers and pace work | NAT, shared offices, viral traffic, scheduled jobs | Strong first-party identity, per-route budgets, current session evidence |
Collect only the fields needed to answer a specific abuse question. A public
page, a login form, and an account export need different evidence and response
thresholds.
### Declared identity
Headers and browser properties are still useful. They establish what the
client claims to be and give later layers something concrete to check. Store
the claim separately from any inferred identity.
For a named crawler, compare the exact token with the operator's current
verification method. Do not treat a plausible browser user agent as verified
browser identity. The same rule applies to language, platform, and device
claims: an inconsistency can add evidence, but an unusual combination is not a
verdict.
### Connection and protocol
Connection metadata is observed outside page JavaScript. Useful fields can
include the TLS fingerprint, negotiated application protocol, source path, and
HTTP version. Collection must happen at the boundary that sees the visitor's
connection. A CDN that terminates TLS may expose different metadata to the
origin.
HTTP/2 starts with a connection preface followed by a `SETTINGS` frame from
each endpoint. [RFC 9113](https://www.rfc-editor.org/rfc/rfc9113.html#section-3.4)
defines the frame and its parameters. Implementations can choose different
valid settings, and those choices can change. Record the raw observation and
client version before labeling a value as browser-like or library-like. Avoid
copying exact browser values from an undated fingerprint list into a permanent
rule.
TLS has the same boundary. A JA3 or JA4 value can help expose a disagreement
with the declared client, but it can be shared, changed, or imitated. [TLS fingerprinting explained](https://www.centinelanalytica.com/blog/tls-fingerprinting-explained) covers the
collection and normalization details.
### Browser environment
Page JavaScript can inspect a browser's reported environment. Start with
coarse, explainable fields that support an actual consistency check. This
minimal sample can be run on a staging page or in the browser console:
```
const sample = {
schemaVersion: 1,
observedAt: new Date().toISOString(),
userAgent: navigator.userAgent,
webdriver: navigator.webdriver,
languages: Array.from(navigator.languages).slice(0, 8),
platform: navigator.userAgentData?.platform ?? navigator.platform,
hardwareConcurrency: navigator.hardwareConcurrency ?? null,
timezone: Intl.DateTimeFormat().resolvedOptions().timeZone,
viewport: { width: innerWidth, height: innerHeight },
coarsePointer: matchMedia("(pointer: coarse)").matches,
};
console.log(JSON.stringify(sample, null, 2));
```
The output is a lab record, not a score. A narrow viewport on a desktop user
agent can be a resized window. A surprising time zone can be travel or a
privacy setting. Missing features can come from browser policy. Compare fields
with one another and with later sessions before assigning risk.
Canvas, audio, font, and graphics probes expand the fingerprinting surface and
the privacy burden. Do not add them merely because they are available. Document
the purpose, retention, access, and deletion policy for every field you keep.
### Automation and control artifacts
Browser-control protocols can create observable state, but the details are
version-specific. The [Chrome DevTools Protocol](https://chromedevtools.github.io/devtools-protocol/) lets tools
instrument and inspect Chromium through commands and events. Its documentation
also warns that the tip-of-tree protocol changes frequently and does not
guarantee backward compatibility.
The CDP Runtime domain supports evaluation and remote objects. [Runtime domain documentation](https://chromedevtools.github.io/devtools-protocol/tot/Runtime/)
Frameworks expose higher-level operations on top of browser control channels.
For example, [Playwright's evaluation documentation](https://playwright.dev/docs/evaluating) separates the test
process from the page's JavaScript environment and shows how `page.evaluate()`
runs code in the page.
Treat any artifact tied to serialization, an injected property, an execution
context, or a framework naming convention as a dated detector. Give it a
browser version, tool version, fixture, expected result, and expiry date. When
the fixture stops reproducing the artifact, retire the detector instead of
assuming the automation disappeared.
Some tools exist to remove these artifacts. [Playwright Stealth](https://www.centinelanalytica.com/blog/what-is-playwright-stealth) patches values that page JavaScript
can read, while [Camoufox](https://www.centinelanalytica.com/blog/what-is-camoufox) changes them inside a
modified Firefox build. Keep both kinds in your fixtures.
### Session behavior
Behavior is a sequence, not a claim about how humans move. Useful observations
include which resources were requested, whether navigation follows reachable
links, the order of actions, repeated failures, and the time between meaningful
steps.
Avoid rules such as “straight mouse movement means bot” or “fast typing means
bot.” Browsers expose input APIs to automation, and frameworks can control
timing. Selenium's [Actions API](https://www.selenium.dev/documentation/webdriver/actions_api/) provides
virtualized pointer, keyboard, and wheel input with configurable pauses.
[Playwright's input documentation](https://playwright.dev/docs/input) includes
clicks, sequential key presses, delays, and programmatic event dispatch.
Legitimate readers also use keyboards, assistive technology, password
managers, translation tools, and unusual navigation patterns. Prefer signals
tied to the protected workflow: impossible state transitions, repeated
high-cost actions, missing prerequisites, or a pace that violates an explicit
route budget.
### Traffic history
History can show patterns that a single page view cannot: repeated account
creation, synchronized retrieval, identifier churn, or the same action spread
across many sessions. It also creates the greatest risk of linking unrelated
people.
Set a bounded window and a documented grouping key. Distinguish account,
first-party session, source network, device evidence, and inferred cluster.
Keep an “insufficient history” state. A shared IP address is not a shared
actor, and a new visitor is not suspicious because the system has never seen
them.
## Run a controlled browser-automation lab
Test every detector against browsers and tools you approve for the lab. The
first run should use default configurations; add one variation at a time only
after the baseline is recorded.
Use this fixture:
1. Create a staging page with one article link, one text field, and one submit
button. Do not use production accounts or reader data.
2. Log the HTTP request, available connection metadata, the environment sample
above, and four server-recognized actions: open, follow link, enter text,
submit.
3. Run 10 clean sessions with a manually operated browser, 10 with current
Playwright, and 10 with current Selenium. Use the same browser engine,
operating system, network, viewport, and staging route.
4. Use a fresh browser profile or context for each session. Randomize the order
of the three client groups so time-dependent infrastructure changes do not
affect only one group.
5. Record observations without blocking. Repeat on another operating system
before drawing a rule from an environment field.
The manual run is a comparison case, not “ground truth” for every legitimate
reader. Playwright uses browser contexts as isolated profiles in its test
model, as documented in its [test guide](https://playwright.dev/docs/writing-tests).
Selenium WebDriver is a remote-control interface for browser sessions, as
described in the [WebDriver documentation](https://www.selenium.dev/documentation/webdriver/).
Keep one row per group and configuration:
| Field | Manual browser | Playwright default | Selenium default |
| --- | --- | --- | --- |
| Test date and owner | Record | Record | Record |
| Browser, tool, OS versions | Record browser and OS | Record browser, Playwright, and OS | Record browser, driver, Selenium, and OS |
| Configuration and launch arguments | Record | Record | Record |
| Sessions completed / attempted | `/10` | `/10` | `/10` |
| `navigator.webdriver` results | Record counts by value | Record counts by value | Record counts by value |
| Connection and environment differences | Link raw observations | Link raw observations | Link raw observations |
| Four-action sequence completed | Record counts | Record counts | Record counts |
| Candidate signal and exceptions | Record | Record | Record |
| Limitation and retest date | Record | Record | Record |
Do not publish a percentage from 10 sessions as an accuracy rate. The lab finds
reproducible differences and broken assumptions. Production validation needs a
larger, representative sample with reviewed labels and a measured cost for
false positives and false negatives.
## Turn observations into decisions
Keep classification separate from response. A useful decision record contains:
- the protected route and action;
- observations from each available layer;
- the confidence and reason for the classification;
- the selected response;
- the rule and model version;
- the fallback when evidence is missing;
- the expiry or review date.
Start new signals in observation mode. Compare them with known QA traffic,
approved monitoring, reader sessions, and confirmed abuse. A detector that
only works against the default configuration in the lab should remain a weak
signal.
Choose the response by risk:
| Situation | Possible response | Guardrail |
| --- | --- | --- |
| Low-risk public article, weak or conflicting evidence | Allow and observe | Do not interrupt the reader for one anomaly |
| Interactive route, several independent concerns | Step up with a bounded challenge or re-authentication | Prevent loops and provide an accessible fallback |
| Expensive action exceeds an explicit account or session budget | Rate-limit that action | Use a strong key and return a documented retry path |
| Restricted action lacks authorization | Deny in the application | Automation classification never replaces authorization |
| High-confidence abusive sequence with reviewed evidence | Block the scoped action or session | Log the reason and keep a fast rollback |
Monitor challenge completion, reader errors, action success, response codes,
and rule reasons after rollout. Break results down by route and client cohort.
If a browser update changes a signal, move the affected rule back to
observation until the lab is repeated.
For non-browser crawlers, choose controls that fit the client instead of
forcing a browser detector onto it. [How to block AI crawlers](https://www.centinelanalytica.com/blog/how-to-block-ai-crawlers) covers `robots.txt`, verified-client
rules, rate limits, authorization, and rollout safety.
If this lab is part of a vendor evaluation, run the same traffic sample
through the checklist above so identity, reader impact, integration, and
rollback are tested together.
Before evaluating Centinel on this workflow, review the [integration options](https://www.centinelanalytica.com/integrations). Ask which observations are available, where the
decision runs, how missing evidence is handled, and how a rule is rolled back.
Do not assume a browser probe protects a direct API request or that every
connection fingerprint describes the visitor's connection to your origin.
If your team needs to establish which automated traffic reaches the affected
pages before choosing controls, [request a site audit](https://www.centinelanalytica.com/free-audit).
## What gets past your existing CDN?
You have a detection checklist. Now test your current protection. Request a free site audit: we run real scraping tools against your site, review the results, and email you the report. Your existing CDN stays in place.
[Request a free site audit](https://www.centinelanalytica.com/free-audit)
## Sources
- [W3C WebDriver interface and navigator.webdriver](https://w3c.github.io/webdriver/#interface)
- [Chrome DevTools Protocol](https://chromedevtools.github.io/devtools-protocol/)
- [Chrome DevTools Protocol Runtime domain](https://chromedevtools.github.io/devtools-protocol/tot/Runtime/)
- [RFC 9113: HTTP/2](https://www.rfc-editor.org/rfc/rfc9113.html)
- [Playwright: Evaluating JavaScript](https://playwright.dev/docs/evaluating)
- [Playwright: Actions](https://playwright.dev/docs/input)
- [Playwright: Writing tests](https://playwright.dev/docs/writing-tests)
- [Selenium: Actions API](https://www.selenium.dev/documentation/webdriver/actions_api/)
- [Selenium: WebDriver](https://www.selenium.dev/documentation/webdriver/)
---
Page: https://www.centinelanalytica.com/blog/patched-chromium-browsers
# Patched Chromium browsers explained
Learn what patched Chromium means, which automation signals a patch can change, and how to test browser-automation detection without relying on one property.
By Frederick Jahn. Published 2026-03-25, updated 2026-10-11.
A patched Chromium setup changes some part of the browser automation stack to make automated sessions look more like ordinary browser sessions. The changed part may be the Chromium binary, a driver, an automation framework, a launch configuration, or code injected into the page.
That distinction matters. A product described as “patched Chromium” is not necessarily a custom browser build, and two tools using the label may leave different evidence.
## What a patched browser changes
Browser automation has several layers:
| Layer | Examples | What can change |
| --- | --- | --- |
| Browser binary | A maintained Chromium fork | Browser-exposed properties, command-line behavior, or protocol handling |
| Driver | ChromeDriver tooling | Driver artifacts and browser launch configuration |
| Automation framework | Playwright or Puppeteer modifications | Which protocol domains and execution contexts the framework uses |
| Page script | Init scripts or stealth plugins | JavaScript-visible values after a document starts |
| Operating environment | Container, display, fonts, network | The context around the browser rather than the browser code itself |
“Patched” should therefore be treated as a description to investigate, not a fingerprint. Ask which layer changed, which upstream version it tracks, and which observations the change is meant to affect.
## Start with the standards
The [WebDriver specification](https://www.w3.org/TR/webdriver2/) defines a webdriver-active flag that is true when the user agent is under remote control. `navigator.webdriver` exposes the state of that flag.
That makes the property useful evidence in its defined context. It does not make `false` proof of a human user. Automation that does not create a standards-conforming WebDriver session may not set the flag, and modified tooling may try to alter what the page observes.
The same restraint applies to headless mode. [Chrome's headless documentation](https://developer.chrome.com/docs/automation-and-testing/headless) says modern headless Chrome uses the regular Chrome code and creates platform windows without displaying them. Headless is an execution mode, not an automation verdict. Legitimate testing and rendering systems use it, while automation can also run with a visible interface.
## Understand the protocol surface
Many Chromium automation tools communicate through the Chrome DevTools Protocol (CDP). The protocol exposes domains for inspecting and controlling the browser. For example, [Runtime.evaluate](https://chromedevtools.github.io/devtools-protocol/tot/Runtime/#method-evaluate) evaluates an expression on the global object, while the Page domain can create an isolated world for a frame.
The fact that CDP has a documented method does not mean a page can reliably infer every use of that method. An observation may depend on a particular Chromium version, framework sequence, execution context, or serialization path. A browser update can change the observation even when the protocol method still exists.
This is why a technique should be recorded with its prerequisites:
- browser and operating-system version;
- automation tool and version;
- launch arguments and extensions;
- page lifecycle stage;
- expected observation and known counterexamples.
Without those details, “detects CDP” is not a reproducible result.
## Read tool claims at the right confidence level
Projects describe their own goals and techniques. Those descriptions are useful for forming tests, but they are not independent proof of bypass or detection performance.
For example, the [rebrowser-patches repository](https://github.com/rebrowser/rebrowser-patches) describes changes intended to avoid specific automation leaks associated with the CDP Runtime domain. That is a project-maintainer claim. Treat it as a hypothesis until you reproduce the behavior against named browser and framework versions.
The same rule applies to driver patchers, Playwright forks, hosted browsers, and custom Chromium builds:
1. Record what the maintainer says is changed.
2. Pin the artifact and upstream browser versions.
3. Reproduce the claimed observation in an isolated lab.
4. Test ordinary browsers using the same environment.
5. Re-run after either side updates.
Do not turn a passing demo into a universal statement about all sessions from that tool.
## Why static checks age quickly
A static check asks whether one property or object has an expected value. These checks are cheap and can still contribute evidence, but they have narrow scope.
A changed browser can normalize a property. An ordinary browser can also produce an unusual value because of enterprise policy, privacy software, accessibility tooling, an extension, hardware, or an unsupported platform. Treating either case as conclusive raises false positive and false negative risk.
Version drift adds another problem. Chromium, drivers, and automation frameworks ship independently. A check observed on one combination may disappear, invert, or become common in legitimate traffic on another.
No single browser property establishes that a session is automated. The useful question is whether several independent observations support the same bounded conclusion.
## Build a layered test
Start with a matrix, not a single proof-of-concept.
| Cohort | Purpose |
| --- | --- |
| Ordinary supported browsers | Measure the normal range and false positive rate |
| Supported browsers with common privacy or accessibility changes | Find legitimate edge cases |
| Stock automation with pinned versions | Confirm the test can observe known automation |
| Patched or configured automation with pinned versions | Measure which observations changed |
| New browser or framework release | Detect drift before rollout |
For each cohort, collect evidence from layers that fail differently:
- **Runtime consistency:** related browser values, APIs, permissions, and lifecycle observations.
- **Protocol and network consistency:** whether the browser claim agrees with transport and request behavior visible at your integration point.
- **Interaction sequence:** navigation, timing, input, and retry patterns across the session.
- **Application context:** whether the sequence resembles the abusive workflow you are protecting, such as repeated login attempts or high-cost search requests.
Then remove one evidence family at a time and measure the effect. This shows whether the model is genuinely layered or merely repeats the same assumption under several feature names.
The [browser automation detection guide](https://www.centinelanalytica.com/blog/detect-browser-automation) contains a fuller evidence model and rollout process.
## Set a bounded response
Detection and response are separate decisions. Weak or conflicting evidence may justify observation or a rate limit, not an immediate deny. Stronger corroborated evidence on a sensitive route may justify a challenge or block.
Define these before rollout:
- the protected resource and abuse case;
- the evidence required for each response;
- how legitimate automation is identified;
- what happens when collection fails;
- the maximum delay and data retained;
- an appeal or recovery path for affected users.
Roll out in observe mode first. Review false positives across supported browsers and important user flows. Use holdouts where practical so a drop in abuse is not confused with a drop in legitimate traffic.
## Where Centinel fits
Centinel combines browser, network, session, and crawler evidence for route-specific automation decisions. It is designed to avoid making one JavaScript property carry the whole verdict.
Coverage still depends on deployment. Browser-side evidence is relevant only when the client runs it. Network and server-side evidence must be collected on the traffic path being evaluated. A buyer should test the actual integration against current ordinary and automated cohorts rather than rely on a generic detection claim.
See the [bot management overview](https://www.centinelanalytica.com/bot-management) for the supported decision model and evaluation checklist.
Patched browsers change evidence. They do not remove the need to define the abuse case, reproduce observations, measure error, and keep the test current.
---
Page: https://www.centinelanalytica.com/blog/tls-fingerprinting-explained
# TLS fingerprinting explained: JA3 vs JA4
A JA3 or JA4 fingerprint summarizes a TLS ClientHello. How each is built, how to read and capture one, and why a TLS fingerprint alone never proves a bot.
By Frederick Jahn. Published 2026-03-22, updated 2026-10-11.
A TLS fingerprint is a compact description of a client's TLS handshake. It can
help distinguish a standard browser connection from a script or reveal a
mismatch between a claimed browser and the network client making the request.
It does not identify a person, prove that a request is automated, or explain
what the client intends to do.
JA3 and JA4 are two ways to turn fields from a TLS `ClientHello` into a value
that can be compared across connections. JA3 preserves the order of several
lists. JA4 sorts cipher and extension IDs before hashing them, so extension
permutation does not produce a new hash for those parts.
## Start with the ClientHello
The `ClientHello` is the client's opening TLS handshake message. In TLS 1.3 it
contains a legacy version field, a random value, a session ID, offered cipher
suites, compression methods, and extensions. The extensions can describe
supported TLS versions, groups, signature algorithms, key shares, the server
name, and application protocols such as HTTP/2. The exact structure is defined
in [RFC 8446](https://www.rfc-editor.org/rfc/rfc8446.html#section-4.1.2).
An abbreviated observation might look like this:
```
transport: TLS over TCP
highest supported version: TLS 1.3
cipher suites: 1301, 1302, 1303, ...
extensions: server_name, supported_groups, alpn, key_share, ...
first ALPN value: h2
```
Those fields reflect choices made by the client's TLS stack and its
configuration. Different programs may produce different combinations. The
same program can also change its handshake after an update, on another
operating system, or when configured differently.
The collection point matters. The original `ClientHello` is available where
the visitor's TLS connection terminates. If a CDN terminates that connection
and opens a new one to your origin, the origin sees the CDN's handshake, not
the visitor's. Collect the fingerprint at the trusted TLS boundary and carry
it forward as connection metadata if the application needs it.
## From handshake to fingerprint
A practical pipeline has six steps:
1. Capture the `ClientHello` at the TLS termination point.
2. Parse only the bounded fields required by the chosen fingerprint format.
3. Remove or normalize values exactly as that format specifies.
4. Calculate the fingerprint and record the format and implementation version.
5. Attach it to the connection or request without treating it as authenticated
identity.
6. Compare it with other evidence available for the same request and session.
Do not invent a private variation and still call it JA3 or JA4. A lookup is
only portable when producers use the same field selection, GREASE handling,
sorting, and hashing rules.
## JA3 and JA4 compared
| Property | JA3 | JA4 |
| --- | --- | --- |
| Primary input | TLS version, offered ciphers, extensions, supported groups, and EC point formats | Transport and protocol metadata, counts, ALPN, ciphers, extensions, and signature algorithms |
| List order | Preserves cipher, extension, group, and point-format order | Sorts cipher IDs and extension IDs; signature algorithms remain in their presented order |
| GREASE | Excludes GREASE values | Excludes GREASE values |
| Output | MD5 of the comma-and-hyphen-delimited input | A readable prefix plus two 12-character truncated SHA-256 hashes |
| Useful property | Simple, widely recognized representation of the observed list order | More stable when a client permutes cipher or extension order |
| Main limitation | Harmless ordering changes can create a different value | Sorting removes ordering information; shared or imitated handshakes still look alike |
[Salesforce's JA3 specification](https://github.com/salesforce/ja3) defines the
five input fields and the ordered string that is hashed. Its implementation
also ignores GREASE values so reserved placeholders do not become new
fingerprints. Salesforce archived the repository on May 1, 2025, and no longer
maintains it. The specification still describes how existing JA3 values were
produced.
### Reading a JA4 string
[FoxIO's JA4 specification](https://github.com/FoxIO-LLC/ja4/blob/main/technical_details/JA4.md)
builds a value in three parts, `a_b_c`, such as
`t13d1516h2_8daaf6152771_b186095e22b6`:
| Part | Value | Meaning |
| --- | --- | --- |
| a, character 1 | `t` | TLS over TCP. `q` means QUIC and `d` means DTLS |
| a, characters 2 and 3 | `13` | TLS version, here TLS 1.3 |
| a, character 4 | `d` | SNI extension present. `i` means no SNI |
| a, characters 5 and 6 | `15` | Cipher suites offered, GREASE excluded |
| a, characters 7 and 8 | `16` | Extensions offered, GREASE excluded |
| a, last two characters | `h2` | First and last characters of the first ALPN value. `00` means no ALPN |
| b | `8daaf6152771` | First 12 characters of a SHA-256 hash of the sorted cipher list |
| c | `b186095e22b6` | First 12 characters of a SHA-256 hash of the sorted extensions, without SNI and ALPN, followed by the signature algorithms in their original order |
Because the parts are separate, you can compare one without the others.
[FoxIO's README](https://github.com/FoxIO-LLC/ja4) gives the example of
tracking a client that rotates ciphers by matching `JA4_ac`, which keeps parts
a and c and drops b. JA4 itself is BSD 3-Clause, like JA3, while most other
JA4+ methods use the FoxIO License.
Modern Chromium can permute TLS extensions in the `ClientHello` to reduce
protocol ossification. Chromium's own feature definition describes the
permutation behavior. [Chromium source](https://chromium.googlesource.com/chromium/src/+/ff3b34ba2dc2d948ac13565d7bc5aa85f4f5e2aa/net/base/features.h)
An order-sensitive JA3 can therefore change across connections even when the
underlying client has not. JA4's sorting makes its cipher and extension hashes
stable under that particular change. It does not make JA4 permanent or
unspoofable. Some scraping browsers, such as [Ulixee Hero](https://www.centinelanalytica.com/blog/what-is-ulixee-hero), document emulation of browser network and TLS
characteristics for this reason.
### What GREASE means here
GREASE values are reserved placeholders that clients can send at TLS extension
points. They are intended to expose implementations that fail when they see an
unknown value, keeping those extension points usable. They are not invalid
bytes and their presence is not proof of Chrome or any other browser. [RFC 8701](https://www.rfc-editor.org/rfc/rfc8701.html) defines the mechanism and
the reserved values.
Both JA3 and JA4 exclude GREASE values as part of their normalization. Keep the
raw, access-controlled observation separately if GREASE behavior matters to
your analysis; it will not be represented fully in the normalized hash.
## Reproduce a fingerprint locally
Wireshark exposes both `tls.handshake.ja3` and `tls.handshake.ja4`. JA4 is
available in Wireshark 4.2 and later, according to its [TLS display-filter reference](https://www.wireshark.org/docs/dfref/t/tls.html).
Use this small lab only on a device and traffic you are allowed to inspect:
1. Record `tshark --version`, `curl --version`, the browser version, operating
system, destination host, and test time.
2. Start a Wireshark capture on the interface carrying your traffic.
3. Make three new TLS-over-TCP connections to the same test host with system
curl. Then make three with a normal browser. Avoid a host for which the
browser already has a reusable connection.
4. Stop the capture and save it as `tls-lab.pcapng`.
5. Extract the fingerprint fields:
```
tshark -r tls-lab.pcapng \
-Y "tls.handshake.type == 1" \
-T fields \
-e tcp.stream \
-e tls.handshake.ja3 \
-e tls.handshake.ja4 \
-e tls.handshake.ja3_full
```
Put the six rows in a table with the client, version, run number, JA3, and JA4.
Compare repeated runs before comparing the two clients. You may see changing
JA3 values with a stable JA4 when extension order is permuted. You may instead
see stable values for both. The result depends on the exact client, build,
configuration, and route.
This is a protocol demonstration, not a detection benchmark. Six local
connections cannot establish false-positive rates, population frequency, or
whether a fingerprint belongs to a bot. HTTP/3 uses QUIC rather than a TCP
stream, so keep the first lab to TLS over TCP or adapt the capture and JA4
transport interpretation deliberately.
## Use lookup data as context, not identity
A fingerprint database can tell you that a value has previously appeared with
a program, browser family, malware sample, or internal client. The record needs
provenance: who observed it, with which implementation, on what date, and under
which configuration.
There are two different kinds of ambiguity:
- **Many-to-one:** different users or programs can produce the same normalized
fingerprint. A common browser fingerprint should be expected across many
unrelated people.
- **One-to-many:** one program can produce different fingerprints after an
update, across platforms, or under different TLS settings.
Hash collisions are also possible in principle. JA3 uses MD5 and JA4 truncates
SHA-256 outputs, but these values are labels, not cryptographic proof. In
operations, shared inputs and changing inputs are usually the more immediate
source of mistaken attribution.
Age every lookup entry. Keep an unknown state. A fingerprint that is absent
from the database is not automatically suspicious, and an old label should not
silently become a permanent block rule.
## What a TLS fingerprint cannot prove
A matching fingerprint cannot prove that the request came from the named
browser. Tools can deliberately imitate browser handshakes.
[curl-impersonate](https://github.com/lwthiker/curl-impersonate) changes curl's
TLS and HTTP/2 behavior to resemble supported browsers.
[uTLS](https://github.com/refraction-networking/utls) exposes low-level
`ClientHello` control and browser-like presets. Both projects also document
limits: configuration changes can alter a signature, and `ClientHello`
parroting does not reproduce every later behavior of a browser.
A TLS fingerprint also cannot answer:
- whether a human or automation controls the client;
- whether a request is benign, abusive, or authorized;
- whether two connections belong to the same actor;
- whether a claimed crawler is operated by the company named in its user agent;
- whether the client will behave like the imitated browser after the handshake.
For a named crawler, use the operator's published verification method rather
than substituting a TLS label. [How to verify AI agents](https://www.centinelanalytica.com/blog/how-to-verify-ai-agents) covers those methods.
## Combine TLS with independent evidence
TLS fingerprinting is most useful as a consistency check. It can raise or lower
confidence when compared with evidence from other layers.
| Observation | Useful comparison | Caution |
| --- | --- | --- |
| Browser user agent with an unfamiliar TLS fingerprint | HTTP protocol behavior, browser APIs, navigation sequence | A browser update or unusual platform may be legitimate |
| Familiar browser JA4 across many sessions | Session behavior, request graph, cookies, challenge result | Common browsers naturally share fingerprints |
| Claimed named crawler | Published IP or DNS verification, documented token, resource policy | User-agent and TLS labels are not operator identity |
| TLS fingerprint changes mid-session | Connection reuse, proxy path, network change, session continuity | Mobile networks and multi-edge routes can change connections |
| Browser-like TLS plus browser-like HTTP | Runtime and session evidence | Impersonation tools can reproduce more than the handshake |
The decision model should preserve which layer contributed each signal. A
single mismatch may justify observation or a step-up check. Blocking needs a
policy that accounts for the route, confidence, impact on legitimate readers,
and rollback.
The next layer is covered in [How to detect browser automation](https://www.centinelanalytica.com/blog/detect-browser-automation).
## Operational checklist
- Collect at the boundary that sees the visitor's original `ClientHello`.
- Version the parser and fingerprint format.
- Bound parsing work and retain an explicit unknown state.
- Store source, date, client version, and configuration with lookup labels.
- Re-test known fingerprints after browser, TLS library, CDN, or proxy changes.
- Keep observation separate from enforcement.
- Measure false positives on your own traffic before blocking.
- Log the rule version and reason so a decision can be audited and reversed.
Centinel helps teams combine connection, request, browser, and session
evidence when classifying automated traffic. Evaluate it with traffic from your
own site and verify which signals are available at your TLS termination point.
To establish a baseline before enforcing a rule, [request a site audit](https://www.centinelanalytica.com/free-audit).
## Sources
- [RFC 8446: The Transport Layer Security Protocol Version 1.3](https://www.rfc-editor.org/rfc/rfc8446.html)
- [RFC 8701: Applying GREASE to TLS Extensibility](https://www.rfc-editor.org/rfc/rfc8701.html)
- [Salesforce JA3 specification](https://github.com/salesforce/ja3)
- [FoxIO JA4 technical details](https://github.com/FoxIO-LLC/ja4/blob/main/technical_details/JA4.md)
- [Chromium TLS extension permutation feature](https://chromium.googlesource.com/chromium/src/+/ff3b34ba2dc2d948ac13565d7bc5aa85f4f5e2aa/net/base/features.h)
- [Wireshark TLS display-filter reference](https://www.wireshark.org/docs/dfref/t/tls.html)
- [curl-impersonate repository](https://github.com/lwthiker/curl-impersonate)
- [uTLS repository](https://github.com/refraction-networking/utls)
---
Page: https://www.centinelanalytica.com/blog/gptbot-vs-claudebot-vs-bytespider
# GPTBot vs ClaudeBot vs Bytespider compared
Compare GPTBot, ClaudeBot and Bytespider: each operator's stated purpose, robots.txt guidance, published IP ranges, and how to check a claim.
By Frederick Jahn. Published 2026-03-18, updated 2026-10-11.
## The short answer
GPTBot, ClaudeBot, and Bytespider are labels used for [AI crawler](https://www.centinelanalytica.com/blog/what-is-an-ai-crawler) traffic, but the evidence available for each label is not equal.
- OpenAI documents GPTBot's purpose, `robots.txt` token, and published IP ranges.
- Anthropic documents separate crawler tokens, including ClaudeBot, [Claude-SearchBot](https://www.centinelanalytica.com/bots/claude-searchbot), and [Claude-User](https://www.centinelanalytica.com/bots/claude-user), and links to published IP ranges.
- Cloudflare lists Bytespider as a ByteDance AI crawler. A ByteDance operator page was not available in this source pack, so treat the label as third-party documentation rather than an operator-verified identity.
A user-agent string is not an identity check. It is a claim made by the requester. Use the claim to select a verification method, then decide what that verified or unverified requester may access.
## Compare the published evidence
This table describes the documentation available on September 4, 2026. It does not claim that every request using one of these names is genuine.
| Claimed crawler | Published purpose | Published identity evidence | `robots.txt` documentation | Source confidence |
| --- | --- | --- | --- | --- |
| GPTBot | OpenAI says GPTBot crawls content that may be used to train its generative AI foundation models. | OpenAI publishes a current JSON list of GPTBot IP ranges. | OpenAI documents the `GPTBot` product token and `Disallow` rules. | Operator-primary |
| ClaudeBot | Anthropic describes ClaudeBot as a crawler used for model development. | Anthropic links to a JSON file containing IP addresses used by its crawlers. | Anthropic documents `ClaudeBot` separately from `Claude-SearchBot` and `Claude-User`. | Operator-primary |
| Bytespider | Cloudflare identifies Bytespider as an AI crawler operated by ByteDance. | No operator-primary identity source was found for this source pack. | No operator-primary guidance was found for this source pack. | Third-party catalogue; verify independently |
Sources: [OpenAI crawler documentation](https://developers.openai.com/api/docs/bots), [OpenAI's GPTBot IP list](https://openai.com/gptbot.json), [Anthropic crawler documentation](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler), and [Cloudflare's AI crawler catalogue](https://developers.cloudflare.com/ai-crawl-control/reference/bots/).
## Keep purpose, identity, and policy separate
These are different questions:
1. **Purpose:** What does the operator say the crawler does?
2. **Identity:** What evidence connects this request to that operator?
3. **Policy:** What may the requester access once you have evaluated the evidence?
Mixing them creates brittle rules. A documented purpose does not prove the current request is genuine. A genuine crawler identity does not automatically grant access to every route. An unverified request does not automatically require a site-wide block.
Centinel's [guide to verifying AI agents](https://www.centinelanalytica.com/blog/how-to-verify-ai-agents) covers the verification sequence in more detail.
## Verify a GPTBot claim
OpenAI publishes separate tokens for different products. GPTBot is documented for training-related crawling. [OAI-SearchBot](https://www.centinelanalytica.com/bots/oai-searchbot) supports search, and [ChatGPT-User](https://www.centinelanalytica.com/bots/chatgpt-user) represents some user-initiated requests. Do not collapse those purposes into one rule just because they share an operator.
For a request claiming to be GPTBot:
1. Record the claimed user-agent and source IP.
2. Fetch OpenAI's current GPTBot IP list through a controlled, cached process.
3. Check whether the source IP belongs to a published range.
4. Apply the policy for the requested resource, not a blanket operator decision.
Published IP ranges can change. Bound the refresh interval, retain the last known-good list, and define failure behavior before depending on a remote file in enforcement.
On the median site in our [crawler verification data](https://www.centinelanalytica.com/blog/fake-crawler-report), 84% of requests claiming to be GPTBot failed OpenAI's published check. See the [GPTBot user agent and IP ranges](https://www.centinelanalytica.com/bots/gptbot).
## Verify a ClaudeBot claim
Anthropic documents several crawler identities with different purposes. ClaudeBot is not interchangeable with Claude-SearchBot or Claude-User. A rule intended for model-development crawling should not silently govern search indexing or user-directed retrieval.
Anthropic links to an IP-address list for its crawlers. Use that operator-primary source as one part of verification, with the same operational safeguards used for any remote allowlist: a bounded refresh, validation of the downloaded data, and a deliberate response when the source is unavailable. To check one address from your logs by hand, [verify a ClaudeBot or GPTBot IP](https://www.centinelanalytica.com/verify-crawler) against the operators' published lists.
Anthropic also documents `robots.txt` directives for its crawler tokens. Those directives express a preference to a conforming crawler; they do not authenticate the request at your edge. See the [ClaudeBot user agent and IP ranges](https://www.centinelanalytica.com/bots/claudebot).
## Treat a Bytespider claim as unresolved until verified
Cloudflare's crawler catalogue associates Bytespider with ByteDance. That is useful classification evidence, but it is not operator-primary identity documentation.
Because a ByteDance operator page was not in the source pack, this article does not prescribe an IP-range check or claim a universal `robots.txt` behavior for Bytespider. Start with observation:
- record the user-agent claim and source network data;
- examine request paths, cadence, protocol characteristics, and session behavior;
- check for a current operator source before creating a trusted allow rule;
- choose a response based on resource sensitivity and your tolerance for unresolved identity.
This is deliberately less tidy than a one-line allow-or-block answer. The available evidence does not justify pretending the identity question is settled. The [Bytespider bot profile](https://www.centinelanalytica.com/bots/bytespider) lists what is documented about it.
## What `robots.txt` can and cannot do
The Robots Exclusion Protocol defines how a crawler can discover and interpret site-owner rules. [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309) standardizes that protocol. It is a coordination mechanism, not an authentication or authorization layer.
A `robots.txt` rule can state that a named product token should not crawl a path. It cannot prove that a request carrying that name belongs to the stated operator, and it cannot force a non-conforming client to comply.
Use `robots.txt` when you want to communicate crawler preferences. Use request-time controls when access must be enforced.
## Turn the evidence into a resource policy
| Evidence state | Low-sensitivity public resource | Sensitive or costly resource |
| --- | --- | --- |
| Operator claim verified | Apply the policy assigned to that crawler and purpose. | Require the explicit rule for that resource; do not inherit a broad allow. |
| Named crawler, identity unresolved | Observe, rate-limit, challenge, or deny according to risk. | Default to the bounded failure behavior your team approved. |
| Claim conflicts with published evidence | Treat the name as spoofable input and evaluate the remaining signals. | Do not grant trusted access from the user-agent claim. |
Write the decision before implementing the rule. The [AI-agent access policy guide](https://www.centinelanalytica.com/blog/how-to-control-ai-agent-access) provides a practical template.
## Where Centinel fits
Centinel helps collect browser, network, session, and crawler evidence so teams can distinguish a declared name from a better-supported identity. It can then contribute that evidence to route-specific allow, observe, challenge, rate-limit, or deny decisions.
The exact enforcement options depend on the integration point. A browser-side deployment does not control every non-browser request, and a server or edge integration should be reviewed against the traffic path it is expected to cover. See the [bot management overview](https://www.centinelanalytica.com/bot-management) for the product boundary and evaluation checklist.
The durable rule is simple: document the purpose, verify what you can, preserve uncertainty where you cannot, and apply policy per resource.
---
Page: https://www.centinelanalytica.com/blog/robots-txt-ai-bots-guide
# robots.txt for AI bots: tokens and examples
Copy-ready robots.txt rules for GPTBot, ClaudeBot, Google-Extended and other AI crawler tokens, how to test them, and why robots.txt cannot stop fakes.
By Frederick Jahn. Published 2026-03-14, updated 2026-10-11.
`robots.txt` is where a site publishes crawl preferences. It is useful for cooperating crawlers, including most [AI crawlers](https://www.centinelanalytica.com/blog/what-is-an-ai-crawler), but it is not an access-control file. A requester can ignore it or copy a trusted crawler's name.
This guide shows how to turn a policy into a small, testable file and how to detect the gap between a published preference and observed traffic. It covers the file only. To deny requests at your server or edge, use [how to block AI crawlers](https://www.centinelanalytica.com/blog/how-to-block-ai-crawlers).
## Start with the policy
Do not begin by copying a long list of crawler names. First decide what each kind of automated use may do with each kind of resource.
| Resource | Search indexing | Model-development crawling | AI search or grounding | User-directed retrieval |
| --- | --- | --- | --- | --- |
| Public documentation | Decide | Decide | Decide | Decide |
| Public product pages | Decide | Decide | Decide | Decide |
| Expensive search endpoints | Usually needs request-time controls | Usually needs request-time controls | Usually needs request-time controls | Usually needs request-time controls |
| Account or private routes | Enforce outside `robots.txt` | Enforce outside `robots.txt` | Enforce outside `robots.txt` | Enforce outside `robots.txt` |
The labels are intentionally separate. OpenAI, for example, documents [GPTBot](https://www.centinelanalytica.com/bots/gptbot), [OAI-SearchBot](https://www.centinelanalytica.com/bots/oai-searchbot), and [ChatGPT-User](https://www.centinelanalytica.com/bots/chatgpt-user) for different purposes. Anthropic documents [ClaudeBot](https://www.centinelanalytica.com/bots/claudebot), [Claude-SearchBot](https://www.centinelanalytica.com/bots/claude-searchbot), and [Claude-User](https://www.centinelanalytica.com/bots/claude-user) separately. Your rules should reflect the use you intend to permit, not just the company behind a token.
Use the [AI-agent access policy guide](https://www.centinelanalytica.com/blog/how-to-control-ai-agent-access) to define the decision before editing the file.
## What the protocol does
The [Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309) defines a file named `/robots.txt` at the service root. The file contains groups of user-agent lines and rules. `Disallow` identifies paths the matching crawler is asked not to access. `Allow` permits a more specific path within a broader disallow rule.
For a matching group, the most specific path rule wins. A group for a named product token is more targeted than the `*` group. Test overlaps rather than relying on the order in which you happened to write the lines.
The protocol also defines retrieval and caching behavior. A crawler may cache a valid file, so a change need not appear in its behavior immediately. If availability matters, serve a small valid file reliably and monitor the response status.
## Use current operator tokens
Treat operator documentation as time-sensitive input. Check it when you create a rule and on a scheduled review date.
### OpenAI
[OpenAI's crawler documentation](https://developers.openai.com/api/docs/bots) distinguishes separate product tokens, including:
- `GPTBot` for model-development crawling;
- `OAI-SearchBot` for search;
- `OAI-AdsBot` with its own control;
- `ChatGPT-User` for certain user-initiated requests.
Blocking the training crawler does not require blocking the search crawler. OpenAI says `ChatGPT-User` requests are initiated by users and `robots.txt` rules may not apply to them, so decide user-directed retrieval separately.
OpenAI also publishes IP ranges for its crawlers. Those ranges help verify a request, but they do not belong in `robots.txt` itself.
To ask GPTBot not to crawl the site:
```
User-agent: GPTBot
Disallow: /
```
### Anthropic
[Anthropic's crawler documentation](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) distinguishes `ClaudeBot`, `Claude-SearchBot`, and `Claude-User`. Anthropic also links to crawler IP addresses and notes how its crawler interprets its documented tokens. Unlike OpenAI's user-triggered fetcher, Anthropic says its bots honor `robots.txt`, including the `Claude-User` control. Do not assume one operator's policy applies to another.
To disallow model-development crawling while leaving the other purposes to their own explicit policy:
```
User-agent: ClaudeBot
Disallow: /
```
### Google
[Google's common crawler reference](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers) documents `Google-Extended` as a control token. [Google-Extended](https://www.centinelanalytica.com/bots/google-extended) does not have a separate HTTP user-agent string; Google says existing Google user agents perform the crawl.
Google also states that the Google-Extended control does not affect inclusion or ranking in Google Search. That makes it different from blocking [Googlebot](https://www.centinelanalytica.com/bots/googlebot) itself.
```
User-agent: Google-Extended
Disallow: /
```
These examples are not a universal block list. They demonstrate how to translate a decided policy into operator-documented tokens.
### Weigh search visibility first
Blocking a search crawler has a visibility tradeoff. OpenAI says opting out of `OAI-SearchBot` removes the site from ChatGPT search answers, though navigational links can remain. Anthropic says disabling `Claude-SearchBot` prevents its search indexing and may reduce visibility. These are the operators' own descriptions. Allowing a crawler does not guarantee traffic from it.
## Restrict a crawler to part of the site
Suppose GPTBot may crawl `/docs/public/` but not the rest of `/docs/`:
```
User-agent: GPTBot
Disallow: /docs/
Allow: /docs/public/
```
Check the result against real URLs, including encoded paths, uppercase variants, query strings, and any redirects that expose the same resource elsewhere. A path rule is only useful when it matches the URLs your application actually serves.
Avoid listing private or secret paths as a security measure. The file is public, and a disallow rule does not stop a client from requesting the path.
## Test the file before release
Run the same checks in preview and after deployment:
1. Fetch the exact production path and retain the headers: `curl -i https://example.com/robots.txt`.
2. Confirm the response is successful, plain text, and contains the intended version.
3. Test representative allowed and disallowed URLs against the parser behavior you expect.
4. Confirm redirects, CDN rules, and host aliases do not serve a different file.
5. Save the last known-good file so a bad generation or deployment can be compared and rolled back.
For a quick read of the deployed file, the [robots.txt checker](https://www.centinelanalytica.com/robots-txt-checker) reports what it says to each AI bot token. It tests the home page only, so step 3 still applies to other paths.
Do not generate an unbounded list at request time. Build or store a reviewed file, validate its size and syntax, and serve it without depending on an operator endpoint during each crawler request.
## Monitor whether the preference changes traffic
A published rule and an enforcement decision answer different questions. Monitor both:
- the status and content returned from `/robots.txt`;
- requests carrying a token you disallowed;
- source IP and verification result for named crawlers;
- requested resource, response, and rate over time;
- the delay between a policy change and observed behavior.
Traffic after a change is not automatically proof that an operator ignored the rule. It may be a cached policy, a different product token, a spoofed user-agent, or a request that falls outside the changed path. Preserve these possibilities in the investigation.
The [AI-agent verification guide](https://www.centinelanalytica.com/blog/how-to-verify-ai-agents) explains how to evaluate a named claim using current operator evidence.
## What robots.txt cannot enforce
`robots.txt` does not authenticate a client, authorize access, rate-limit requests, or challenge a session. A user-agent string is unvalidated request input. Anyone can send one.
Google-Extended shows the gap. Google never sends that token in a request, so any request with
`Google-Extended` as its user agent is fake, and the `Disallow` rule above does not stop it. Our
[crawler verification data](https://www.centinelanalytica.com/blog/fake-crawler-report) shows how often crawler names are copied.
When a resource needs technical protection, enforce the decision at a point that actually sees the relevant request. Depending on the route and integration, that may include source-network verification, protocol evidence, behavioral limits, authentication, rate controls, or a challenge. No single signal proves every kind of automation. [How to block AI crawlers](https://www.centinelanalytica.com/blog/how-to-block-ai-crawlers) compares those request-time controls, including how to enforce a block against clients that copy a crawler's name.
Centinel can contribute browser, network, session, and crawler evidence to those request-time decisions. Coverage depends on the integration: browser-side collection does not govern every non-browser request, while server or edge controls must be placed on the traffic path they are meant to cover.
Use `robots.txt` to communicate. Use verified evidence and resource-specific rules when you need to enforce.
---
Page: https://www.centinelanalytica.com/blog/how-to-block-ai-crawlers
# How to block AI crawlers and verify the block
Block GPTBot, ClaudeBot and other AI crawlers: robots.txt rules by purpose, a verified request rule for copied names, and two curl checks to test it.
By Frederick Jahn. Published 2026-03-12, updated 2026-10-11.
Blocking an AI crawler takes two rules. A `robots.txt` group tells the documented
crawler to stay out, and the large operators say their crawlers follow it. A
request rule at your server or edge refuses everything `robots.txt` cannot reach:
clients that copy a crawler's name, fetchers the operators exempt, and scrapers
that send no name at all. Before you block, know
[what an AI crawler is](https://www.centinelanalytica.com/blog/what-is-an-ai-crawler) and which ones you want.
The group below covers the training crawlers of OpenAI and Anthropic and the
training opt-outs of Google and Apple.
```
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /
```
## Pick the method for what you want to stop
| You want to | Use | It will not stop |
| --- | --- | --- |
| Keep your pages out of model training | `robots.txt` groups for training tokens | A client that ignores the file, or one that copies the name |
| Refuse a named crawler's requests | A rule that matches the name and checks the operator's published IP list or DNS | Automation that sends a browser user agent |
| Slow a crawler you otherwise accept | A rate limit keyed on verified identity, per route group | A client spread over many addresses, if the key is the IP |
| Keep subscriber or account content private | Authentication and authorization in the application | Scraping of public pages |
| Stop automation that arrives as a browser | Detection on browser and connection evidence, with a challenge where a browser is expected | Plain HTTP clients on routes without the check |
Start with the first two rows. Add another when your logs show the
traffic it addresses, because each extra control is one more thing that can
block a reader or a search crawler by mistake.
## Step 1: write the `robots.txt` groups by purpose
Operators publish one token per purpose, so you can refuse training and keep
search. OpenAI separates `GPTBot` (training) from `OAI-SearchBot` (ChatGPT
search), and says a site that disallows `OAI-SearchBot` will not be shown in
ChatGPT search answers
([OpenAI crawlers](https://developers.openai.com/api/docs/bots)). Anthropic
separates `ClaudeBot` from `Claude-SearchBot` the same way
([Anthropic crawlers](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)).
Perplexity says `PerplexityBot` is a search crawler and is not used to collect
content for foundation models
([Perplexity crawlers](https://docs.perplexity.ai/docs/resources/perplexity-crawlers)).
`Google-Extended` and `Applebot-Extended` work differently. Neither one fetches
pages. [Googlebot](https://www.centinelanalytica.com/bots/googlebot) and [Applebot](https://www.centinelanalytica.com/bots/applebot) do the crawling, and the extended token only
controls whether the content may be used for Gemini training and grounding or
for Apple's foundation models
([Google's common crawlers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers),
[About Applebot](https://support.apple.com/en-us/119829)). Disallowing either
one leaves your search listing alone. It also means a firewall rule that matches
`Google-Extended` in the user-agent header does nothing useful: Google never
sends that string, so every request carrying it is a copy.
Put the file on every host you serve. Anthropic asks for the rule on each
subdomain you want to opt out. The
[robots.txt guide for AI bots](https://www.centinelanalytica.com/blog/robots-txt-ai-bots-guide) has the
matching rules, path examples, and the search tradeoff for each token.
## Step 2: know what `robots.txt` leaves open
When a person asks an assistant to read a page, the
request comes from a different client, and operators treat it as the person's
request. OpenAI says of `ChatGPT-User` that "robots.txt rules may not apply".
Perplexity says `Perplexity-User` "generally ignores robots.txt rules". Meta
says `meta-externalfetcher` may bypass the file, and Amazon says `Amzn-User` may
not follow every directive
([Meta web crawlers](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/),
[Amazonbot](https://developer.amazon.com/amazonbot)). Anthropic is the exception
in this group: it documents `Claude-User` as a token site owners can disallow.
If you want these fetches refused, you need the request rule in step 3.
Any HTTP client can also send `GPTBot` as its user agent. In our
[crawler verification data](https://www.centinelanalytica.com/blog/fake-crawler-report), which covers news and
media sites, 84% of requests that called themselves [GPTBot](https://www.centinelanalytica.com/bots/gptbot) on the typical site
failed OpenAI's published IP check. A copy has no reason to read your
`robots.txt`.
OpenAI, Perplexity, Meta, and Amazon each say a `robots.txt` change can take
about 24 hours to apply. Amazon's crawlers may also work from a copy of the
file up to 30 days old.
[RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html) says the rules "are not
a form of access authorization", and every path you list is readable by anyone.
Content that must stay restricted needs a login.
## Step 3: refuse the request at the server or edge
A rule that matches the user-agent string alone refuses the honest crawler and
is bypassed by changing one header. Tie the name to the proof its operator
publishes:
```
claim = match an exact configured crawler token
identity = verify the source using that client's published method
if claim is blocked and identity is verified:
deny with the configured response
else if claim exists and identity failed:
apply the spoofed-claim policy
else if verification is unavailable:
apply the preselected unknown-state fallback
else:
continue to the next policy check
```
The proof differs by operator. OpenAI and Perplexity publish one JSON IP list
per client, Anthropic publishes a single list for all three of its bots, and
Google and Apple support both IP ranges and forward-confirmed reverse DNS.
[How to verify AI agents](https://www.centinelanalytica.com/blog/how-to-verify-ai-agents) walks through each
method, and the [crawler IP checker](https://www.centinelanalytica.com/verify-crawler) tests one address by hand.
Check for four mistakes:
1. The rule reads the client address from `X-Forwarded-For` as the client sent
it. Take the address from your own proxy or CDN.
2. The range list is fetched once and never refreshed. Refresh it on a schedule,
bound the download size and time, and keep the last good copy when a refresh
fails.
3. All of a company's ranges are merged into one block. That removes the search
crawler along with the training crawler.
4. The block covers `/robots.txt`. Anthropic warns that blocking its addresses
"may not work correctly or persistently guarantee an opt-out", because the
bot can no longer read the file. Serve `/robots.txt` to everyone.
Your CDN may have a switch for this. Cloudflare, for example, lets a site block
bots by the behavior it assigns them (Search, Agent, or Training), including
some unverified ones
([Cloudflare: Block AI bots](https://developers.cloudflare.com/bots/additional-configurations/block-ai-bots/)).
A switch like that acts on the vendor's classification, so check which clients
it covers before you rely on it.
We do not publish Nginx, Apache, or WAF snippets here. A rule copied from an article has not been tested against your proxy chain,
and a wrong client address makes it match the wrong requests.
## Step 4: limit what you do not want to block
Some crawlers are welcome at a lower volume. Anthropic's bots honor the
non-standard `Crawl-delay` directive, and Amazon says its crawlers do not. For
everything else, a rate limit does the job, and its key decides whether it
works. An IP key is simple to run, but it groups unrelated visitors behind one
address and misses a client that rotates addresses. A verified crawler identity
is the better key for a named crawler, and an account or session is better on
logged-in routes.
Set the window, the burst allowance, and the routes before launch, and give an
expensive search endpoint a tighter budget than a cached article. The response
is `429 Too Many Requests`.
[RFC 6585](https://www.rfc-editor.org/rfc/rfc6585.html#section-4) lets you add a
`Retry-After` header and leaves the counting method to the server.
A browser challenge belongs only on routes where a browser is expected. A feed
reader, an API client, or a real crawler cannot complete one, and a challenge in front of every article interrupts readers.
[Fixing challenge loops](https://www.centinelanalytica.com/blog/fix-bot-challenge-loops) covers the failure
cases.
## Step 5: test the block
Check that the file is served, then that the request rule answers a claimed
crawler:
```
curl -i https://example.com/robots.txt
curl -I -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot" https://example.com/
```
The first command should return `200` and your groups. The
[robots.txt checker](https://www.centinelanalytica.com/robots-txt-checker) reads the same file and lists the
verdict for each AI bot token. The second comes from
your own machine, so it is a copied name: it should get your spoofed-claim
response, whatever you chose for that. It cannot show how the verified crawler
is treated. For that, wait a day and read the log lines whose source address is
on the operator's list.
Before you enforce, run these cases:
| Case | Expected result |
| --- | --- |
| Verified crawler covered by the block | Denied on the intended paths, with the reason logged |
| Verified crawler outside the block | Same access as before |
| Copied crawler name from an unrelated address | Treated as a spoofed claim |
| Client with no useful name | Gets the unknown-client policy |
| Range list refresh fails | Last good list stays in use and the failure is reported |
| Client sends its own forwarding header | Recorded source address does not change |
| Reader on an affected route | Page loads, with no challenge loop |
| Restricted page without credentials | Denied, whatever the crawler name |
After rollout, watch requests, status codes, bytes, and reader-facing errors,
split by policy reason. A falling count for one crawler name has three possible
causes: the block worked, the client changed its name, or your classifier
stopped recognizing it. Check the verified share for that name, and whether the
same source addresses now appear under another name.
## Roll back
Give every blocking rule an owner, a review date, and a one-step way to turn it
off, and keep the previous rule set. Switch to observe-only when an approved
search or partner crawler is denied, when reader errors or challenge loops rise,
or when the verifier loses its range source. Keep the logged reasons and the
rule version afterwards, so the next attempt fixes the cause.
## Blocking scrapers that send no crawler name
The steps above handle clients that name themselves, truthfully or otherwise. A
scraper that runs a real or modified browser sends a browser user agent, and no
crawler rule sees it. Finding it takes evidence from the browser and the connection.
Centinel helps teams identify and control that traffic. The
[scraping and AI crawler use case](https://www.centinelanalytica.com/use-cases/scraping-and-ai-crawlers)
describes how the two are handled together, and the
[integration options](https://www.centinelanalytica.com/integrations) show where the decision can run.
If you are not sure which clients your current rules would cut off,
[request a site audit](https://www.centinelanalytica.com/free-audit) with the routes and the crawlers you want to
keep.
## Sources
- [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309.html)
- [RFC 6585: 429 Too Many Requests](https://www.rfc-editor.org/rfc/rfc6585.html#section-4)
- [OpenAI crawler documentation](https://developers.openai.com/api/docs/bots)
- [Anthropic crawler documentation](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)
- [Perplexity crawler documentation](https://docs.perplexity.ai/docs/resources/perplexity-crawlers)
- [Google's common crawlers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers)
- [About Applebot](https://support.apple.com/en-us/119829)
- [Meta web crawlers](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/)
- [Amazonbot](https://developer.amazon.com/amazonbot)
- [Cloudflare: Block AI bots](https://developers.cloudflare.com/bots/additional-configurations/block-ai-bots/)
---
Page: https://www.centinelanalytica.com/blog/what-is-web-scraping
# What is web scraping?
Understand how web scraping works, how it differs from crawling and browser automation, and how to build a resource-specific response.
By Frederick Jahn. Published 2026-03-10, updated 2026-10-11.
Web scraping is the automated extraction of information from web resources. A scraper may download HTML, call an endpoint used by a page, render the page in a browser, select the relevant fields, and store the result in a structured form.
The technique is not inherently one thing. Search tools, accessibility services, testing systems, internal integrations, researchers, price monitors, and abusive operators can all automate web access. Whether traffic is wanted depends on identity, purpose, resource, rate, and site policy.
## How web scraping works
A scraping workflow commonly contains these stages:
1. **Discover:** obtain URLs from a sitemap, link graph, feed, search result, or supplied list.
2. **Fetch:** request a document or endpoint using HTTP.
3. **Render if needed:** run browser code when the required data is not present in the initial response.
4. **Extract:** select text, links, attributes, or structured data.
5. **Normalize:** convert the result into records with stable fields.
6. **Store and revisit:** save the records and schedule updates.
[RFC 9110](https://www.rfc-editor.org/rfc/rfc9110) defines HTTP semantics, but it does not assign a benign or abusive purpose to a request. The same `GET` request can come from a person, an approved integration, a search crawler, or an unwanted scraper.
Simple scrapers may use an HTTP library and an HTML parser. Browser-based scrapers can execute JavaScript, interact with controls, and retain session state. Distributed systems can spread retrieval across many workers or networks. These are implementation choices, not reliable policy categories by themselves.
## Web crawling, scraping, and browser automation are different
The terms overlap but answer different questions:
| Term | Primary action | Example output |
| --- | --- | --- |
| Web crawling | Discover and retrieve resources | A URL index or fetched-document set |
| Web scraping | Extract selected information | Structured records, text, or media references |
| Browser automation | Control a browser workflow | Navigation, form completion, testing, or rendered data |
A crawler can fetch pages without extracting their contents. A scraper can work from a fixed URL list without discovering links. Browser automation can perform legitimate quality assurance without scraping, while a scraper may use a browser when the page requires rendering. Scraping APIs such as [Firecrawl](https://www.centinelanalytica.com/blog/what-is-firecrawl) and [Zyte](https://www.centinelanalytica.com/blog/what-is-zyte) can fetch with plain HTTP requests or managed browsers, so one service can reach your site in more than one form.
The [WebDriver specification](https://www.w3.org/TR/webdriver2/) standardizes one form of remote browser control. Its existence is also a reminder that legitimate automation is common. “Browser controlled” and “abusive scraping” are not interchangeable verdicts.
## Named crawlers need purpose-specific rules
Some operators publish crawler names and guidance. Others do not. Even a documented operator may use separate product tokens for model development, search, or user-directed retrieval.
`robots.txt` can express a crawl preference to a conforming crawler. [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309) defines the protocol. It does not authenticate a requester or enforce authorization, so a user-agent claim must not be treated as trusted identity on its own.
For a declared crawler:
- record the exact token and requested resource;
- consult current operator-primary documentation;
- verify the source using the operator's published method when one exists;
- assign policy by purpose and resource;
- define behavior when identity cannot be resolved.
For undeclared traffic, preserve uncertainty. “Unknown” is a useful state. It is better than forcing a weak observation into a named operator or human/bot label.
## Define the unwanted behavior first
Before choosing a control, write down the outcome you need to change. Examples include:
- repeated retrieval that creates material origin or application work;
- automated extraction of a resource your policy does not permit in bulk;
- inventory or price monitoring at a rate that harms a service;
- automated account actions such as credential testing or mass registration;
- bypass of an explicit authentication or authorization boundary.
The [OWASP Automated Threats to Web Applications project](https://owasp.org/www-project-automated-threats-to-web-applications/) provides a vocabulary for automated misuse. Use that vocabulary to keep the problem statement tied to an application action rather than a vague dislike of bots.
Different resources can justify different responses. A public product page, an expensive search operation, and a private account endpoint should not inherit the same rule just because requests share a fingerprint.
## How to investigate suspected scraping
Use observations that can be reproduced and that fail independently.
### Resource and rate
Identify the affected routes, request methods, response sizes, cache behavior, and downstream work. Compare the suspected cohort with an appropriate baseline. A high request count can be normal for a feed or monitoring integration and harmful for a costly search route.
### Claimed and verified identity
Separate the user-agent claim from verification. Where an operator publishes current network ranges or reverse-DNS guidance, test the claim against that source. Record the source version and the unresolved cases.
### Client consistency
Compare browser, transport, header, and runtime observations that should agree. A mismatch contributes evidence, but network middleware, privacy tools, enterprise software, and unusual devices can also change these surfaces.
### Session and application sequence
Look at navigation order, retries, concurrency, state transitions, and protected-action outcomes. Do not assume that fast, regular, or non-interactive behavior is automatically malicious. Internal systems and assistive tools may have similar shapes.
### Ground truth and error
Build labeled test cohorts where possible: ordinary supported browsers, approved automation, stock automation, and the suspected workflow. Measure false positive and false negative behavior. Without reviewed examples, a score change is not proof that scraping was found.
## Choose a bounded response
Match the control to the evidence and the resource:
| Situation | Possible response |
| --- | --- |
| Known approved integration | Authenticate, scope, and monitor |
| Documented crawler with verified identity | Apply its explicit resource policy |
| Unknown low-impact traffic | Observe or use a generous bounded rate |
| Uncertain traffic on a costly route | Rate-limit or challenge with a recovery path |
| Corroborated policy violation | Deny the affected action and retain review evidence |
Start in observation mode. Review the cohorts that would be affected, then roll out on a limited set of routes. Define rollback thresholds and a way to correct false positives before enforcement expands.
Controls can include authenticated APIs, cache design, pagination limits, request budgets, rate limits, crawler verification, session analysis, challenges, and route-level deny rules. Their usefulness depends on placement. Browser code cannot govern a direct request that never loads it, and an edge rule may not understand application state unless that state is provided safely.
The [scraping and AI crawler use case](https://www.centinelanalytica.com/use-cases/scraping-and-ai-crawlers) explains how these controls fit together.
## Separate legal review from traffic enforcement
This section is operational guidance, not legal advice. The legal treatment of scraping can depend on jurisdiction, the resource, authorization, contract formation, copyright or database rights, privacy obligations, technical circumvention, and the facts of the collection and use.
Avoid universal statements such as “public data is always legal to scrape” or “a `robots.txt` rule makes access unlawful.” Neither is a safe technical conclusion. Ask qualified counsel to review the specific policy and use case.
Engineering can make that review more concrete by preserving:
- the resource and access state at the time of each request;
- the policy version and how it was communicated;
- available identity evidence and its source;
- rate, sequence, response, and enforcement outcome;
- retention limits and access controls for the evidence.
A legal position does not replace an enforceable technical boundary, and a technical block does not settle a legal question. Keep ownership and review for both.
## Build a policy you can revisit
Scraping tools, browser versions, operator documentation, and business priorities change. Give each rule an owner, evidence source, review date, and expiry condition. Remove rules whose original justification no longer exists.
The [AI-agent access policy guide](https://www.centinelanalytica.com/blog/how-to-control-ai-agent-access) provides a reusable resource-by-identity matrix. The [bot management overview](https://www.centinelanalytica.com/bot-management) lists the product and integration questions to test before choosing an enforcement layer.
Web scraping is an extraction method. A defensible response starts with the affected resource and observed behavior, then adds identity evidence, error measurement, and a proportionate control.
---
Page: https://www.centinelanalytica.com/blog/what-is-an-ai-crawler
# What is an AI crawler? Types and examples
AI crawlers collect web content for model development, AI search and retrieval. How they work, the main operators' crawlers, and what your logs can show.
By Frederick Jahn. Published 2026-03-10, updated 2026-10-11.
An AI crawler is an automated web client that collects or retrieves content
for an AI system. It may gather pages that could be used in model development,
build an index for AI search, or keep a retrieval system current.
The name describes what the operator does with the pages. On the wire an AI
crawler is an ordinary HTTP client, the same as a search crawler. It usually
sends a user-agent string such as `GPTBot/1.4`, and that string is a claim until
the source address is checked against the operator's published list.
The category overlaps with search. Apple says pages fetched by [Applebot](https://www.centinelanalytica.com/bots/applebot) for
search may also be used to train its foundation models, and Google uses one
crawl for Search and for Gemini unless a site opts out. User-triggered fetchers
and browser agents add AI-related traffic that follows no crawl schedule.
## AI crawler, search crawler, or agent?
| Client type | What triggers it | Main job | Examples |
| --- | --- | --- | --- |
| Search crawler | An operator's crawl schedule | Discover and refresh a search index | `Googlebot`, `Applebot` |
| AI training crawler | An operator's collection schedule | Collect content that may be used in model development | [`GPTBot`](https://www.centinelanalytica.com/bots/gptbot), [`ClaudeBot`](https://www.centinelanalytica.com/bots/claudebot) |
| AI search crawler | An operator's crawl or indexing system | Make content available to an AI search product | `OAI-SearchBot`, `Claude-SearchBot`, `PerplexityBot` |
| User-triggered fetcher | A person's live request to a product | Retrieve one or more resources for that request | `ChatGPT-User`, `Claude-User`, `Perplexity-User` |
| Agentic client | A delegated user or application goal | Choose and perform steps through pages or APIs | Browser agents, which often send a browser user agent |
One operator can run a client in every row and publish a separate control for
each, so record the stated purpose beside the name.
[Agentic traffic](https://www.centinelanalytica.com/blog/what-is-ai-agent-traffic) covers user-triggered
retrieval and task-running clients.
## How an AI crawler works
### 1. Discover URLs
The crawler starts with known URLs, sitemaps, feeds, or links found on pages it
has already fetched. Site owners may also submit URLs through an operator's
product.
### 2. Fetch a resource
The client sends an HTTP request for HTML, images, PDFs, feeds, JavaScript, or
whatever else the operator's job needs. The request usually carries the
operator's published user-agent token.
Support for `robots.txt`, crawl-rate settings, caching, and page-level
directives differs by operator, so read each one's current documentation.
### 3. Parse or render the response
A basic fetcher can read the returned document and extract text and links. A
crawler that needs client-rendered content may also execute JavaScript and load
supporting assets. Access logs can show those additional requests, but they do
not reveal the crawler's complete parsing pipeline.
### 4. Store or use the result
The operator may place the result in a search index, a retrieval system, or a
collection used for model development. This step happens outside the site's
infrastructure.
### 5. Revisit
Crawlers return to find new pages or refresh old ones. Cache validators such as
`ETag` and `Last-Modified` can reduce the amount of unchanged content
transferred when a crawler supports them. Google documents its use of both in
its [crawler overview](https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers).
## Current AI crawler reference
This table was checked against operator documentation on October 6, 2026. It
leaves out names whose purpose could not be sourced. Meta, Amazon, and Common
Crawl also document crawlers; the [AI crawler list](https://www.centinelanalytica.com/ai-crawler-list) has more
names.
| Operator and client | Documented purpose | HTTP identifier | Verification source | Published control |
| --- | --- | --- | --- | --- |
| OpenAI `GPTBot` | Crawls content that may be used to train generative AI foundation models | `GPTBot` user agent | Product-specific JSON IP list | `GPTBot` rules in `robots.txt` |
| OpenAI `OAI-SearchBot` | Surfaces sites in ChatGPT search results | `OAI-SearchBot` user agent | Product-specific JSON IP list | `OAI-SearchBot` rules in `robots.txt` |
| Anthropic `ClaudeBot` | Collects web content that could contribute to model training | `ClaudeBot` user agent | IP list linked from Anthropic's crawler page | `ClaudeBot` rules in `robots.txt` |
| Anthropic `Claude-SearchBot` | Indexes content to improve search results | `Claude-SearchBot` user agent | IP list linked from Anthropic's crawler page | `Claude-SearchBot` rules in `robots.txt` |
| Perplexity `PerplexityBot` | Surfaces and links sites in Perplexity search; the operator says it is not used for foundation-model training | `PerplexityBot` user agent | Product-specific JSON IP list | `PerplexityBot` rules in `robots.txt` |
| Google `Googlebot` and `Google-Extended` | [Googlebot](https://www.centinelanalytica.com/bots/googlebot) crawls for Search; [Google-Extended](https://www.centinelanalytica.com/bots/google-extended) controls specified training and grounding uses of content Google crawls | `Googlebot` appears in requests; `Google-Extended` does not have its own HTTP user agent | Published IP ranges or forward-confirmed reverse DNS | Separate `Googlebot` and `Google-Extended` rules |
| Apple `Applebot` and `Applebot-Extended` | Applebot crawls for search in Apple products, and Apple says the data may also be used to train its foundation models; [Applebot-Extended](https://www.centinelanalytica.com/bots/applebot-extended) is the opt-out from that training use | `Applebot` appears in requests; `Applebot-Extended` does not crawl webpages | Published CIDR list or forward-confirmed reverse DNS | Separate `Applebot` and `Applebot-Extended` rules |
Sources: [OpenAI](https://developers.openai.com/api/docs/bots),
[Anthropic](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler),
[Perplexity](https://docs.perplexity.ai/docs/resources/perplexity-crawlers),
[Google](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers),
and [Apple](https://support.apple.com/en-us/119829).
`ChatGPT-User`, `Claude-User`, and `Perplexity-User` are left out of the table
because their operators describe them as acting on a user's request. OpenAI and
Perplexity also say `robots.txt` rules may not apply to them.
## What appears in access logs
A GPTBot request in a combined-format access log looks like this (the address
is a documentation placeholder):
```
203.0.113.7 - - [06/Oct/2026:09:14:02 +0000] "GET /pricing HTTP/1.1" 200 18432 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot"
```
Nothing in the line proves the sender. The user agent is the string OpenAI
documents, and anyone can send it. The address is the only field to check, and
only if your proxy or CDN recorded it. To make a line like this usable, retain:
- time, method, host, path, query, status, and response bytes;
- the complete user-agent header;
- the source IP observed by trusted infrastructure;
- request and trace identifiers;
- cache status and origin timing where available;
- the result of any identity or automation check;
- the policy action and its reason.
Verify named traffic against the method its operator publishes before you
attach an allow rule to the name.
[How to verify AI agents](https://www.centinelanalytica.com/blog/how-to-verify-ai-agents) explains the IP, DNS,
and signature options.
Copying is common on sites the real crawler rarely visits. Our
[crawler verification data](https://www.centinelanalytica.com/blog/fake-crawler-report) covers news and media
sites, and on the typical site in it 84% of requests claiming to be `GPTBot`
failed OpenAI's published check. Any request that sends `Google-Extended` as its
user agent is fake, because Google never sends that token. A log search for
`Google-Extended` or `Applebot-Extended` therefore counts no real crawler.
A request with an empty or generic user agent has no name to check. Connection
and behavioral signals may establish that it is automated, but its operator and
purpose can stay unknown.
## What logs cannot establish
Server logs cannot directly show whether served content later entered a
training dataset, appeared in a search index, grounded an answer, or was
discarded. That processing happens after the response leaves the site.
Logs also do not prove compliance with a published purpose. They can show which
rules were presented, which routes were requested, how much data was returned,
and whether a verified client stayed within the site's access policy. Keep
those observations separate from claims about downstream use.
## Effects a site owner can measure
Measure effects that appear at your own boundary:
| Measure | Why it helps |
| --- | --- |
| Requests and unique resources | Shows crawl breadth without pretending every request is useful content |
| Response bytes | Quantifies data served by client and route group |
| Cache hits and origin work | Separates cheap edge responses from requests that consume application resources |
| Response status | Finds errors, redirects, challenges, and denied requests |
| Revisit interval | Shows how often a client refreshes the same resource |
| Restricted-route attempts | Reveals whether the crawler reaches archives, account routes, APIs, or other controlled areas |
| Verified identity rate | Separates named claims from requests that pass the operator's published check |
Measure referrals from AI search products separately in analytics.
## Build a crawler inventory before setting policy
Start with a short inventory of clients that already appear in your logs.
1. Extract claimed crawler tokens and the routes they request.
2. Create one record per client and purpose.
3. Store the official documentation URL and last review date.
4. Record whether the identifier is an HTTP user agent or only a control token.
5. Add the published verification method.
6. Measure requests, bytes, cache behavior, and restricted-route attempts.
7. Assign an explicit policy owner and review date.
A static "all AI bots" list goes stale. OpenAI's documentation has moved to a
new address, and tokens such as `Claude-Web` that older lists still carry
are absent from Anthropic's current page. Review each record against the
operator's page on the date you set.
[How to audit AI crawler traffic](https://www.centinelanalytica.com/blog/audit-ai-crawler-traffic) covers the
measurement in detail.
Once the policy is clear, [How to block AI crawlers](https://www.centinelanalytica.com/blog/how-to-block-ai-crawlers) covers the `robots.txt` group, the
request rule, and the tests. For different responses by identity and resource,
use [How to control AI agent access](https://www.centinelanalytica.com/blog/how-to-control-ai-agent-access).
## FAQ
### Is Googlebot an AI crawler?
Googlebot is Google's search crawler. Google can also use what it crawls for
Gemini training and grounding, and the `Google-Extended` token in `robots.txt`
opts a site out of that use without affecting Search.
### Do AI crawlers obey robots.txt?
OpenAI, Anthropic, Google, Apple, Meta, and Amazon each say their automatic
crawlers follow it. The protocol is voluntary, a client that copies a crawler's
name is not bound by it, and several operators exempt user-triggered fetchers.
### Does an AI crawler use AI to crawl?
No. The name refers to where the content goes. The fetching is ordinary HTTP,
and some crawlers also render the page in a browser. Apple says Applebot may.
## Finding the crawlers on your own site
Centinel helps teams identify and control automated traffic, including
[AI crawlers and scrapers](https://www.centinelanalytica.com/use-cases/scraping-and-ai-crawlers). For an
inventory of the automation reaching your current pages,
[request a site audit](https://www.centinelanalytica.com/free-audit).
## Sources
- [OpenAI crawler documentation](https://developers.openai.com/api/docs/bots)
- [Anthropic crawler documentation](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)
- [Perplexity crawler documentation](https://docs.perplexity.ai/docs/resources/perplexity-crawlers)
- [Google crawler overview](https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers)
- [Google's common crawlers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers)
- [Applebot documentation](https://support.apple.com/en-us/119829)
---
Page: https://www.centinelanalytica.com/compare/centinel-vs-akamai
# Centinel vs Akamai Bot Manager: 2026 Detection Test
Centinel and Akamai Bot Manager head to head: in our September 2026 test Centinel detected 8 of 8 scraping tools and Akamai 2. Fit, setup, and CSV.
By Frederick Jahn. Published 2026-09-01, updated 2026-09-30.
## Summary
| Compared on | Akamai Bot Manager | Centinel |
| ----------------------------- | ---------------------------------------------------------------------- | ---------------------------------------------------------------------------------- |
| September 2026 test (8 tools) | 2 of 8 detected | 8 of 8 detected |
| Where it runs | Akamai's edge, for traffic delivered through Akamai | Edge, CDN, reverse-proxy, server, and CMS integrations, including Akamai EdgeWorkers |
| Buying model | Confirm with Akamai | Quote from the team after scoping; the site audit is free |
| Data location | Confirm with Akamai | Data stored in the EU |
Akamai Bot Manager runs on Akamai's edge. Centinel is a separate layer that runs on Akamai
EdgeWorkers or on another stack, so you can add it next to Akamai or switch to it.
## The benchmark
In September 2026 we ran eight scraping and browser-automation tools against Centinel and against Akamai: Browser-use, Firecrawl, Browserbase, kernel.sh, Playwright Stealth, Camoufox, Zyte, and Ulixee Hero.
Centinel detected all eight. Akamai detected 2 of 8.
## Per-tool detection results
| Scraping tool | Centinel | Akamai |
| ------------------ | --------- | ------------ |
| Browser-use | Detected | Not detected |
| Firecrawl | Detected | Not detected |
| Browserbase | Detected | Not detected |
| kernel.sh | Detected | Not detected |
| Playwright Stealth | Detected | Detected |
| Camoufox | Detected | Not detected |
| Zyte | Detected | Not detected |
| Ulixee Hero | Detected | Detected |
| **Detected** | **8 / 8** | **2 / 8** |
The table reports detection in the September 2026 test. See the
[benchmark scope](https://www.centinelanalytica.com/compare/centinel-vs-cdn-bot-protection#september-2026-benchmark-scope) for the
tools and products compared.
[Download the published per-tool results (CSV)](https://www.centinelanalytica.com/research/2026-09-bot-detection.csv).
## When Akamai fits better
- **Your traffic already runs through Akamai and you want one vendor.** Bot Manager
[assigns a bot score at the edge](https://www.akamai.com/products/bot-manager), so it acts before
traffic reaches your origin without another integration.
- **You want graded response segments.** Akamai describes Cautious, Strict, and Aggressive response
segments on the bot score, with APIs for automation. Teams that tune responses that way may not
need another layer.
- **The result is good enough for your routes.** If the automation that costs you money is already
handled, keep it.
## Where Centinel fits
Centinel fits when browser-like automation still reaches a route that matters. In the September
test, Centinel detected all eight tools, including Browser-use, Firecrawl, Browserbase, kernel.sh,
Camoufox, and Zyte, which Akamai did not detect.
It also fits sites that are not fully on Akamai. Centinel has guides for Akamai EdgeWorkers and for
other CDNs, proxies, servers, and CMSs, so one policy can cover a mixed stack. Centinel separates
declared and verified crawlers from spoofed or hidden automation; you can check one address with the
[crawler verification tool](https://www.centinelanalytica.com/verify-crawler).
## Feature comparison
| Area | Akamai Bot Manager (from its product page) | Centinel |
| ----------------- | -------------------------------------------------------------- | ---------------------------------------------------------------------------------------- |
| Deployment | Akamai edge | 15 documented guides, including Akamai EdgeWorkers, AWS CloudFront, and Cloudflare |
| Scoring | Bot score assigned at the edge | Request, browser, and connection evidence evaluated together |
| Responses | Cautious, Strict, and Aggressive response segments | Identify, observe, rate-limit, challenge, allow, or block, depending on the integration |
| Crawler identity | Confirm current crawler verification with Akamai | Separates declared and verified crawlers from spoofed ones |
| Pricing model | Confirm with Akamai | Quote after scoping |
| EU and GDPR | Confirm data location with Akamai | Data stored in the EU; ISO/IEC 27001 certified; assessed against SOC 2 Type 1 |
Source: Akamai's [Bot Manager product page](https://www.akamai.com/products/bot-manager).
Centinel's guides are on the [integrations page](https://www.centinelanalytica.com/integrations); its security posture is in the
[trust center](https://trust.getkantis.com/centinel-analytica-gmbh).
## Setup effort and pricing model
Bot Manager runs in your Akamai configuration; ask Akamai for its setup steps. Centinel on Akamai
runs as an EdgeWorker; on other stacks it is a module on the CDN, proxy, or server. Plan for one
engineer who can change that configuration, the owner of the protected workflow, and a rollback path.
Ask Akamai for its current pricing model. Centinel quotes per deployment: bring your domain,
typical and peak traffic, and current stack. The [site audit](https://www.centinelanalytica.com/free-audit) is free and needs no
card.
## Adding Centinel to Akamai, or moving off it
You do not have to leave Akamai to use Centinel. You can keep the CDN and add Centinel for the
routes where automation still gets through.
1. **Pick the routes.** List where automation costs the most and which automation you want to
keep, such as search crawlers and monitoring.
2. **Deploy in observe mode.** Add the Centinel EdgeWorker on those routes while Bot Manager keeps
enforcing. Nothing changes for visitors.
3. **Compare decisions.** Review where the two disagree, using the tools above and any workflow
you can reproduce safely.
4. **Decide who enforces.** Give one product the final decision per route, so an allow in one
cannot silently override a block in the other.
5. **Switch route by route.** Move enforcement, check legitimate-user completion and error rates,
and keep a rollback.
The [proof-of-concept scorecard](https://www.centinelanalytica.com/blog/bot-protection-proof-of-concept) turns these steps into a
written evaluation plan.
## FAQ
### Is Centinel an Akamai Bot Manager alternative?
Yes, and it can also run next to Akamai. In the September 2026 test, Centinel detected 8 of the 8
tools and Akamai detected 2.
### Do I have to leave Akamai?
No. Centinel has an Akamai EdgeWorkers integration and works alongside your existing CDN and WAF.
### Does Centinel publish its prices?
No. Contact the team for current pricing, included usage, and contract terms. The site audit is
free.
### Where does Centinel store data?
Data is stored in the EU. Centinel is certified to ISO/IEC 27001 and assessed against SOC 2 Type 1.
## Related
- [Centinel vs. CDN and WAF bot protection](https://www.centinelanalytica.com/compare/centinel-vs-cdn-bot-protection)
- [All comparisons](https://www.centinelanalytica.com/compare)
- [How to detect browser automation](https://www.centinelanalytica.com/blog/detect-browser-automation)
---
Page: https://www.centinelanalytica.com/compare/centinel-vs-cloudflare
# Centinel vs Cloudflare Bot Management: 2026 Detection Test
Centinel and Cloudflare Bot Management head to head: in our September 2026 test Centinel detected 8 of 8 scraping tools and Cloudflare 3. Setup and CSV.
By Frederick Jahn. Published 2026-09-01, updated 2026-09-30.
## Summary
| Compared on | Cloudflare | Centinel |
| ----------------------------- | --------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------- |
| September 2026 test (8 tools) | 3 of 8 detected | 8 of 8 detected |
| Where it runs | Cloudflare's network, for traffic proxied through Cloudflare | Edge, CDN, reverse-proxy, server, and CMS integrations, including Cloudflare Workers |
| Buying model | Bot Fight Mode on Free; Super Bot Fight Mode on Pro and up; Bot Management as an Enterprise add-on | Quote from the team after scoping; the site audit is free |
| Data location | Data Localization Suite, an Enterprise add-on, sets where data is inspected and stored | Data stored in the EU |
Cloudflare's bot features come with its CDN. Centinel is a separate layer that runs on Cloudflare
Workers or on another stack, so you can add it next to Cloudflare or switch to it.
## The benchmark
In September 2026 we ran eight scraping and browser-automation tools against Centinel and against Cloudflare: Browser-use, Firecrawl, Browserbase, kernel.sh, Playwright Stealth, Camoufox, Zyte, and Ulixee Hero.
Centinel detected all eight. Cloudflare detected 3 of 8.
## Per-tool detection results
| Scraping tool | Centinel | Cloudflare |
| ------------------ | --------- | ------------ |
| Browser-use | Detected | Not detected |
| Firecrawl | Detected | Not detected |
| Browserbase | Detected | Not detected |
| kernel.sh | Detected | Detected |
| Playwright Stealth | Detected | Detected |
| Camoufox | Detected | Not detected |
| Zyte | Detected | Not detected |
| Ulixee Hero | Detected | Detected |
| **Detected** | **8 / 8** | **3 / 8** |
The table reports detection in the September 2026 test. See the
[benchmark scope](https://www.centinelanalytica.com/compare/centinel-vs-cdn-bot-protection#september-2026-benchmark-scope) for the
tools and products compared.
[Download the published per-tool results (CSV)](https://www.centinelanalytica.com/research/2026-09-bot-detection.csv).
## When Cloudflare fits better
- **You want bot controls with no second vendor.** Every Cloudflare plan includes a bot feature.
[Bot Fight Mode](https://developers.cloudflare.com/bots/plans/free/) on the Free plan challenges
simple bots from cloud hosting providers and headless browsers across the whole domain.
[Super Bot Fight Mode](https://developers.cloudflare.com/bots/plans/biz-and-ent/) on Pro,
Business, and Enterprise lets you allow, block, or challenge detected bots.
- **You write your own rules.** With
[Bot Management for Enterprise](https://developers.cloudflare.com/bots/get-started/bot-management/),
each request gets a [bot score](https://developers.cloudflare.com/bots/concepts/bot-score/) from 1
to 99 that you use in WAF custom rules or Workers. Teams that want that control inside Cloudflare
may not need another layer.
- **Your traffic already goes through Cloudflare, and the result is good enough.** If the
automation that costs you money is already handled, keep it.
## Where Centinel fits
Centinel fits when browser-like automation still reaches a route that matters. In the September
test, Centinel detected all eight tools, including Browser-use, Firecrawl, Browserbase, Camoufox,
and Zyte, which Cloudflare did not detect.
It also fits sites that are not fully on Cloudflare. Centinel has guides for Cloudflare Workers and
for other CDNs, proxies, servers, and CMSs, so one policy can cover a mixed stack. Centinel
separates declared and verified crawlers from spoofed or hidden automation; Cloudflare also
[classifies verified bots](https://developers.cloudflare.com/bots/concepts/bot/verified-bots/) by
category, so compare how each handles the crawlers you care about. You can check one address with
the [crawler verification tool](https://www.centinelanalytica.com/verify-crawler).
## Feature comparison
| Area | Cloudflare (from its documentation) | Centinel |
| ---------------------- | ------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------- |
| Deployment | Built into the Cloudflare proxy | 15 documented guides, including Cloudflare Workers, AWS CloudFront, Akamai, and Fastly |
| Detection | Bot Management: heuristics, machine learning, anomaly detection, and JavaScript detections | Request, browser, and connection evidence evaluated together |
| Crawler identity | Verified bots, grouped by category | Separates declared and verified crawlers from spoofed ones |
| Responses | Allow, block, or challenge; custom rules on the bot score | Identify, observe, rate-limit, challenge, allow, or block, depending on the integration |
| Reporting | Bot Analytics (Enterprise) | Centinel dashboard |
| Pricing model | Included by plan; Bot Management added to Enterprise plans by the account team | Quote after scoping |
| EU and GDPR | Data Localization Suite (Enterprise add-on) | Data stored in the EU; ISO/IEC 27001 certified; assessed against SOC 2 Type 1 |
Sources: Cloudflare's [Bot Management subscription page](https://developers.cloudflare.com/bots/plans/bm-subscription/)
and [Data Localization Suite](https://developers.cloudflare.com/data-localization/). Centinel's
guides are on the [integrations page](https://www.centinelanalytica.com/integrations); its security posture is in the
[trust center](https://trust.getkantis.com/centinel-analytica-gmbh).
## Setup effort and pricing model
Turning on Bot Fight Mode or Super Bot Fight Mode is a setting in the Cloudflare dashboard.
Bot Management scores need rules written against them. Centinel on Cloudflare runs as a Worker in
front of your site; on other stacks it is a module on the CDN, proxy, or server. Plan for one
engineer who can change that configuration, the owner of the protected workflow, and a rollback
path.
Cloudflare's bot features come with the plan, and Bot Management is sold through its account team.
Centinel quotes per deployment: bring your domain, typical and peak traffic, and current stack. The
[site audit](https://www.centinelanalytica.com/free-audit) is free and needs no card.
## Adding Centinel to Cloudflare, or moving off it
You do not have to leave Cloudflare to use Centinel. You can keep the CDN and add Centinel for
the routes where automation still gets through.
1. **Pick the routes.** List where automation costs the most and which automation you want to
keep, such as search crawlers and monitoring.
2. **Deploy in observe mode.** Add the Centinel Worker on those routes. Cloudflare's rules keep
enforcing; nothing changes for visitors.
3. **Compare decisions.** Review where the two disagree, using the tools above and any workflow
you can reproduce safely.
4. **Decide who enforces.** Give one product the final decision per route, so an allow in one
cannot silently override a block in the other.
5. **Switch route by route.** Move enforcement, check legitimate-user completion and error rates,
and keep a rollback.
The [proof-of-concept scorecard](https://www.centinelanalytica.com/blog/bot-protection-proof-of-concept) turns these steps into a
written evaluation plan.
## FAQ
### Is Centinel a Cloudflare Bot Management alternative?
Yes, and it can also run next to Cloudflare. In the September 2026 test, Centinel detected 8 of the
8 tools and Cloudflare detected 3.
### Do I have to leave Cloudflare?
No. Centinel has a Cloudflare Workers integration and works alongside your existing CDN and WAF.
### Does Centinel publish its prices?
No. Contact the team for current pricing, included usage, and contract terms. The site audit is
free.
### Where does Centinel store data?
Data is stored in the EU. Centinel is certified to ISO/IEC 27001 and assessed against SOC 2 Type 1.
## Related
- [Cloudflare Bot Management alternatives compared](https://www.centinelanalytica.com/alternatives/cloudflare)
- [Centinel vs. CDN and WAF bot protection](https://www.centinelanalytica.com/compare/centinel-vs-cdn-bot-protection)
- [All comparisons](https://www.centinelanalytica.com/compare)
---
Page: https://www.centinelanalytica.com/compare/centinel-vs-datadome
# Centinel vs DataDome: 2026 Detection Test
Centinel and DataDome head to head: in our September 2026 test Centinel detected 8 of 8 scraping tools and DataDome 5. Fit, setup, migration, and CSV.
By Frederick Jahn. Published 2026-09-01, updated 2026-09-30.
## Summary
| Compared on | DataDome | Centinel |
| ----------------------------------- | -------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------- |
| September 2026 test (8 tools) | 5 of 8 detected | 8 of 8 detected |
| Where it runs | Server-side modules, CDN and edge recommended, plus a JavaScript tag and SDKs | Edge, CDN, reverse-proxy, server, and CMS integrations with browser-side checks |
| Buying model | Plans published on its pricing page; some products priced by request volume | Quote from the team after scoping; the site audit is free |
| Data location | Regional API endpoints, including Europe | Data stored in the EU |
Both products detect automated traffic and let a site decide how to respond.
## The benchmark
In September 2026 we ran eight scraping and browser-automation tools against Centinel and against DataDome: Browser-use, Firecrawl, Browserbase, kernel.sh, Playwright Stealth, Camoufox, Zyte, and Ulixee Hero.
Centinel detected all eight. DataDome detected 5 of 8.
## Per-tool detection results
| Scraping tool | Centinel | DataDome |
| ------------------ | --------- | ------------ |
| Browser-use | Detected | Not detected |
| Firecrawl | Detected | Detected |
| Browserbase | Detected | Not detected |
| kernel.sh | Detected | Detected |
| Playwright Stealth | Detected | Detected |
| Camoufox | Detected | Detected |
| Zyte | Detected | Detected |
| Ulixee Hero | Detected | Not detected |
| **Detected** | **8 / 8** | **5 / 8** |
The table reports detection in the September 2026 test. See the
[benchmark scope](https://www.centinelanalytica.com/compare/centinel-vs-cdn-bot-protection#september-2026-benchmark-scope) for the
tools and products compared.
[Download the published per-tool results (CSV)](https://www.centinelanalytica.com/research/2026-09-bot-detection.csv).
## When DataDome fits better
- **You want to start from a published plan.** DataDome lists its plans on its
[pricing page](https://datadome.co/pricing/), and prices some products by request volume.
Centinel does not publish prices; you get a quote after a scoping call.
- **You need native mobile SDKs.** DataDome documents a client-side JavaScript tag and iOS and
Android SDKs in its [getting-started guide](https://docs.datadome.co/docs/getting-started).
Centinel's integration guides do not include a native mobile SDK.
- **Your team already runs it well.** If DataDome covers your important routes and the traffic that
worries you, keep it. Add a second layer only where a test shows a gap.
## Where Centinel fits
Centinel fits when browser-like automation still reaches a route that matters, such as content
pages, sign-up, or an API used by your own front end. In the September test, Centinel detected
all eight tools, including Browser-use, Browserbase, and Ulixee Hero, which DataDome did not
detect.
It also fits teams that need to tell declared crawlers apart. Centinel separates declared and
verified crawlers from spoofed or hidden automation, so search engines and approved partners can
get a different policy than a scraper using their name. You can check one address with the
[crawler verification tool](https://www.centinelanalytica.com/verify-crawler).
## Feature comparison
| Area | DataDome (from its documentation) | Centinel |
| ------------------------ | ---------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------ |
| Server-side integrations | Modules for CDNs, edge platforms, web servers, and frameworks; CDN or edge recommended | 15 documented guides across edge and CDN, reverse proxy and server, application and CMS |
| Client-side collection | JavaScript tag, iOS and Android SDKs | Browser-side checks where the integration supports browser collection |
| Crawler identity | Confirm current crawler verification with DataDome | Separates declared and verified crawlers from spoofed ones |
| Responses | Confirm the responses available on your plan | Identify, observe, rate-limit, challenge, allow, or block, depending on the integration |
| Pricing model | Published plans; some products by request volume | Quote after scoping |
| EU and GDPR | Regional API endpoints include Frankfurt, Warsaw, and Paris | Data stored in the EU; ISO/IEC 27001 certified; assessed against SOC 2 Type 1 |
DataDome's server-side modules include Akamai EdgeWorkers, Apache, AWS CloudFront, Cloudflare,
Fastly, HAProxy, Nginx, Node.js, Next.js, and Varnish, among others, per its
[integration list](https://docs.datadome.co/docs/getting-started). Its regional endpoints are
listed in the [API server documentation](https://docs.datadome.co/docs/api-server). Centinel's
guides are on the [integrations page](https://www.centinelanalytica.com/integrations), and its security posture is in the
[trust center](https://trust.getkantis.com/centinel-analytica-gmbh).
## Setup effort and pricing model
Both products install at a similar point: a module on the CDN, edge, or server that asks for a
decision, plus browser-side collection. The effort depends on your stack more than on the vendor.
Plan for one engineer who can change the CDN or proxy configuration, the owner of the protected
workflow, and a rollback path.
For pricing, DataDome publishes its plans. Centinel quotes per deployment: bring your domain,
typical and peak traffic, and current stack. The [site audit](https://www.centinelanalytica.com/free-audit) is free and needs no
card.
## Migrating from DataDome
1. **Pick the routes.** List the routes where automation causes the most cost, and the automation
you want to keep, such as search crawlers and monitoring.
2. **Run side by side.** Add Centinel in observe mode on those routes while DataDome keeps
enforcing. Nothing changes for visitors yet.
3. **Compare decisions.** Review where the two products disagree, using the tools in the table
above and any workflow you can reproduce safely.
4. **Move enforcement route by route.** Switch one route to Centinel's policy, check
legitimate-user completion and error rates, then continue.
5. **Remove the old module last.** Keep DataDome in place until each route has run on the new
policy long enough to trust it, then remove its integration.
The [proof-of-concept scorecard](https://www.centinelanalytica.com/blog/bot-protection-proof-of-concept) turns these steps into a
written evaluation plan.
## FAQ
### Is Centinel a DataDome alternative?
Yes. Both detect automated traffic and return a decision your integration enforces. In the
September 2026 test, Centinel detected 8 of the 8 tools and DataDome detected 5.
### Can I run Centinel alongside DataDome?
Yes. Centinel works alongside an existing stack. Run it in observe mode next to DataDome, compare
the decisions, and decide from your own traffic.
### Does Centinel publish its prices?
No. Contact the team for current pricing, included usage, and contract terms. The site audit is
free.
### Where does Centinel store data?
Data is stored in the EU. Centinel is certified to ISO/IEC 27001 and assessed against SOC 2 Type 1.
Details are in the [trust center](https://trust.getkantis.com/centinel-analytica-gmbh).
## Related
- [DataDome alternatives compared](https://www.centinelanalytica.com/alternatives/datadome)
- [All comparisons](https://www.centinelanalytica.com/compare)
- [How to detect browser automation](https://www.centinelanalytica.com/blog/detect-browser-automation)
---
Page: https://www.centinelanalytica.com/compare/centinel-vs-fastly
# Centinel vs Fastly Bot Management: 2026 Detection Test
Centinel and Fastly Bot Management head to head: in our September 2026 test Centinel detected 8 of 8 scraping tools and Fastly 0. Fit, setup, and CSV.
By Frederick Jahn. Published 2026-09-01, updated 2026-09-30.
## Summary
| Compared on | Fastly Bot Management | Centinel |
| ----------------------------- | ------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------- |
| September 2026 test (8 tools) | 0 of 8 detected | 8 of 8 detected |
| Where it runs | Fastly's network, before cache, or after cache with Next-Gen WAF | Edge, CDN, reverse-proxy, server, and CMS integrations, including Fastly Compute and VCL |
| Buying model | A separately purchased Fastly product | Quote from the team after scoping; the site audit is free |
| Data location | Confirm with Fastly | Data stored in the EU |
Fastly Bot Management runs on Fastly's network. Centinel is a separate layer that runs on Fastly
Compute or VCL, or on another stack, so you can add it next to Fastly or switch to it.
## The benchmark
In September 2026 we ran eight scraping and browser-automation tools against Centinel and against Fastly: Browser-use, Firecrawl, Browserbase, kernel.sh, Playwright Stealth, Camoufox, Zyte, and Ulixee Hero.
Centinel detected all eight. Fastly detected 0 of 8.
## Per-tool detection results
| Scraping tool | Centinel | Fastly |
| ------------------ | --------- | ------------ |
| Browser-use | Detected | Not detected |
| Firecrawl | Detected | Not detected |
| Browserbase | Detected | Not detected |
| kernel.sh | Detected | Not detected |
| Playwright Stealth | Detected | Not detected |
| Camoufox | Detected | Not detected |
| Zyte | Detected | Not detected |
| Ulixee Hero | Detected | Not detected |
| **Detected** | **8 / 8** | **0 / 8** |
The table reports detection in the September 2026 test. See the
[benchmark scope](https://www.centinelanalytica.com/compare/centinel-vs-cdn-bot-protection#september-2026-benchmark-scope) for the
tools and products compared.
[Download the published per-tool results (CSV)](https://www.centinelanalytica.com/research/2026-09-bot-detection.csv).
## When Fastly fits better
- **Your traffic already runs through Fastly and you want one vendor.**
[Fastly Bot Management](https://www.fastly.com/documentation/guides/security/bot-management/about-bot-management/)
inspects requests on Fastly's network, before cache or after cache with Next-Gen WAF, and offers
client challenges and client-side detections.
- **You already run Next-Gen WAF.** Keeping bot and WAF rules in one product can be simpler to
operate. Check your platform first: Fastly documents that its Essential Next-Gen WAF platform
does not support Bot Management.
- **The result is good enough for your routes.** If the automation that costs you money is already
handled, keep it.
## Where Centinel fits
Centinel fits when browser-like automation still reaches a route that matters. In the September
test, Centinel detected all eight tools; Fastly detected none of them.
It also fits sites that are not fully on Fastly. Centinel has guides for Fastly Compute and Fastly
VCL and for other CDNs, proxies, servers, and CMSs, so one policy can cover a mixed stack. Centinel
separates declared and verified crawlers from spoofed or hidden automation; you can check one
address with the [crawler verification tool](https://www.centinelanalytica.com/verify-crawler).
## Feature comparison
| Area | Fastly Bot Management (from its documentation) | Centinel |
| ----------------- | --------------------------------------------------------------------- | ---------------------------------------------------------------------------------------- |
| Deployment | Fastly network, before cache or after cache with Next-Gen WAF | 15 documented guides, including Fastly Compute, Fastly VCL, and AWS CloudFront |
| Client-side | Client challenges and client-side detections | Browser-side checks where the integration supports browser collection |
| Crawler identity | Confirm current crawler verification with Fastly | Separates declared and verified crawlers from spoofed ones |
| Responses | Confirm the responses available on your plan | Identify, observe, rate-limit, challenge, allow, or block, depending on the integration |
| Pricing model | Purchased separately | Quote after scoping |
| EU and GDPR | Confirm data location with Fastly | Data stored in the EU; ISO/IEC 27001 certified; assessed against SOC 2 Type 1 |
Source: Fastly's
[About Bot Management guide](https://www.fastly.com/documentation/guides/security/bot-management/about-bot-management/).
Centinel's guides are on the [integrations page](https://www.centinelanalytica.com/integrations); its security posture is in the
[trust center](https://trust.getkantis.com/centinel-analytica-gmbh).
## Setup effort and pricing model
Fastly Bot Management is enabled on Fastly services once purchased. Centinel on Fastly runs on
Compute or VCL; on other stacks it is a module on the CDN, proxy, or server. Plan for one engineer
who can change that configuration, the owner of the protected workflow, and a rollback path.
Ask Fastly for its current pricing. Centinel quotes per deployment: bring your domain, typical and
peak traffic, and current stack. The [site audit](https://www.centinelanalytica.com/free-audit) is free and needs no card.
## Adding Centinel to Fastly, or moving off it
You do not have to leave Fastly to use Centinel. You can keep the CDN and add Centinel for the
routes where automation still gets through.
1. **Pick the routes.** List where automation costs the most and which automation you want to
keep, such as search crawlers and monitoring.
2. **Deploy in observe mode.** Add Centinel on Compute or VCL for those routes. Nothing changes for
visitors.
3. **Compare decisions.** Review where Centinel and your current Fastly rules disagree, using the
tools above and any workflow you can reproduce safely.
4. **Decide who enforces.** Give one product the final decision per route, so an allow in one
cannot silently override a block in the other.
5. **Switch route by route.** Move enforcement, check legitimate-user completion and error rates,
and keep a rollback.
The [proof-of-concept scorecard](https://www.centinelanalytica.com/blog/bot-protection-proof-of-concept) turns these steps into a
written evaluation plan.
## FAQ
### Is Centinel a Fastly Bot Management alternative?
Yes, and it can also run next to Fastly. In the September 2026 test, Centinel detected 8 of the 8
tools and Fastly detected 0.
### Do I have to leave Fastly?
No. Centinel has Fastly Compute and Fastly VCL integrations and works alongside your existing CDN
and WAF.
### Does Centinel publish its prices?
No. Contact the team for current pricing, included usage, and contract terms. The site audit is
free.
### Where does Centinel store data?
Data is stored in the EU. Centinel is certified to ISO/IEC 27001 and assessed against SOC 2 Type 1.
## Related
- [Centinel vs. CDN and WAF bot protection](https://www.centinelanalytica.com/compare/centinel-vs-cdn-bot-protection)
- [All comparisons](https://www.centinelanalytica.com/compare)
- [How to detect browser automation](https://www.centinelanalytica.com/blog/detect-browser-automation)
---
Page: https://www.centinelanalytica.com/compare/centinel-vs-human-security
# Centinel vs HUMAN Security: 2026 Detection Test
Centinel and HUMAN Security (PerimeterX) head to head: in our September 2026 test Centinel detected 8 of 8 scraping tools and HUMAN 2. Setup and CSV.
By Frederick Jahn. Published 2026-09-01, updated 2026-09-30.
## Summary
| Compared on | HUMAN Security | Centinel |
| ----------------------------- | --------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------- |
| September 2026 test (8 tools) | 2 of 8 detected | 8 of 8 detected |
| Where it runs | JavaScript sensor, cloud detector, and an enforcer on the app, load balancer, or CDN | Edge, CDN, reverse-proxy, server, and CMS integrations with browser-side checks |
| Buying model | Application Protection licensed by requests per month | Quote from the team after scoping; the site audit is free |
| Data location | Confirm with HUMAN | Data stored in the EU |
HUMAN Security merged with PerimeterX in
[July 2022](https://www.humansecurity.com/newsroom/human-and-perimeterx-unite-in-market-changing-merger-to-safeguard-customers-from-sophisticated-bot-attacks-fraud-and-account-abuse/),
and its bot product is Bot Defender.
Comparing more than one vendor? See [HUMAN Security alternatives](https://www.centinelanalytica.com/alternatives/human-security).
## The benchmark
In September 2026 we ran eight scraping and browser-automation tools against Centinel and against HUMAN Security: Browser-use, Firecrawl, Browserbase, kernel.sh, Playwright Stealth, Camoufox, Zyte, and Ulixee Hero.
Centinel detected all eight. HUMAN Security detected 2 of 8.
## Per-tool detection results
| Scraping tool | Centinel | HUMAN Security |
| ------------------ | --------- | -------------- |
| Browser-use | Detected | Not detected |
| Firecrawl | Detected | Not detected |
| Browserbase | Detected | Detected |
| kernel.sh | Detected | Not detected |
| Playwright Stealth | Detected | Detected |
| Camoufox | Detected | Not detected |
| Zyte | Detected | Not detected |
| Ulixee Hero | Detected | Not detected |
| **Detected** | **8 / 8** | **2 / 8** |
The table reports detection in the September 2026 test. See the
[benchmark scope](https://www.centinelanalytica.com/compare/centinel-vs-cdn-bot-protection#september-2026-benchmark-scope) for the
tools and products compared.
[Download the published per-tool results (CSV)](https://www.centinelanalytica.com/research/2026-09-bot-detection.csv).
## When HUMAN fits better
- **You need a prebuilt enforcer for an uncommon platform.** HUMAN documents
[over forty pre-built integrations](https://docs.humansecurity.com/applications/bd-overview) for
its enforcer. Check whether your platform is on that list and on Centinel's.
- **You want to license by request volume.** HUMAN licenses Application Protection by
[requests per month](https://docs.humansecurity.com/applications/application-protection-pricing).
Centinel quotes after a scoping call.
- **Your team already runs it well.** If Bot Defender covers the routes and automation that cost
you, keep it. Add a second layer only where a test shows a gap.
## Where Centinel fits
Centinel fits when browser-like automation still reaches a route that matters. In the September
test, Centinel detected all eight tools, including Browser-use, Firecrawl, kernel.sh, Camoufox,
Zyte, and Ulixee Hero, which HUMAN did not detect.
It also fits teams that need to tell declared crawlers apart. Centinel separates declared and
verified crawlers from spoofed or hidden automation, so search engines and approved partners can
get a different policy than a scraper using their name. You can check one address with the
[crawler verification tool](https://www.centinelanalytica.com/verify-crawler).
## Feature comparison
| Area | HUMAN Bot Defender (from its documentation) | Centinel |
| ---------------------- | -------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------- |
| Architecture | Sensor (JavaScript snippet), Detector (cloud machine-learning risk score), Enforcer (module) | Integration module plus browser-side checks where browser collection is supported |
| Integrations | Over forty pre-built enforcer integrations | 15 documented guides across edge and CDN, reverse proxy and server, application and CMS |
| Challenge | HUMAN Challenge | Challenge where the integration and policy support it |
| Crawler identity | Documents handling of known bots and crawlers | Separates declared and verified crawlers from spoofed ones |
| Responses | Confirm the responses available on your plan | Identify, observe, rate-limit, challenge, allow, or block, depending on the integration |
| Pricing model | Requests per month | Quote after scoping |
| EU and GDPR | Confirm data location with HUMAN | Data stored in the EU; ISO/IEC 27001 certified; assessed against SOC 2 Type 1 |
Sources: HUMAN's [Bot Defender overview](https://docs.humansecurity.com/applications/bd-overview)
and [pricing documentation](https://docs.humansecurity.com/applications/application-protection-pricing).
Centinel's guides are on the [integrations page](https://www.centinelanalytica.com/integrations); its security posture is in the
[trust center](https://trust.getkantis.com/centinel-analytica-gmbh).
## Setup effort and pricing model
Both products install the same way in outline: browser-side collection plus a module on the app,
proxy, or CDN that enforces the decision. The effort depends on your stack more than on the vendor.
Plan for one engineer who can change that configuration, the owner of the protected workflow, and
a rollback path.
HUMAN licenses by requests per month. Centinel quotes per deployment: bring your domain, typical
and peak traffic, and current stack. The [site audit](https://www.centinelanalytica.com/free-audit) is free and needs no card.
## Migrating from HUMAN
1. **Pick the routes.** List where automation costs the most and which automation you want to
keep, such as search crawlers and monitoring.
2. **Run side by side.** Add Centinel in observe mode on those routes while HUMAN's enforcer keeps
enforcing. Nothing changes for visitors yet.
3. **Compare decisions.** Review where the two disagree, using the tools above and any workflow
you can reproduce safely.
4. **Move enforcement route by route.** Switch one route to Centinel's policy, check
legitimate-user completion and error rates, then continue.
5. **Remove the old enforcer and sensor last.** Keep HUMAN in place until each route has run on
the new policy long enough to trust it.
The [proof-of-concept scorecard](https://www.centinelanalytica.com/blog/bot-protection-proof-of-concept) turns these steps into a
written evaluation plan.
## FAQ
### Is Centinel a HUMAN Security or PerimeterX alternative?
Yes. PerimeterX merged into HUMAN in 2022, and HUMAN's bot product is Bot Defender. In the
September 2026 test, Centinel detected 8 of the 8 tools and HUMAN detected 2.
### Can I run Centinel alongside HUMAN?
Yes. Centinel works alongside an existing stack. Run it in observe mode next to HUMAN, compare the
decisions, and decide from your own traffic.
### Does Centinel publish its prices?
No. Contact the team for current pricing, included usage, and contract terms. The site audit is
free.
### Where does Centinel store data?
Data is stored in the EU. Centinel is certified to ISO/IEC 27001 and assessed against SOC 2 Type 1.
## Related
- [HUMAN Security alternatives compared](https://www.centinelanalytica.com/alternatives/human-security)
- [All comparisons](https://www.centinelanalytica.com/compare)
- [How to detect browser automation](https://www.centinelanalytica.com/blog/detect-browser-automation)
---
Page: https://www.centinelanalytica.com/compare/centinel-vs-kasada
# Centinel vs Kasada: 2026 Detection Test
Centinel and Kasada head to head: in our September 2026 test Centinel detected 8 of 8 scraping tools and Kasada 3. Fit, setup, migration, and CSV.
By Frederick Jahn. Published 2026-09-01, updated 2026-09-30.
## Summary
| Compared on | Kasada | Centinel |
| ----------------------------- | ---------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------- |
| September 2026 test (8 tools) | 3 of 8 detected | 8 of 8 detected |
| Where it runs | JavaScript tags, mobile SDKs, NPM packages for CDN edge functions, or a proxy | Edge, CDN, reverse-proxy, server, and CMS integrations with browser-side checks |
| Buying model | Confirm with Kasada | Quote from the team after scoping; the site audit is free |
| Data location | Confirm with Kasada | Data stored in the EU |
Both products detect automated traffic and let a site decide how to respond.
## The benchmark
In September 2026 we ran eight scraping and browser-automation tools against Centinel and against Kasada: Browser-use, Firecrawl, Browserbase, kernel.sh, Playwright Stealth, Camoufox, Zyte, and Ulixee Hero.
Centinel detected all eight. Kasada detected 3 of 8.
## Per-tool detection results
| Scraping tool | Centinel | Kasada |
| ------------------ | --------- | ------------ |
| Browser-use | Detected | Not detected |
| Firecrawl | Detected | Detected |
| Browserbase | Detected | Not detected |
| kernel.sh | Detected | Not detected |
| Playwright Stealth | Detected | Detected |
| Camoufox | Detected | Detected |
| Zyte | Detected | Not detected |
| Ulixee Hero | Detected | Not detected |
| **Detected** | **8 / 8** | **3 / 8** |
The table reports detection in the September 2026 test. See the
[benchmark scope](https://www.centinelanalytica.com/compare/centinel-vs-cdn-bot-protection#september-2026-benchmark-scope) for the
tools and products compared.
[Download the published per-tool results (CSV)](https://www.centinelanalytica.com/research/2026-09-bot-detection.csv).
## When Kasada fits better
- **You need native mobile SDKs.** Kasada documents mobile SDKs alongside its JavaScript tags on
its [integration page](https://www.kasada.io/integration). Centinel's integration guides do
not include a native mobile SDK.
- **You want a no-code start.** Kasada offers a proxy integration that needs no server-side code,
only a routing change. Centinel installs as a module on your CDN, proxy, server, or CMS.
- **Your team already runs it well.** If Kasada covers the routes and automation that cost you,
keep it. Add a second layer only where a test shows a gap.
## Where Centinel fits
Centinel fits when browser-like automation still reaches a route that matters. In the September
test, Centinel detected all eight tools, including Browser-use, Browserbase, kernel.sh, Zyte, and
Ulixee Hero, which Kasada did not detect.
It also fits teams that need to tell declared crawlers apart. Centinel separates declared and
verified crawlers from spoofed or hidden automation, so search engines and approved partners can
get a different policy than a scraper using their name. You can check one address with the
[crawler verification tool](https://www.centinelanalytica.com/verify-crawler).
## Feature comparison
| Area | Kasada (from its integration page) | Centinel |
| ----------------- | ------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------- |
| Deployment | JavaScript tags, NPM packages for CDN edge functions, or a proxy | 15 documented guides across edge and CDN, reverse proxy and server, application and CMS |
| Mobile apps | Mobile SDKs | No native mobile SDK in the integration guides |
| Crawler identity | Confirm current crawler verification with Kasada | Separates declared and verified crawlers from spoofed ones |
| Responses | Confirm the responses available on your plan | Identify, observe, rate-limit, challenge, allow, or block, depending on the integration |
| Pricing model | Confirm with Kasada | Quote after scoping |
| EU and GDPR | Confirm data location with Kasada | Data stored in the EU; ISO/IEC 27001 certified; assessed against SOC 2 Type 1 |
Source: Kasada's [integration page](https://www.kasada.io/integration). Centinel's guides are on the
[integrations page](https://www.centinelanalytica.com/integrations); its security posture is in the
[trust center](https://trust.getkantis.com/centinel-analytica-gmbh).
## Setup effort and pricing model
Kasada's proxy needs only a routing change, not server-side code; its tags and edge packages need
code or CDN changes. Centinel is a module on the CDN, proxy, server, or CMS. Plan for one engineer
who can change that configuration, the owner of the protected workflow, and a rollback path.
Centinel quotes per deployment: bring your domain, typical and peak traffic, and current stack.
The [site audit](https://www.centinelanalytica.com/free-audit) is free and needs no card.
## Migrating from Kasada
1. **Pick the routes.** List where automation costs the most and which automation you want to
keep, such as search crawlers and monitoring.
2. **Run side by side.** Add Centinel in observe mode on those routes while Kasada keeps
enforcing. Nothing changes for visitors yet.
3. **Compare decisions.** Review where the two disagree, using the tools above and any workflow
you can reproduce safely.
4. **Move enforcement route by route.** Switch one route to Centinel's policy, check
legitimate-user completion and error rates, then continue.
5. **Remove the old integration last.** Keep Kasada in place until each route has run on the new
policy long enough to trust it.
The [proof-of-concept scorecard](https://www.centinelanalytica.com/blog/bot-protection-proof-of-concept) turns these steps into a
written evaluation plan.
## FAQ
### Is Centinel a Kasada alternative?
Yes. Both detect automated traffic and return a decision your integration enforces. In the
September 2026 test, Centinel detected 8 of the 8 tools and Kasada detected 3.
### Can I run Centinel alongside Kasada?
Yes. Centinel works alongside an existing stack. Run it in observe mode next to Kasada, compare the
decisions, and decide from your own traffic.
### Does Centinel publish its prices?
No. Contact the team for current pricing, included usage, and contract terms. The site audit is
free.
### Where does Centinel store data?
Data is stored in the EU. Centinel is certified to ISO/IEC 27001 and assessed against SOC 2 Type 1.
## Related
- [All comparisons](https://www.centinelanalytica.com/compare)
- [How to detect browser automation](https://www.centinelanalytica.com/blog/detect-browser-automation)
---
Page: https://www.centinelanalytica.com/compare/centinel-vs-cdn-bot-protection
# Centinel vs. CDN and WAF bot protection
A practical method for deciding whether Centinel adds useful automation evidence to the CDN, WAF, and bot controls you already operate.
By Frederick Jahn. Published 2026-07-22, updated 2026-09-15.
A CDN, WAF, bot-management product, and Centinel are not interchangeable labels. They can overlap, and the capabilities available from an existing provider may depend on the plan, configuration, traffic path, and data collected.
The decision is therefore not “WAF or Centinel.” It is whether the configured stack already makes the automation decisions you need with acceptable error, coverage, latency, and operating cost.
## Start with the job of each layer
The [OWASP description of a web application firewall](https://owasp.org/www-community/Web_Application_Firewall) places a WAF between a client and a web application to inspect application-layer communication. Many CDN and WAF providers offer additional bot features, but the category name does not reveal which features are enabled in a particular deployment.
[HTTP semantics](https://www.rfc-editor.org/rfc/rfc9110) define requests and responses, not requester intent. A browser-like request can be generated by a person, approved automation, testing software, or an abusive workflow. A useful bot decision needs enough context to separate the cases relevant to the protected route.
Centinel is designed to contribute browser, network, crawler, and session evidence to that decision. What it can observe and enforce depends on the integration. Browser collection applies only where its code runs; server or edge controls must sit on the traffic path they are expected to govern.
## Use this evaluation matrix
Fill this table with observed behavior from the actual products and plans under evaluation.
| Decision area | Existing CDN, WAF, or bot controls | Centinel evidence to validate |
| ------------------ | ------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- |
| Traffic coverage | Which hosts, routes, methods, APIs, and direct-origin paths pass through the control? | Which of the same paths are observed and which component can enforce? |
| Declared crawlers | How are product tokens and operator identity verified? | Which operator sources are used, how often are they refreshed, and what happens when verification is unresolved? |
| Browser automation | Which runtime, transport, and consistency signals are available? | Which supported integrations collect browser and network evidence? |
| Session context | How are requests correlated, for how long, and across which identifiers? | Which session behaviors affect classification, with what bounds and retention? |
| Responses | Which routes support observe, rate-limit, challenge, deny, or allow decisions? | Which responses are supported at the chosen integration point? |
| Failure behavior | What happens when a signal source or policy service is unavailable? | Is each dependency bounded, and does the route fail open, closed, or to a fallback? |
| Error review | How are false positives and false negatives labeled and corrected? | What evidence, reason code, holdout, and appeal path are available? |
| Operations | Who owns rule changes, detection updates, and incident response? | Which updates are managed and which remain the customer's responsibility? |
| Data handling | What is collected, where is it processed, and how long is it retained? | Does the proposed deployment meet the same privacy and security requirements? |
Do not fill the left column from a generic product page. Inspect the current account and enabled controls. Do not fill the right column from this article. Confirm it in product documentation, a technical review, and a representative test.
## Run a controlled comparison
Use the same target behavior and ground truth for every configuration.
1. Choose a small set of important routes and define the unwanted automated action.
2. Build labeled cohorts: ordinary supported browsers, important accessibility and privacy configurations, approved automation, named crawlers, stock browser automation, and the specific abusive workflow you can reproduce safely.
3. Record the current stack, plan, enabled rules, integration, and software versions.
4. Run the cohorts against the current controls in a non-destructive environment or bounded production observation.
5. Add Centinel in observation mode and repeat without silently changing the other controls.
6. Compare route coverage, decision reason, false positive and false negative behavior, latency, user-flow completion, and origin work.
7. Keep raw observations and document limitations, exclusions, and failed test runs.
A vendor score without the cohort, configuration, and expected outcome is not a comparison. It is a result that cannot be reproduced.
For a named crawler cohort, use current operator documentation. [OpenAI's crawler reference](https://developers.openai.com/api/docs/bots), for example, separates its crawler purposes and publishes verification data. A copied user-agent string should not pass a trusted-crawler test.
## Interpret overlap correctly
More signals do not automatically produce a better decision. Two products may rely on correlated evidence, making the combined system look layered while repeating the same assumption. Conversely, a narrow additional signal may be useful if it covers a known blind spot on a high-impact route.
Test the combined path as a system:
- Which component makes the final decision?
- Can one component's allow override another component's deny?
- Are challenge and rate-limit responses duplicated?
- Does a retry create a loop between layers?
- Can operators see why the final response occurred?
- Is rollback possible without disabling unrelated security controls?
Prefer one clear owner for each decision. Use the other layers as bounded evidence providers or enforcement points.
## When Centinel may add value
Centinel is worth a deeper evaluation when the current stack has a measured gap involving browser-like automation, residential or rotating networks, crawler verification, or behavior spread across a session, and the proposed integration can observe that traffic.
The evidence should change a decision. If an additional classification never changes observe, allow, rate-limit, challenge, or deny behavior on a relevant route, it adds operational complexity without policy value.
The supporting article on [browser-like crawlers and WAF rules](https://www.centinelanalytica.com/blog/why-your-waf-cant-see-them) explains why request and session scopes need to be tested separately.
## When you may not need another layer
Do not add Centinel merely to increase the number of security products. You may not need another layer when:
- the existing deployment covers every relevant traffic path;
- it meets the agreed detection and false-positive thresholds on representative cohorts;
- its response options fit the resource policy;
- operators can explain, change, and roll back decisions;
- the remaining automation risk is lower than the integration and operating cost.
You should also defer an integration when there is no owner for policy, ground truth, review, or incident response. Another score does not repair an undefined decision process.
## Make the decision from evidence
Keep the CDN and WAF capabilities that already work. Add a specialist layer only where a controlled test shows a meaningful gap and the integration closes it without unacceptable regressions.
The [bot management overview](https://www.centinelanalytica.com/bot-management) provides a broader requirements checklist. Use it to turn product claims into test cases before selecting an architecture.
## Verified vendor benchmarks
The following pages report the supplied September 2026 benchmark results and per-tool detection outcomes.
- [Centinel vs. Akamai](https://www.centinelanalytica.com/compare/centinel-vs-akamai)
- [Centinel vs. Cloudflare](https://www.centinelanalytica.com/compare/centinel-vs-cloudflare)
- [Centinel vs. DataDome](https://www.centinelanalytica.com/compare/centinel-vs-datadome)
- [Centinel vs. Fastly](https://www.centinelanalytica.com/compare/centinel-vs-fastly)
- [Centinel vs. HUMAN Security](https://www.centinelanalytica.com/compare/centinel-vs-human-security)
- [Centinel vs. Kasada](https://www.centinelanalytica.com/compare/centinel-vs-kasada)
September 2026 benchmark results
Centinel's September 2026 benchmark reports detection outcomes for eight named
scraping and browser-automation tools. The results cover Centinel, Akamai,
Cloudflare, DataDome, Fastly, HUMAN Security, and Kasada. They are Centinel's
observations from the tested configurations.
| Benchmark | Details |
| ----------- | --------------------------------------------------------------------------------------------------- |
| Test period | September 2026 |
| Tools | Browser-use, Firecrawl, Browserbase, kernel.sh, Playwright Stealth, Camoufox, Zyte, and Ulixee Hero |
| Results | Per-tool detection outcomes and detected-tool totals for each product |
[Download the September 2026 results (CSV)](https://www.centinelanalytica.com/research/2026-09-bot-detection.csv).
The file contains the same per-tool outcomes as the linked comparisons. Each
cell records whether the product detected that tool in the benchmark.
Use the [proof-of-concept scorecard](https://www.centinelanalytica.com/blog/bot-protection-proof-of-concept) to
plan an evaluation for your own traffic and deployment.
---
Page: https://www.centinelanalytica.com/alternatives/cloudflare
# Cloudflare Bot Management Alternatives, Tested
Cloudflare Bot Management alternatives compared on one September 2026 test of eight scraping tools, and how to add one without leaving Cloudflare.
By Frederick Jahn. Published 2026-09-30.
## Short answer
The alternatives to Cloudflare Bot Management are the specialist products Centinel, DataDome,
Kasada, and HUMAN Security, and the CDN-bound products from Akamai and Fastly. In our September
2026 test of eight scraping and browser-automation tools, Centinel detected all eight, DataDome
five, and Cloudflare three.
You do not have to leave Cloudflare to add one. A specialist product can run on Cloudflare Workers
or behind Cloudflare, next to the CDN you already use. Test any option on your own routes before
you change enforcement.
## September 2026 results
We ran the same eight tools against each product: Browser-use, Firecrawl, Browserbase, kernel.sh,
Playwright Stealth, Camoufox, Zyte, and Ulixee Hero.
| Product | Tools detected |
| -------------- | -------------- |
| Centinel | 8 / 8 |
| DataDome | 5 / 8 |
| Cloudflare | 3 / 8 |
| Kasada | 3 / 8 |
| Akamai | 2 / 8 |
| HUMAN Security | 2 / 8 |
| Fastly | 0 / 8 |
[Download the per-tool results (CSV)](https://www.centinelanalytica.com/research/2026-09-bot-detection.csv). The
[benchmark scope](https://www.centinelanalytica.com/compare/centinel-vs-cdn-bot-protection#september-2026-benchmark-scope) lists
what was tested.
## Which Cloudflare bot product do you have?
Cloudflare's bot features depend on the plan
([plans overview](https://developers.cloudflare.com/bots/plans/)):
- **Free:** [Bot Fight Mode](https://developers.cloudflare.com/bots/plans/free/) challenges simple
bots from cloud hosting providers and headless browsers across the whole domain.
- **Pro, Business, and Enterprise:**
[Super Bot Fight Mode](https://developers.cloudflare.com/bots/plans/biz-and-ent/) lets you
allow, block, or challenge detected bots.
- **Enterprise add-on:**
[Bot Management](https://developers.cloudflare.com/bots/get-started/bot-management/) gives each
request a [bot score](https://developers.cloudflare.com/bots/concepts/bot-score/) from 1 to 99
for WAF custom rules or Workers, plus Bot Analytics.
If you are on Free or Pro, the next step may be a higher Cloudflare tier rather than another vendor.
Compare both on the same traffic.
## What to check in an alternative
- **Detection on your traffic.** Rerun the automation you care about against the candidate, on the
routes where it costs you.
- **Whether it runs on Cloudflare.** A product that runs as a Worker, or behind Cloudflare at your
origin, lets you keep the CDN.
- **Who has the final say.** With two layers, decide which one enforces on each route so an allow
in one cannot override a block in the other.
- **Crawler handling.** Cloudflare
[groups verified bots by category](https://developers.cloudflare.com/bots/concepts/bot/verified-bots/).
Compare how each candidate treats the crawlers you want to allow.
- **Data location.** Cloudflare's
[Data Localization Suite](https://developers.cloudflare.com/data-localization/) is an Enterprise
add-on. Ask each candidate where data is processed and stored.
## The options
### Centinel
A specialist bot-management layer with a Cloudflare Workers integration and 14 other guides across
edge, proxy, server, and CMS. It separates declared and verified crawlers from spoofed ones. Data
is stored in the EU; ISO/IEC 27001 certified and assessed against SOC 2 Type 1. Priced by quote.
Detected 8 of 8. [Centinel vs Cloudflare](https://www.centinelanalytica.com/compare/centinel-vs-cloudflare).
### DataDome
Server-side modules, with a CDN or edge integration recommended (Cloudflare is on its list), plus a
JavaScript tag and mobile SDKs ([getting started](https://docs.datadome.co/docs/getting-started)).
Publishes plans on its [pricing page](https://datadome.co/pricing/). Detected 5 of 8.
[Centinel vs DataDome](https://www.centinelanalytica.com/compare/centinel-vs-datadome).
### Kasada
JavaScript tags, mobile SDKs, NPM packages for CDN edge functions, or a proxy that needs no
server-side code ([integration](https://www.kasada.io/integration)). Detected 3 of 8.
[Centinel vs Kasada](https://www.centinelanalytica.com/compare/centinel-vs-kasada).
### HUMAN Security
Bot Defender uses a JavaScript sensor, a cloud detector, and an enforcer on your app, load
balancer, or CDN ([overview](https://docs.humansecurity.com/applications/bd-overview)). Licensed by
requests per month. Detected 2 of 8. [Centinel vs HUMAN](https://www.centinelanalytica.com/compare/centinel-vs-human-security).
### Akamai or Fastly
Both sell bot products tied to their own CDN
([Akamai Bot Manager](https://www.akamai.com/products/bot-manager),
[Fastly Bot Management](https://www.fastly.com/documentation/guides/security/bot-management/about-bot-management/)),
so choosing one means moving CDN. Akamai detected 2 of 8 and Fastly 0 of 8.
[Centinel vs Akamai](https://www.centinelanalytica.com/compare/centinel-vs-akamai),
[Centinel vs Fastly](https://www.centinelanalytica.com/compare/centinel-vs-fastly).
## When staying with Cloudflare alone makes sense
Stay if Cloudflare already handles the automation that costs you, or if you want one vendor and one
dashboard. A second layer adds an integration and policies to maintain. It is worth it only when a
test on your traffic shows a gap it closes.
## How to add an alternative
Deploy it in observe mode on the routes that matter, compare its decisions with Cloudflare's, then
choose which layer enforces on each route. The
[Centinel vs Cloudflare](https://www.centinelanalytica.com/compare/centinel-vs-cloudflare) page lists the steps.
## FAQ
### What is the best Cloudflare Bot Management alternative?
The one that detects the automation you care about on your own routes. In our September 2026 test,
Centinel detected all eight tools and Cloudflare detected three.
### Can I use an alternative and keep Cloudflare as my CDN?
Yes. Centinel runs as a Cloudflare Worker, DataDome lists a Cloudflare integration, and Kasada
offers packages for CDN edge functions. Keep one layer as the final decision per route.
### Is Super Bot Fight Mode enough?
It lets you allow, block, or challenge detected bots on Pro, Business, and Enterprise plans. Test it
against the automation you see before deciding it is enough.
## Related
- [All comparisons](https://www.centinelanalytica.com/compare)
- [Centinel vs. CDN and WAF bot protection](https://www.centinelanalytica.com/compare/centinel-vs-cdn-bot-protection)
---
Page: https://www.centinelanalytica.com/alternatives/datadome
# DataDome Alternatives: 2026 Detection Test
DataDome alternatives compared on one September 2026 test of eight scraping tools, with deployment, buying model, and EU data notes for each option.
By Frederick Jahn. Published 2026-09-30.
## Short answer
If DataDome misses automation that matters to you, the options are another specialist bot
product (Centinel, HUMAN, or Kasada) or the bot features of your CDN (Cloudflare, Akamai, or
Fastly). In our September 2026 test of eight scraping and browser-automation tools, Centinel
detected all eight and DataDome detected five. Test any option on your own routes before you
switch.
## September 2026 results
We ran the same eight tools against each product: Browser-use, Firecrawl, Browserbase, kernel.sh,
Playwright Stealth, Camoufox, Zyte, and Ulixee Hero.
| Product | Tools detected |
| -------------- | -------------- |
| Centinel | 8 / 8 |
| DataDome | 5 / 8 |
| Cloudflare | 3 / 8 |
| Kasada | 3 / 8 |
| Akamai | 2 / 8 |
| HUMAN Security | 2 / 8 |
| Fastly | 0 / 8 |
[Download the per-tool results (CSV)](https://www.centinelanalytica.com/research/2026-09-bot-detection.csv). The
[benchmark scope](https://www.centinelanalytica.com/compare/centinel-vs-cdn-bot-protection#september-2026-benchmark-scope) lists
what was tested.
## What to check in a DataDome alternative
- **Detection on your traffic.** Rerun the automation you care about against the candidate, on the
routes where it costs you.
- **Where it installs.** DataDome recommends a CDN or edge integration and also offers a JavaScript
tag and mobile SDKs ([getting started](https://docs.datadome.co/docs/getting-started)). Check the
candidate supports the same points in your stack.
- **Crawler handling.** Search engines and approved partners should get a different policy than a
scraper using their name.
- **Buying model.** DataDome publishes plans on its [pricing page](https://datadome.co/pricing/).
- **Data location.** DataDome lists regional API endpoints, including Europe
([API server](https://docs.datadome.co/docs/api-server)). Ask each candidate where data is
processed and stored.
## The options
### Centinel
A specialist bot-management layer that works alongside your CDN and WAF, with 15 integration
guides across edge, proxy, server, and CMS. It separates declared and verified crawlers from
spoofed ones. Data is stored in the EU; ISO/IEC 27001 certified and assessed against SOC 2 Type 1.
Priced by quote. Detected 8 of 8. [Centinel vs DataDome](https://www.centinelanalytica.com/compare/centinel-vs-datadome).
### HUMAN Security
HUMAN's Bot Defender uses a JavaScript sensor, a cloud detector, and an enforcer on your app, load
balancer, or CDN, with over forty prebuilt integrations
([overview](https://docs.humansecurity.com/applications/bd-overview)). Licensed by requests per
month. Detected 2 of 8. [Centinel vs HUMAN](https://www.centinelanalytica.com/compare/centinel-vs-human-security).
### Kasada
JavaScript tags, mobile SDKs, NPM packages for CDN edge functions, or a proxy that needs no
server-side code ([integration](https://www.kasada.io/integration)). Detected 3 of 8.
[Centinel vs Kasada](https://www.centinelanalytica.com/compare/centinel-vs-kasada).
### Cloudflare
Bot features come with every Cloudflare plan, from Bot Fight Mode on Free to Bot Management as an
Enterprise add-on ([plans](https://developers.cloudflare.com/bots/plans/)). Traffic must be proxied
through Cloudflare. Detected 3 of 8. [Centinel vs Cloudflare](https://www.centinelanalytica.com/compare/centinel-vs-cloudflare).
### Akamai Bot Manager
Assigns a bot score at Akamai's edge with Cautious, Strict, and Aggressive response segments
([product page](https://www.akamai.com/products/bot-manager)). Detected 2 of 8.
[Centinel vs Akamai](https://www.centinelanalytica.com/compare/centinel-vs-akamai).
### Fastly Bot Management
A purchased product that inspects requests on Fastly's network, before cache or after cache with
Next-Gen WAF
([guide](https://www.fastly.com/documentation/guides/security/bot-management/about-bot-management/)).
Detected 0 of 8. [Centinel vs Fastly](https://www.centinelanalytica.com/compare/centinel-vs-fastly).
## When staying with DataDome makes sense
Stay if DataDome already covers the routes and automation that cost you, if you rely on its mobile
SDKs, or if its published plans suit your budget process. A switch has a cost: a new integration,
new policies, and a period of running two products. It is worth it only when a test on your traffic
shows a gap the alternative closes.
## How to switch without a gap
Run the alternative in observe mode next to DataDome, compare decisions on the same routes, then
move enforcement one route at a time and remove DataDome last. The
[Centinel vs DataDome](https://www.centinelanalytica.com/compare/centinel-vs-datadome) page lists the steps.
## FAQ
### What is the best DataDome alternative?
The one that detects the automation you care about on your own routes. In our September 2026 test,
Centinel detected all eight tools and no other product detected more than DataDome's five.
### Can I try an alternative without removing DataDome?
Yes. Run the alternative in observe mode next to DataDome and compare decisions before you change
enforcement.
### Which alternatives store data in the EU?
Centinel stores data in the EU. For the others, ask the vendor where data is processed and stored;
Cloudflare sells its Data Localization Suite as an Enterprise add-on.
## Related
- [All comparisons](https://www.centinelanalytica.com/compare)
- [Bot protection proof-of-concept scorecard](https://www.centinelanalytica.com/blog/bot-protection-proof-of-concept)
---
Page: https://www.centinelanalytica.com/alternatives/human-security
# HUMAN Security Alternatives: 2026 Detection Test
HUMAN Security (PerimeterX) alternatives compared on one September 2026 test of eight scraping tools, with deployment and buying notes for each option.
By Frederick Jahn. Published 2026-09-30.
## Short answer
If HUMAN's Bot Defender misses automation that matters to you, the options are another specialist
bot product (Centinel, DataDome, or Kasada) or the bot features of your CDN (Cloudflare, Akamai, or
Fastly). In our September 2026 test of eight scraping and browser-automation tools, Centinel
detected all eight and HUMAN Security detected two. Test any option on your own routes before you
switch.
HUMAN merged with PerimeterX in
[July 2022](https://www.humansecurity.com/newsroom/human-and-perimeterx-unite-in-market-changing-merger-to-safeguard-customers-from-sophisticated-bot-attacks-fraud-and-account-abuse/),
so a PerimeterX deployment is now a HUMAN one and the same options apply.
## September 2026 results
We ran the same eight tools against each product: Browser-use, Firecrawl, Browserbase, kernel.sh,
Playwright Stealth, Camoufox, Zyte, and Ulixee Hero.
| Product | Tools detected |
| -------------- | -------------- |
| Centinel | 8 / 8 |
| DataDome | 5 / 8 |
| Cloudflare | 3 / 8 |
| Kasada | 3 / 8 |
| Akamai | 2 / 8 |
| HUMAN Security | 2 / 8 |
| Fastly | 0 / 8 |
[Download the per-tool results (CSV)](https://www.centinelanalytica.com/research/2026-09-bot-detection.csv). The
[benchmark scope](https://www.centinelanalytica.com/compare/centinel-vs-cdn-bot-protection#september-2026-benchmark-scope) lists
what was tested.
## What to check in a HUMAN alternative
- **Detection on your traffic.** Rerun the automation you care about against the candidate, on the
routes where it costs you.
- **Where the enforcer sits.** HUMAN's enforcer runs on your app, load balancer, or CDN, with over
forty prebuilt integrations
([Bot Defender overview](https://docs.humansecurity.com/applications/bd-overview)). Check the
candidate supports the same point in your stack.
- **Crawler handling.** Search engines and approved partners should get a different policy than a
scraper using their name.
- **Buying model.** HUMAN licenses Application Protection by
[requests per month](https://docs.humansecurity.com/applications/application-protection-pricing).
Compare on the same traffic volume.
- **Data location.** Ask each candidate where data is processed and stored.
## The options
### Centinel
A specialist bot-management layer that works alongside your CDN and WAF, with 15 integration
guides across edge, proxy, server, and CMS. It separates declared and verified crawlers from
spoofed ones. Data is stored in the EU; ISO/IEC 27001 certified and assessed against SOC 2 Type 1.
Priced by quote. Detected 8 of 8. [Centinel vs HUMAN](https://www.centinelanalytica.com/compare/centinel-vs-human-security).
### DataDome
Server-side modules, with a CDN or edge integration recommended, plus a JavaScript tag and mobile
SDKs ([getting started](https://docs.datadome.co/docs/getting-started)). Publishes plans on its
[pricing page](https://datadome.co/pricing/). Detected 5 of 8.
[Centinel vs DataDome](https://www.centinelanalytica.com/compare/centinel-vs-datadome).
### Kasada
JavaScript tags, mobile SDKs, NPM packages for CDN edge functions, or a proxy that needs no
server-side code ([integration](https://www.kasada.io/integration)). Detected 3 of 8.
[Centinel vs Kasada](https://www.centinelanalytica.com/compare/centinel-vs-kasada).
### Cloudflare
Bot features come with every Cloudflare plan, from Bot Fight Mode on Free to Bot Management as an
Enterprise add-on ([plans](https://developers.cloudflare.com/bots/plans/)). Traffic must be proxied
through Cloudflare. Detected 3 of 8. [Centinel vs Cloudflare](https://www.centinelanalytica.com/compare/centinel-vs-cloudflare).
### Akamai Bot Manager
Assigns a bot score at Akamai's edge with Cautious, Strict, and Aggressive response segments
([product page](https://www.akamai.com/products/bot-manager)). Detected 2 of 8.
[Centinel vs Akamai](https://www.centinelanalytica.com/compare/centinel-vs-akamai).
### Fastly Bot Management
A purchased product that inspects requests on Fastly's network, before cache or after cache with
Next-Gen WAF
([guide](https://www.fastly.com/documentation/guides/security/bot-management/about-bot-management/)).
Detected 0 of 8. [Centinel vs Fastly](https://www.centinelanalytica.com/compare/centinel-vs-fastly).
## When staying with HUMAN makes sense
Stay if Bot Defender already covers the routes and automation that cost you, or if your platform
depends on one of HUMAN's prebuilt enforcers that a candidate does not offer. A switch has a cost:
a new integration, new policies, and a period of running two products. It is worth it only when a
test on your traffic shows a gap the alternative closes.
## How to switch without a gap
Run the alternative in observe mode next to HUMAN, compare decisions on the same routes, then move
enforcement one route at a time and remove HUMAN's enforcer and sensor last. The
[Centinel vs HUMAN](https://www.centinelanalytica.com/compare/centinel-vs-human-security) page lists the steps.
## FAQ
### What is the best HUMAN Security alternative?
The one that detects the automation you care about on your own routes. In our September 2026 test,
Centinel detected all eight tools and HUMAN Security detected two.
### Is PerimeterX the same as HUMAN?
HUMAN and PerimeterX merged in 2022, and HUMAN's bot product is Bot Defender.
### Can I try an alternative without removing HUMAN?
Yes. Run the alternative in observe mode next to HUMAN and compare decisions before you change
enforcement.
## Related
- [All comparisons](https://www.centinelanalytica.com/compare)
- [Bot protection proof-of-concept scorecard](https://www.centinelanalytica.com/blog/bot-protection-proof-of-concept)