Sep 30, 2026
Fake Googlebot: how to spot an impersonator
Any client can send a Googlebot user agent. Verify the IP with reverse and forward DNS or Google's published ranges, and avoid the Google Cloud trap.
Frederick Jahn
AI crawlers and agents
Who operates each crawler, how to verify a request that claims to be one, and how to turn an access decision into a policy your site enforces.
Sep 30, 2026
Pay per crawl explained: HTTP 402 and TollBit
Pay per crawl charges AI crawlers per request. How Cloudflare, TollBit, RSL and x402 work, and why it depends on knowing which crawler is asking.
Frederick Jahn
Sep 22, 2026
Fake Googlebot and AI crawlers: 2026 data
On the typical news site, 1 in 7 requests claiming to be Googlebot failed verification, and 84% of GPTBot claims were fake. Data, method and CSV download.
Frederick Jahn
Sep 5, 2026
How to audit AI crawler traffic on your website
Build a crawler inventory from request logs. Separate claimed names, verified operators, resource use, and policy outcomes before changing crawler access.
Frederick Jahn
Apr 20, 2026
How to control AI agent access
Build a practical access policy for AI crawlers and agents with a resource matrix, explicit identity states, bounded responses, and tested failure behavior.
Frederick Jahn
Apr 20, 2026
Verify AI crawlers and agents: spot fake GPTBot
Check whether a request claiming to be GPTBot, ClaudeBot or another AI agent is real, using published IP ranges, reverse DNS or signed requests.
Frederick Jahn
Apr 20, 2026
What is agentic traffic? AI agents explained
Agentic traffic comes from software that picks and performs steps toward a user's goal. How it differs from crawlers and fetchers, and how to measure it.
Frederick Jahn
Mar 18, 2026
GPTBot vs ClaudeBot vs Bytespider compared
Compare GPTBot, ClaudeBot and Bytespider: what each operator says the crawler is for, its robots.txt guidance, published IP ranges, and how to check a claim.
Frederick Jahn
Mar 14, 2026
robots.txt for AI bots: tokens and examples
Copy-ready robots.txt rules for GPTBot, ClaudeBot, Google-Extended and other AI crawler tokens, how to test them, and why robots.txt cannot stop fakes.
Frederick Jahn
Mar 12, 2026
How to block AI crawlers and verify the block
Block GPTBot, ClaudeBot and other AI crawlers: robots.txt rules by purpose, a verified request rule for copied names, and two curl checks to test the block.
Frederick Jahn
Mar 10, 2026
What is an AI crawler? Types and examples
AI crawlers collect web content for model development, AI search and retrieval. How they work, the main operators' crawlers, and what your access logs can show.
Frederick Jahn