Apr 20, 2026
What is AI agent traffic?
AI agent traffic is a new class of traffic: training crawlers, retrieval crawlers, agentic workflows, and spoofed scrapers. How it differs from old bots, and what publishers can do.
Frederick Jahn
Understand the threats
Practical guides on AI crawlers, web scraping, and content protection.
AI scraper APIs
Mar 18, 2026
GPTBot vs ClaudeBot vs Bytespider: Comparison
A close look at the three most active AI crawlers. Who runs each one, how it behaves, and what it takes from your site.
Frederick Jahn
Mar 10, 2026
What is an AI crawler?
How AI crawlers differ from search engine bots, what they collect, and why they matter to your business.
Frederick Jahn
Real-browser automation
Mar 28, 2026
How to detect browser automation beyond user agents
How to catch browser automation when the user agent lies. Use TLS fingerprints, HTTP/2 signals, CDP artifacts, and behavior.
Frederick Jahn
Mar 25, 2026
Patched Chromium browsers explained
How scrapers patch Chromium to hide automation fingerprints, and why navigator.webdriver checks no longer catch them.
Frederick Jahn
Mar 22, 2026
TLS fingerprinting explained
How TLS fingerprinting spots bots by the shape of their handshake. JA3, JA4, and why it catches scrapers that user-agent checks miss.
Frederick Jahn
Crawler verification
Apr 20, 2026
How to verify AI agents
How to tell a real AI agent from a spoofed one. IP ranges, reverse-DNS, TLS fingerprints, request signing, and the policy on top.
Frederick Jahn
Apr 1, 2026
Why an interstitial challenge page is inevitable
Passive bot detection fails against modern scrapers. A challenge page is the one reliable way to protect your content from AI crawlers.
Frederick Jahn
Publisher traffic quality
Apr 20, 2026
Why manage AI agents, not just block them
Blocking everything leaves money on the table. The three paths every publisher needs are block, verify-and-allow, and watchlist. Per-agent policy beats one site-wide setting.
Frederick Jahn
Mar 14, 2026
robots.txt for AI bots: Complete guide
How to set up robots.txt for AI crawlers. Every directive, every major bot, and why robots.txt alone is not enough.
Frederick Jahn
Mar 12, 2026
How to block AI crawlers
A plain walkthrough of every way to block AI crawlers, from robots.txt to edge detection, and the tradeoff each one carries.
Frederick Jahn
Mar 10, 2026
What is web scraping?
How web scraping works, why companies do it, where the law stands, and how AI changed it.
Frederick Jahn