Here is every way to block AI crawlers, from robots.txt to edge detection, and what each one costs you.
What is AI crawler blocking?
AI crawler blocking stops automated clients that scrape your content. These bots feed AI training, retrieval, and agents. The methods vary widely. One is a two-line entry in robots.txt. Another inspects each request at the edge. They also differ in what they enforce.
Two lines of robots.txt stop the polite crawlers. The rest ignore it. To block those, you inspect the request itself. You read the TLS handshake and check that the headers stay consistent. You watch how fast the requests come. Then you decide, per request, before the bot reaches your origin.
Why blocking matters now
Volume makes the case. In our data, we counted 9 billion AI bot scrapes across 550 billion website visits in late 2025. Of those, 2.9 billion bypassed robots.txt entirely. Publisher sites saw one AI bot visit for every 31 human visits. Two quarters earlier, the ratio was 1 to 50. We measure AI bots reaching 39% of the top million sites. Only 2.98% block them in robots.txt.
The gap between policy and enforcement is where theft happens. A site that relies on robots.txt alone only states a preference. It stops nothing. The real question is which layer does the enforcing.
Types of blocking methods
Six methods matter, from simplest to most effective.
robots.txt. Add directives that tell specific crawlers to stay away. To block GPTBot, add User-agent: GPTBot and then Disallow: /. It takes 30 seconds. It needs no code. But it is voluntary. Our data shows 30% of AI scrapes ignore robots.txt. ChatGPT-User fetched 42% of the sites that blocked it.
HTTP header checks. Read the User-Agent header. Reject known AI crawler signatures. You can do this in the web server, like Nginx or Apache, or in your app code. It is easy to deploy. It is also easy to beat. A bot just changes its user agent string.
IP blocking. Block the IP ranges owned by AI companies. OpenAI, Anthropic, and others publish theirs. This is harder to bypass than a user agent check. But the ranges change often. Residential proxy networks route around the block.
Rate limiting. Cap requests per IP or session inside a time window. This cuts volume without a full block. But skilled scrapers spread requests across thousands of IPs. Tight limits also hurt real users.
JavaScript challenges. Make the visitor run JavaScript before you serve content. This stops basic scrapers that only speak HTTP. But modern tools render JavaScript in full. Playwright, Puppeteer, and patched Chromium all do. Challenges also add latency for real users.
Edge-level detection. Run a detection layer at the CDN or edge. It analyzes every request in real time. It combines TLS fingerprinting, behavioral analysis, IP reputation, device fingerprinting, and crawler database matching. It catches crawlers no matter the user agent or IP. Latency stays under 2ms. It needs a specialized provider. Centinel runs at this level and matches 1,600+ crawler signatures.
How crawler blocking works
The six methods are not swaps for each other. They are layers. Each one catches a different kind of bot.
robots.txt filters the honest crawlers. They read the file and leave. Header and IP checks catch the lazy bots that name themselves. Rate limiting catches the loud scrapers that hammer one origin. JavaScript challenges catch the HTTP-only libraries that cannot run code. Edge detection catches what survives the first five. It reads low-level signals the bot cannot fake cheaply.
A layered defense costs less and works better than any single method pushed to its limit. The bot that beats your user agent check may fail your TLS check. The one that fakes TLS may fail the behavioral check on page two. Each layer raises the cost of evasion.
How to pick the right method
Start with robots.txt. It costs nothing and handles well-behaved crawlers. Add header and IP checks at the server. They are cheap and cover the lazy 20%.
Some sites face real scraping pressure. Think publishers, shops with proprietary catalogs, and SaaS dashboards with valuable screens. For those, add rate limiting and challenge pages on high-value routes. For real protection at scale, use edge detection. It enforces instead of informs.
The page decides. A marketing site loses little when scraped. A paywalled publisher loses its business model. The higher the content value, the deeper the enforcement must live.
What to do when a method gets bypassed
Every method has a counter. The fix is not to swap one method for another. The fix is to layer detection so no single bypass wins the session.
Watch the bypass signal. Say traffic spikes from one user agent, ASN, or TLS fingerprint after a block goes live. That tells you the rule got routed around. Escalate one layer down. Go from user agent filtering to IP blocking. Go from IP blocking to TLS fingerprinting. Go from TLS to behavioral analysis. Refresh your fingerprint databases on a schedule. Published crawler signatures drift. New ones appear. A stale database misses traffic a current one would catch.
Key takeaways
robots.txthandles honest crawlers and nothing else. By our count, 30% of AI scrapes ignore it.ChatGPT-Userbypassed 42% of the sites that blocked it.- Six methods cover the field:
robots.txt, header checks, IP blocks, rate limits, JavaScript challenges, and edge detection. Each catches a different class of bot. - Layered defense beats any single method pushed to its limit. Edge detection at the CDN is the only layer that enforces instead of informs.
- The response is a cadence, not a one-time setting. Watch bypass signals. Refresh fingerprint databases. Escalate one layer down when a method gets routed around.