Passive bot detection fails against modern scrapers, and an interstitial challenge page is the only reliable way to protect content from AI crawlers.
What is an interstitial challenge?
An interstitial challenge is a gate served before the real content loads. The page injects a small piece of code. It might be a puzzle, a web-API probe, or a browser-quirk test. The client has to run it correctly to continue. No code runs, no content loads.
The model flips the question. Passive detection asks what are you? It inspects the signals the client chose to send. An interstitial challenge asks what can you do? It measures what the client actually runs. A good enough library can spoof the first. The second needs the client to run the code. Running code is what scrapers try hardest to avoid at scale.
New to TLS fingerprinting? Start with our primer, TLS fingerprinting explained. This article picks up where that one ends.
Why interstitial challenges matter right now
TLS fingerprinting spots bots by reading the first bytes of a connection. For years it worked. In 2023, Chrome broke the dominant fingerprinting method. A wave of spoofing tools filled the gap. Passive detection no longer stops modern scrapers. Watching what a client sends stopped working once clients learned to lie.
JA3 was the industry's default TLS fingerprinting method from 2019. It hashed the cipher suites and extensions a client announced during the TLS handshake. Every browser, scraping library, and bot framework produced a unique hash. A Python script claiming to be Chrome got caught the moment the handshake hit the wire.
Then Chrome started randomizing the order of its TLS extensions. One Chrome client with 16 extensions can order them 16 factorial ways. That is roughly 20.9 trillion distinct JA3 hashes from the same browser on the same machine. Researchers measured this after Chrome's change. JA3 is now useless for identifying clients and user agents. For the full story on how it fell, see TLS fingerprinting explained.
JA4 fixed the ordering problem. It sorts extensions before hashing. But it did not fix the deeper issue. A growing set of tools rebuild real browser handshakes from scripts. curl-impersonate compiles against BoringSSL to produce byte-identical Chrome Client Hellos. uTLS and Noble TLS do the same in Go and other languages. They match any TLS fingerprint to whatever user-agent string the developer provides. The fingerprint is no longer something the client reveals. It is something the client chooses.
In 2024, 95% of advanced bot attacks went undetected. Nearly two in three businesses have no protection against even basic bot attacks. Only 15.82% of bots impersonating Chrome were detected. And 83% of simple curl-based bots passed unnoticed. CAPTCHA solving farms now charge $0.80 per 1,000 solves, down from $3 in 2018. They also solve 5x faster than they did six years ago. The economics have flipped. Spoofing every signal a passive system checks is now cheaper than the detection itself.
Types of interstitial challenges
Four challenge families cover what production systems deploy.
JavaScript execution probes. A payload runs in the page. It measures behaviors only real browsers produce. A background JavaScript challenge is the classic example. It runs non-interactive tests in the background that gather signals about the visitor or browser environment. The visitor sees nothing, or at most a brief loading indicator. This approach cut the average challenge time from 32 seconds in the old visual CAPTCHA era to roughly one second.
CAPTCHA (interactive). These are visual or audio puzzles that need a human response. They still guard high-risk actions like account creation and payment. As a mass detection layer they have weakened. Solving farms now route CAPTCHAs through human and model-driven pipelines at the cost and speed noted above.
Proof-of-work. The client has to compute a cryptographic puzzle before it gets content. The Anubis project runs on Arch Wiki, GNOME, WineHQ, FFmpeg, and UNESCO. It presents a SHA-256 challenge. The client must find a nonce so the hash of (challenge + nonce) has N leading zeros. A real browser solves this in milliseconds. A single human visitor barely notices. A botnet hitting thousands of pages per minute pays that CPU cost on every request. The cumulative cost becomes significant.
Behavioral and web-API probes. These check mouse timing, pointer accuracy, and APIs only real browsers implement. Examples include storage quotas, permission states, and rendering-engine quirks. The page watches whether the rendering engine behaves like the one the fingerprint claims.
How interstitial challenges work
An interstitial challenge flips the verification model. Instead of asking what are you?, it asks what can you do?
The mechanism works because it trusts no signal the client sent. It generates a new signal on the spot. The client cannot fake it without running the code.
A background JavaScript challenge adapts the outcome to each visitor or browser. First it runs a series of small non-interactive JavaScript challenges to gather signals about the environment. Proof-of-work goes further. It forces the client to burn CPU before it gets content. Behavioral probes layer on top. They check whether the rendering engine returns the quirk values a real Chrome would return. They check whether the web APIs respond with the latencies a real browser produces.
The economics of AI crawling make challenges effective. Anthropic's crawl-to-refer ratio reached 500,000:1. It crawled half a million pages for every one it sent back as referral traffic. AI training crawl traffic rose 65% in six months. AI agent crawling grew more than 15x in 2025. At those volumes, any per-page cost compounds. A challenge takes a real browser one second. It takes a headless Chrome instance the same time. But the headless instance also needs CPU allocation, memory, a full rendering engine, and network coordination. Simple HTTP scrapers cannot run JavaScript at all. That includes curl, Python requests, and Go net/http. They hit the challenge page and get nothing. Stepping up to headless browsers adds cost, latency, and a new surface for detection.
How to identify which content needs a challenge
Not every page needs an interstitial. Challenges carry a small UX cost. Deploy them on the routes where scraping is most expensive or most damaging.
Start with high-value content. That means paywalled articles, proprietary pricing pages, search APIs, RSS feeds, and any endpoint that returns structured data at scale. Add challenges to authentication flows. In the first week of March 2025, 94% of authentication requests on the internet came from bots. Leave them off low-value marketing pages, where the scraping cost is less than the friction cost.
Adaptive risk scoring takes the decision off the page and onto the session. Low-risk visitors skip challenges entirely. Think of a clean IP, a normal fingerprint, and a returning session. High-risk visitors see them. Think of a residential proxy, a mismatched fingerprint, or a first-touch session. The UX cost falls on the traffic that earned it.
Open-source tools like Anubis prove the same point. It runs on Arch Wiki, GNOME, WineHQ, FFmpeg, and UNESCO's infrastructure. Millions of visitors never notice the challenge is there. The difficulty scales. It stays low enough for human browsers to clear without a felt delay. It stays high enough for botnets to feel the cost at scale.
How to respond to challenge bypass attempts
No defense is permanent. Challenges are no exception. The response is layering, not a single silver bullet.
Anti-CDP frameworks are built to clear JavaScript probes while dodging the Chrome DevTools Protocol signals a detector would normally catch. nodriver had 590+ GitHub stars by mid-2024. Ghost cursor libraries do similar work. The response is to watch for bypass indicators. A challenge solved in an unnaturally consistent time is one. An abnormally high pass rate from a single ASN is another. So are cursor paths that reach the submit button on geometrically clean trajectories. When you see them, refresh the challenge payload. Challenge content should rotate. Client-side tests should sample from a pool rather than repeat. Adaptive difficulty should ramp when a session looks like it has seen the test before.
robots.txt is the signal of what failed. Only 37% of the top 10,000 domains even have a robots.txt file. In Q4 2025, 30% of all AI bot scrapes did not abide by explicit robots.txt permissions. OpenAI's ChatGPT-User agent accessed content from 42% of the sites that explicitly blocked it. A challenge page is not a request. It is a technical gate.
Key takeaways
- Passive detection misses 95% of advanced bot attacks. TLS fingerprinting is spoofable with
curl-impersonateanduTLS. In Q4 2025, 30% of AI bot scrapes ignoredrobots.txt. The only signal a bot cannot fake is one it generates on demand, in an environment you control. - Four challenge families cover production deployments: JavaScript execution probes, CAPTCHA, proof-of-work, and behavioral/web-API probes. A background JavaScript challenge runs in roughly one second, down from the 32-second CAPTCHA era.
- Target challenges at high-value routes like paywalls, pricing, search APIs, and authentication. Adaptive risk scoring keeps low-risk visitors friction-free and puts the UX cost on traffic that earned it.
- Challenges are the enforcement layer, not a replacement for fingerprinting or behavioral analysis. Centinel integrates challenge-based verification with 1,600+ crawler fingerprints and layered behavioral detection.