If you run a WAF or CDN bot product, it probably does useful work. The controls you have enabled can act on the identity, reputation, and request patterns they observe. The crawlers that cause the hardest content-access problem are built to avoid supplying a useful signal.
The bots a WAF can act on
Declared crawlers make an operator and purpose easier to identify. Search crawlers, AI training bots, and retrieval agents may state who they are in their user agent and publish documentation about their access. You can decide which of those agents to allow, block, or watch.
Where a policy is configured for a known identity, IP range, rate pattern, or request signature, the decision is straightforward. If the crawler identifies itself consistently, you can decide whether to allow, block, or watch it.
The traffic that blends in
Stealth crawlers take the opposite approach. They try to look like ordinary readers: a browser session, a consumer-network address, ordinary page requests, and a familiar user agent. Some use automated browsers; others mix browser behavior with proxy rotation and a spoofed client identity.
That makes a rule-only decision difficult. A rule cannot safely block every visitor that looks like Chrome on a residential network. Real readers look that way too. A broad rule risks blocking the audience you are trying to protect.
Detect behavior, not just identity
The useful question is not only who a request claims to be. It is whether the session behaves like the client it claims to be.
Centinel evaluates each session from the first request. It compares signals across the connection, browser, and request layers, then adds the behavior of the session over time. A client that claims to be a normal browser but leaves inconsistent transport or automation signals can be classified before it reaches more of your content.
This lets your team apply a separate policy to stealth crawlers. You do not need to turn every new scraper tactic into another rule. Centinel applies the policy you set to the sessions it classifies.
Keep the crawlers that help you
The goal is not to block every automated request. Some crawlers support search visibility, answer engines, accessibility tools, or partners you want to serve. A useful control plane separates those decisions:
- Allow verified crawlers that support your distribution strategy.
- Watch agents while you learn their volume and behavior.
- Block unlicensed or deceptive crawlers that are extracting content outside your policy.
Centinel reports humans, identified crawlers, and stealth crawler activity separately so product, audience, and commercial teams can make that choice with the same traffic picture.
Use both layers for their strengths
Your WAF should continue protecting the application: injection attempts, exploit traffic, abusive request patterns, and known malicious clients. Centinel adds the bot-management layer for crawlers that imitate readers and do not fit a stable rule.
That division matters when content is part of the business. Licensing, subscription access, archive protection, and product data all depend on being able to enforce the access policy you have set. You cannot make a reliable decision about that traffic while it is indistinguishable from audience activity.