Crawlers are running up your server bill. Block the ones that don't pay.

Frequently asked questions

How much bandwidth do AI crawlers actually consume?

Enough to be a line item. Cloudflare reports AI training crawl traffic up 65% in six months and AI agent crawling up over 15x across 2025. A single publisher on Centinel saw 42,000 euros per month in CDN bandwidth savings after blocking unauthorized crawlers. The cost is invisible on your invoice because it bills against aggregate egress, which is why teams underestimate it until they block the traffic.

Will blocking crawlers break anything on my site?

Not if configured correctly. Verified search engines (Googlebot, Bingbot) and legitimate API clients pass through on a separate allowlist. Centinel's default policy blocks unverified AI crawlers while letting everything that affects discoverability through. Every policy is per-crawler, so you can monitor first and block second if you want to see the traffic before acting on it.

How quickly do I see cost savings?

Bandwidth and compute savings land the same day blocking turns on. CDN billing cycles vary, but the first post-deployment invoice typically reflects the drop. Flowlane, a B2B SaaS on Centinel, measured a 78% reduction in origin load within weeks of deploying, which translated directly into a smaller docs infrastructure footprint.

Does edge blocking add latency for real users?

No. Detection runs in under 2ms at the edge, well under the threshold real users can perceive. Blocked requests never reach your origin, which frees capacity for humans. The net effect on p95 response time for legitimate traffic is typically a small improvement, not a regression.

Can I cap what I spend on crawler protection?

Yes. Every plan has a request cap scoped to your traffic. Requests beyond the cap default to pass-through, not a surprise overage. You decide whether to upgrade the tier, rate-limit the excess, or accept the passthrough. We do not bill retroactively against a traffic spike you did not authorize.