Control what scrapers and AI crawlers can access
Protect content, catalogs, and application data without blocking every crawler. Start by checking which scraping tools can retrieve your pages, then choose controls that preserve approved access.
Request a site auditA crawler name is a claim, not proof
Copied pages, repeated catalog requests, and unexpected origin load can prompt a scraping review. A crawler name alone cannot settle it: declared clients need identity checks, and disguised scrapers need automation evidence. Search crawlers, monitoring, and partners may still need access. Decide which resources each client may reach rather than blocking automation as one category.
Read the researchWhat the traffic looks like
A browser, not a script
Automation frameworks can drive a full browser, execute page scripts, and produce requests that resemble an ordinary visit.
A different IP on every request
Residential proxy networks can rotate requests across consumer connections, which makes a static address rule incomplete.
A crawler name anyone can claim
A user-agent string is easy to copy. The rest of the request still needs to match the operator's published verification method.
What runs against this traffic
Verify declared identities
Centinel can compare a declared crawler identity with verification data published by its operator.
A matching identity can receive its intended policy. An unverified claim receives no automatic trust.
Read more: Verify declared identitiesEvaluate browser evidence
In supported browser deployments, Centinel combines request and browser evidence instead of relying on one automation tell.
Teams can challenge or restrict suspicious automation without putting every visitor through the same check.
Read more: Evaluate browser evidenceA policy for each traffic class
Supported integrations apply the configured response for the client's identity, behavior, and requested resource.
Approved crawlers can keep the access they need without extending that trust to disguised scrapers. Confirm the enforcement point and available responses for your stack before changing policy.
Read more: A policy for each traffic class
Want to see what reaches your site?
Start with a free scraping audit: tested tools, exposed pages, and fixes to consider, reviewed by hand and emailed to you. This checks scraping exposure, not every abuse pattern or all production traffic. Discuss broader workflow coverage in a demo.