To reduce unwanted web scraping, control access at the point where your server returns the data. Enforce permissions for private resources, limit expensive or high-volume operations, and use automation evidence to decide which clients need a different response. Keep a separate path for approved crawlers and integrations.
Start with one route. A blanket rule across the whole site makes it harder to see what improved and which legitimate workflow broke.
This guide is for the engineer choosing those controls. For the mechanics of collection and extraction, start with what web scraping is.
Define what you need to prevent
Write down the unwanted outcome before selecting a tool. Someone collecting public records presents a different problem from an account reading private records it should not see. An expensive search endpoint has another concern: how much work each request triggers. A search budget cannot fix a private-record permission bug.
OWASP's scraping taxonomy covers collection through web pages and APIs, including resources behind accounts. A login screen alone therefore does not settle the scraping question.
For a public page, the practical goal is to restrict unwanted automated collection while preserving the access you intend to offer. Once you deliver readable data to a client, a bot policy cannot take that copy back. For private data, the first requirement is stricter: a requester without the necessary permissions must not receive it, whether that requester is a person or a script.
Consider a hypothetical application with public detail pages, a search endpoint, and an account export. Protecting the detail-page HTML would leave a gap if the same data remained freely available through search or an export response. Inventory the responses that contain the data, including alternate formats and cached copies.
Separate crawler instructions from enforcement
Use robots.txt to state where cooperative crawlers should go. RFC 9309 defines that protocol and explicitly separates its rules from access authorization. Your server still needs to enforce any restriction that must hold for a client that ignores the file.
Google's robots.txt guidance also distinguishes crawling from indexing: a disallowed URL can appear in search results when other pages link to it. Neither excluding a route from crawling nor hiding it from search is a substitute for securing its response.
Keep private paths protected by server-side permissions. Use the robots.txt guide for crawler-policy syntax, and check approved crawler access separately when changing enforcement rules.
Build a route-specific control matrix
The following matrix is a proposed planning method, not a tested Centinel configuration. Use it to review one route with its application owner.
| Resource | Intended access and useful evidence | First control to evaluate | Wrong-decision cost and rollback |
|---|---|---|---|
| Public detail page | Visitors and approved crawlers; verified operator identity and available session evidence | Keep intended access; review repeated collection before adding selective restrictions | Blocked visitors or crawlers; revert the new restriction and recheck access |
| Public search endpoint | Interactive users and explicitly supported integrations; request cost, result count, and request history | Bound result size and execution cost; evaluate a route-specific request budget | Broken searches or legitimate bursts; restore the previous budget without removing resource caps |
| Private record endpoint | Accounts entitled to the requested record; server-validated identity and object permissions | Enforce authorization on every request, then assess automated misuse within valid permissions | Data disclosure or denied account access; preserve authorization while reverting the new bot rule |
| Account export | Roles entitled to export; account identity, export size, and concurrent jobs | Check export permissions and bound job size and concurrency | Failed legitimate exports; restore the prior job policy while retaining permission checks |
| Partner API | Approved clients within their assigned scope; validated credentials and usage history | Apply client-specific permissions and budgets | Broken integration jobs; restore the prior approved policy, not an unrestricted bypass |
Separate the evidence from the decision. A crawler name is a claim to verify. A valid account establishes an authenticated identity, not permission to read every object. Missing browser telemetry tells you that an observation is absent; it does not, by itself, explain why.
For each row, name the enforcement component and the person responsible for reverting the policy. If nobody can point to where a denied request stops, the matrix is unfinished.
Layer controls in a safe order
Enforce permissions before returning private data
Apply the check on the server, including API and download routes. OWASP's authorization guidance recommends checking permissions on every request and protecting static resources where access is restricted. Hiding a button or requiring the browser to visit another page first does not perform that check.
A bot decision must not override an account's lack of permission. Keep authorization independent so a rollback of a new scraping rule cannot expose private records.
Bound the work a request can trigger
A request counter treats a small detail response and a large export as one request each. Review the work behind the endpoint as well as its frequency. OWASP's API resource-consumption guidance recommends limits on operations and data sizes, including server-side validation of parameters that select how many records a response returns.
In the hypothetical application, start by reviewing search result limits and export concurrency. Then choose request budgets that fit the intended clients. Do not copy a threshold from an unrelated site's configuration.
Decide what the rate limit counts
Record the counter's key, window, scope, and behavior when the key is unavailable. An IP budget groups all requests using that address. If several legitimate users share it, they share the budget too. A credential-based budget groups traffic differently and needs a trustworthy credential binding.
RFC 6585 defines 429 Too Many Requests and optional Retry-After guidance. It leaves client identification and request counting to the implementation. A 429 response communicates the limit; it is not evidence that the client is a scraper.
Test an approved integration's retry behavior. Confirm that your enforcement layer actually stops the expensive operation rather than doing the work and rejecting the response afterward.
Add automation evidence where it changes the response
For browser routes, review available browser and session observations alongside the request history. For direct API clients, use the evidence that exists on that path; a browser challenge is not a sensible default for a client that cannot render it.
The browser-automation detection guide explains how to interpret the evidence layers. Use the evidence to investigate a client and apply the route's policy. Avoid turning one unusual property into a site-wide block.
Run a control-and-failure test
This is a proposed reader exercise. No run or measured outcome is reported here.
The versioned plan is web-scraping-control-matrix-v1, dated 2026-09-04. It calls for 20 sessions in each of four cohorts, 80 sessions in total. Those counts are a planned functional sample, not a production accuracy estimate.
Use synthetic records on a staging route you own or are authorized to test. Every client must be test-owned or explicitly authorized. In the cohort names, "unapproved" means the test route's policy does not approve that automated collection; it does not mean the operator lacks permission to conduct the test.
| Cohort ID | Known client setup | Question for the test |
|---|---|---|
ordinary-browser | A person using an ordinary browser to complete the intended route | Can the user finish, including after an ordinary retry? |
approved-crawler | A test-owned crawler with an explicitly approved identity and route scope | Does approved collection still work within its budget? |
unapproved-direct-client | A test-owned HTTP client assigned to the restricted collection case | Does the route enforce its policy without depending on browser JavaScript? |
unapproved-browser-automation | A test-owned automated browser assigned to the same restricted collection case | Is the policy enforced when the client can execute browser code? |
Pin the client versions and policy revision. Keep the target data and intended task consistent, and retain a known-good policy for comparison. The operator should record each session's assigned ground truth before inspecting the detector's output.
Record the following separately:
- The observations available at the integration point, with absent observations marked unknown.
- The automation classification and the policy decision. An approved crawler can correctly be classified as automated and still be allowed.
- The actual response sent to the client, including retries or challenges.
- Whether the task completed, how much synthetic data was returned, and whether origin work occurred before rejection.
Keep timeout and incomplete sessions visible. A missing result must not silently become a successful block. Retain per-session evidence privately; use aggregate counts and a traceable evidence reference for review.
Also rehearse failure cases in isolation: a missing session, unavailable browser telemetry, a detector timeout, legitimate users sharing an IP budget, and a rule rollback. Record these checks separately from the 80-session plan; do not mix them into its cohort counts or describe them as completed until they have run.
Measure the policy and keep a rollback path
Before enforcement, write down the outcome that would make you revert. The application owner should set tolerances for legitimate task failures and response time based on that route's requirements.
Compare the old and proposed policy using the same tasks. Then review unwanted extraction completion alongside legitimate-user completion, approved-crawler access, challenge or retry frequency, and origin workload. Include counts and denominators. "More blocked requests" alone cannot tell you whether the policy is useful.
Treat an unavailable detector as a separate failure mode. For a public route, the team may choose continued access under existing resource budgets. Private routes must keep their authorization checks. Decide the timeout and fallback before rollout; do not discover the default during an incident.
Keep rollback narrow: restore the previous bot-policy revision, leave access controls in place, and rerun the legitimate-user and approved-crawler tasks. Confirm the integration actually uses the restored revision.
Where Centinel fits
Centinel's bot-management overview describes the evidence model and evaluation questions. For this method, the integration boundary matters: the customer's server calls the validation API and enforces its response. Browser-to-collector observations and the protected-request metadata sent by that server are different inputs. A direct HTTP client does not supply a browser payload.
Take particular care with an observation-only rollout. Centinel's policy documentation says monitor mode returns terminal blocks as allows, while challenges still return normally. It affects the request beyond the rule that enabled it. Do not treat that switch as a promise that test traffic will encounter no intervention; verify the complete policy on an isolated staging route.
For the broader product use case, see scraping and AI crawler protection. The next practical step is to fill in the matrix for one route, identify the enforcement point, and review it with the team that owns the application before changing production traffic.