Low-and-slow scraping is automated collection spread over enough time that a short rate-limit window may never look busy. To investigate it, compare what a client collects across a longer period: distinct resources, repeated traversal, returned data, and the permissions that make those responses available.
A quiet request graph can still describe substantial extraction. The useful question is whether access fits the client's purpose and allowance.
OWASP's scraping threat description includes recurring collection of accessible pages and API responses. It does not give a universal request rate that separates scraping from normal use.
Change the unit of investigation
Start with one valuable resource family, such as a directory or a collection of detailed records. Choose a bounded review window that covers its normal usage cycle. A minute, a session, and a working day answer different questions.
For authenticated access, group by a server-validated account or API credential. For anonymous access, keep the uncertainty visible: a cookie may disappear, and an IP address may represent many visitors. Do not silently turn a shared fingerprint into a person.
Record the number of distinct resources returned, total response bytes, and whether requests revisit earlier material or steadily cover new material. Use normalized routes for grouping, but preserve a restricted reference to the original request if an investigator needs it.
A small example of what a rate chart misses
The following is hypothetical. Two clients each make six requests in every ten-minute window. Neither looks unusual by rate alone.
| Observation over the review period | Client A | Client B |
|---|---|---|
| Resource selection | Revisits a small working set | Continues through new records |
| Navigation | Alternates reading and editing | Repeats list-to-detail traversal |
| Successful data returned | Mostly repeated material | Mostly previously unseen material |
| Known purpose | Documented work session | No documented bulk-access need |
Client B deserves investigation, not an automatic guilty verdict. A migration, accessibility tool, or approved synchronizer could produce a similar record. Conversely, Client A could still misuse the material it receives.
The example's value is the next question it creates: who is entitled to retrieve this collection at that scale?
Look for continuity without inventing identity
A source IP changing halfway through a collection does not prove a new visitor. Equally, similar navigation from several IPs does not prove coordination.
Keep strong and weak grouping keys separate in the investigation. An authenticated account is useful for account-level policy; a repeated route sequence is a lead. If you combine weak signals into a suspected group, record which signals were used and what alternative explanation remains.
OWASP's bot-management guidance describes controls at several observation layers. Use that layered approach to corroborate the sequence instead of making one browser property carry the decision.
Respond to extraction, not just speed
For the selected resource family, consider a cumulative allowance for unique records or delivered data, alongside a short burst limit. Put approved bulk jobs on an explicit, bounded access path. A long-window allowance should have a reset rule, an owner, and a way for legitimate users to request more access.
Begin by recording which requests the proposed policy would affect. Check a representative busy workflow and an approved automation job. Then enforce a narrow change where authorization and response handling are understood.
Watch whether successful extraction falls, whether ordinary work still completes, and whether activity moves to another route. A higher block count alone does not answer any of those questions.
Use the scraping and crawler use case to scope the protection boundary. For the broader route-by-route control plan, continue with web scraping prevention.