Web scraping is the automated extraction of information from web resources. A scraper may download HTML, call an endpoint used by a page, render the page in a browser, select the relevant fields, and store the result in a structured form.
The technique is not inherently one thing. Search tools, accessibility services, testing systems, internal integrations, researchers, price monitors, and abusive operators can all automate web access. Whether traffic is wanted depends on identity, purpose, resource, rate, and site policy.
How web scraping works
A scraping workflow commonly contains these stages:
- Discover: obtain URLs from a sitemap, link graph, feed, search result, or supplied list.
- Fetch: request a document or endpoint using HTTP.
- Render if needed: run browser code when the required data is not present in the initial response.
- Extract: select text, links, attributes, or structured data.
- Normalize: convert the result into records with stable fields.
- Store and revisit: save the records and schedule updates.
RFC 9110 defines HTTP semantics, but it does not assign a benign or abusive purpose to a request. The same GET request can come from a person, an approved integration, a search crawler, or an unwanted scraper.
Simple scrapers may use an HTTP library and an HTML parser. Browser-based scrapers can execute JavaScript, interact with controls, and retain session state. Distributed systems can spread retrieval across many workers or networks. These are implementation choices, not reliable policy categories by themselves.
Web crawling, scraping, and browser automation are different
The terms overlap but answer different questions:
| Term | Primary action | Example output |
|---|---|---|
| Web crawling | Discover and retrieve resources | A URL index or fetched-document set |
| Web scraping | Extract selected information | Structured records, text, or media references |
| Browser automation | Control a browser workflow | Navigation, form completion, testing, or rendered data |
A crawler can fetch pages without extracting their contents. A scraper can work from a fixed URL list without discovering links. Browser automation can perform legitimate quality assurance without scraping, while a scraper may use a browser when the page requires rendering. Scraping APIs such as Firecrawl and Zyte can fetch with plain HTTP requests or managed browsers, so one service can reach your site in more than one form.
The WebDriver specification standardizes one form of remote browser control. Its existence is also a reminder that legitimate automation is common. “Browser controlled” and “abusive scraping” are not interchangeable verdicts.
Named crawlers need purpose-specific rules
Some operators publish crawler names and guidance. Others do not. Even a documented operator may use separate product tokens for model development, search, or user-directed retrieval.
robots.txt can express a crawl preference to a conforming crawler. RFC 9309 defines the protocol. It does not authenticate a requester or enforce authorization, so a user-agent claim must not be treated as trusted identity on its own.
For a declared crawler:
- record the exact token and requested resource;
- consult current operator-primary documentation;
- verify the source using the operator's published method when one exists;
- assign policy by purpose and resource;
- define behavior when identity cannot be resolved.
For undeclared traffic, preserve uncertainty. “Unknown” is a useful state. It is better than forcing a weak observation into a named operator or human/bot label.
Define the unwanted behavior first
Before choosing a control, write down the outcome you need to change. Examples include:
- repeated retrieval that creates material origin or application work;
- automated extraction of a resource your policy does not permit in bulk;
- inventory or price monitoring at a rate that harms a service;
- automated account actions such as credential testing or mass registration;
- bypass of an explicit authentication or authorization boundary.
The OWASP Automated Threats to Web Applications project provides a vocabulary for automated misuse. Use that vocabulary to keep the problem statement tied to an application action rather than a vague dislike of bots.
Different resources can justify different responses. A public product page, an expensive search operation, and a private account endpoint should not inherit the same rule just because requests share a fingerprint.
How to investigate suspected scraping
Use observations that can be reproduced and that fail independently.
Resource and rate
Identify the affected routes, request methods, response sizes, cache behavior, and downstream work. Compare the suspected cohort with an appropriate baseline. A high request count can be normal for a feed or monitoring integration and harmful for a costly search route.
Claimed and verified identity
Separate the user-agent claim from verification. Where an operator publishes current network ranges or reverse-DNS guidance, test the claim against that source. Record the source version and the unresolved cases.
Client consistency
Compare browser, transport, header, and runtime observations that should agree. A mismatch contributes evidence, but network middleware, privacy tools, enterprise software, and unusual devices can also change these surfaces.
Session and application sequence
Look at navigation order, retries, concurrency, state transitions, and protected-action outcomes. Do not assume that fast, regular, or non-interactive behavior is automatically malicious. Internal systems and assistive tools may have similar shapes.
Ground truth and error
Build labeled test cohorts where possible: ordinary supported browsers, approved automation, stock automation, and the suspected workflow. Measure false positive and false negative behavior. Without reviewed examples, a score change is not proof that scraping was found.
Choose a bounded response
Match the control to the evidence and the resource:
| Situation | Possible response |
|---|---|
| Known approved integration | Authenticate, scope, and monitor |
| Documented crawler with verified identity | Apply its explicit resource policy |
| Unknown low-impact traffic | Observe or use a generous bounded rate |
| Uncertain traffic on a costly route | Rate-limit or challenge with a recovery path |
| Corroborated policy violation | Deny the affected action and retain review evidence |
Start in observation mode. Review the cohorts that would be affected, then roll out on a limited set of routes. Define rollback thresholds and a way to correct false positives before enforcement expands.
Controls can include authenticated APIs, cache design, pagination limits, request budgets, rate limits, crawler verification, session analysis, challenges, and route-level deny rules. Their usefulness depends on placement. Browser code cannot govern a direct request that never loads it, and an edge rule may not understand application state unless that state is provided safely.
The scraping and AI crawler use case explains how these controls fit together.
Separate legal review from traffic enforcement
This section is operational guidance, not legal advice. The legal treatment of scraping can depend on jurisdiction, the resource, authorization, contract formation, copyright or database rights, privacy obligations, technical circumvention, and the facts of the collection and use.
Avoid universal statements such as “public data is always legal to scrape” or “a robots.txt rule makes access unlawful.” Neither is a safe technical conclusion. Ask qualified counsel to review the specific policy and use case.
Engineering can make that review more concrete by preserving:
- the resource and access state at the time of each request;
- the policy version and how it was communicated;
- available identity evidence and its source;
- rate, sequence, response, and enforcement outcome;
- retention limits and access controls for the evidence.
A legal position does not replace an enforceable technical boundary, and a technical block does not settle a legal question. Keep ownership and review for both.
Build a policy you can revisit
Scraping tools, browser versions, operator documentation, and business priorities change. Give each rule an owner, evidence source, review date, and expiry condition. Remove rules whose original justification no longer exists.
The AI-agent access policy guide provides a reusable resource-by-identity matrix. The bot management overview lists the product and integration questions to test before choosing an enforcement layer.
Web scraping is an extraction method. A defensible response starts with the affected resource and observed behavior, then adds identity evidence, error measurement, and a proportionate control.