Price scraping is the automated collection of prices, and usually the availability and listing details around them, from product, search, listing, or fare pages. Competitors, aggregators, and resellers run it to track what you charge and react faster than they could by hand, often through rotating proxies so each request looks like a new shopper.
It is a narrow case of general web scraping. OWASP's automated threat entry for scraping (OAT-011) describes collecting accessible data "for subsequent use such as exploitation of proprietary data, or price-fixing" and lists "comparative shopping" and "bargain hunting" among its other names. The same entry notes that scraping can be ongoing or periodic, and that it may use accounts or need none at all.
This article is about recognizing price scraping and choosing a response. The route-by-route control plan lives in how to prevent web scraping.
Who scrapes prices, and why
The people behind price collection usually want one of a few things:
- A competing store or marketplace wants to match or undercut your prices, or to see when you run out of stock.
- An aggregator or comparison site wants your catalog and prices without a feed agreement.
- A reseller watches for items priced below what they can sell them for elsewhere, and needs to see the change before other buyers do.
- A travel metasearch or booking reseller wants fares and availability, which change often and are costly to generate on your side.
What they have in common is a need for freshness. A price that is a day old is worth much less than one from ten minutes ago, so price scrapers revisit the same items on a schedule. That repeat pattern is often the most useful evidence you have.
Not all price collection is unwanted
Plenty of automated price reading is expected. Google's Shopping crawler, Storebot-Google, crawls pages for Google Shopping surfaces. Comparison partners may pull prices under an agreement. Your own pricing team may run a monitoring tool against your site to check that published prices match the catalog.
So the question is not "is this a bot reading prices?" The questions are who the client claims to be, whether that claim holds up, and whether your policy allows that client to collect at that rate.
A user-agent string settles none of that. Google itself warns that the user agent can be spoofed and documents how to verify a Google crawler: a reverse DNS lookup on the source IP, a check that the hostname belongs to googlebot.com, google.com, or googleusercontent.com, and a forward lookup that must return the original IP. Google also publishes IP range files for automated matching. A request that says Googlebot but fails that check is not Googlebot.
Robots.txt does not decide this either. RFC 9309 states that robots.txt rules "are not a form of access authorization." A scraper that ignores the file faces nothing from it.
What price scraping looks like in logs
The example below is illustrative. The paths, parameters, and addresses are invented for this article and are not Centinel data.
Suppose a store serves its category page as HTML and loads prices from a JSON endpoint behind it. An ordinary visit fetches the page, its scripts and images, then the price API with a session cookie, and sometimes moves on to a product page or the cart. The requests below arrive at the price API during one hour:
203.0.113.14 GET /api/v2/listings?cat=headphones&sort=price_asc&page=1 200 no cookie
198.51.100.72 GET /api/v2/listings?cat=headphones&sort=price_asc&page=2 200 no cookie
192.0.2.201 GET /api/v2/listings?cat=headphones&sort=price_asc&page=3 200 no cookie
203.0.113.90 GET /api/v2/listings?cat=headphones&sort=price_asc&page=4 200 no cookie
...
198.51.100.9 GET /api/v2/listings?cat=headphones&sort=price_asc&page=38 200 no cookie
No single address sends more than a handful of requests, so a per-IP rate limit never triggers. Looked at together, the requests tell a clearer story:
| Observation | Why it matters | What else could explain it |
|---|---|---|
| Pages 1 to 38 are covered in order, with the same sort and filter | A person browsing rarely reads every page of a category | A partner integration or your own monitoring job |
| Price API calls with no matching page load or asset requests | The client may be calling the data endpoint directly | Your CDN may serve assets, so the origin never sees them |
| No session cookie carried between requests | The client is not keeping browser state | Privacy settings, or a first visit on each request |
| The same sequence returns every six hours | Scheduled revisits to keep prices fresh | A cache-warming or feed job you forgot about |
| Addresses spread across many networks with one shared request pattern | Proxy rotation to spread volume | Many real shoppers following the same popular link |
Each row is a lead, not proof. Put them together over a long enough window and you have a reasonable case for a closer look. Group the requests by pattern (endpoint, parameters, header set, timing) rather than by IP. Detecting low-and-slow scraping covers how to choose that window and how to avoid turning a weak grouping key into a named person.
Before acting, rule out your own traffic. Check the addresses and patterns against your monitoring tools, feed partners, and any verified crawler you intend to allow.
Choosing a response
Public prices are visible to anyone who loads the page. No control makes them secret, and a determined collector can fall back to slower, more human-looking collection. A realistic goal is narrower: keep scraping from getting the freshest data at the lowest cost, protect your origin from the load, and keep approved clients working. Where the line sits is your decision, and so is the choice to allow some collection.
Observe. Start by recording which clients a proposed rule would affect, without changing their responses. This is how you find the partner integration or internal job before you break it.
Rate-limit. Limit the price endpoint by what it returns, not only by request count per IP. A budget for distinct items or pages per session, credential, or request pattern over hours catches coverage that a per-minute limit misses. OWASP's bot management cheat sheet recommends applying rate limits "at multiple keys, not just IP," and notes that per-IP limits are defeated by residential proxy networks.
Serve differently. Some sites give suspected automation a slower or less fresh response, such as cached prices or a challenge before the price API answers. The same OWASP cheat sheet goes further and suggests returning "plausible but slightly wrong data (price ±1%, fake stock counts)" to scrapers. That carries a real cost: a misclassified shopper sees a price you will not honor. Talk to whoever owns pricing and customer terms before trying it.
Offer a sanctioned path. If aggregators want your prices and you are willing to share them, a feed or partner API with its own credentials and limits gives you control over what they get. It also makes the remaining unapproved collection easier to separate out.
Block. Blocking fits clients that fail identity checks, such as a fake Googlebot, and clients that keep collecting after softer responses. Keep verified search crawlers and named partners on their own allow policy so a block rule cannot reach them.
Whatever you choose, detection does not tell you the operator's intent or whether their collection is lawful. It tells you how the request was made and whether it matches the policy you set for that route.
Where Centinel fits
Centinel evaluates how a request was made. In supported browser deployments, it combines request and browser evidence on search and item routes instead of relying on one address or header. It can compare a declared crawler identity with the verification data the operator publishes, so a copied crawler name gets no automatic trust. Your integration then applies the response configured for that route, such as allow, rate-limit, challenge, or block.
One limit matters for price scraping in particular. A client that calls your JSON endpoint directly does not run your page, so it supplies no browser evidence. Decisions on that path rely on request evidence and on whether the endpoint expects a browser session at all.
The marketplace industry page shows how this applies to search, item, sign-up, and messaging routes. The scraping and AI crawler use case covers crawler verification and per-client policy.