Here is how web scraping works, why companies do it, where the law stands, and how AI changed the game.
What web scraping is
Web scraping is the automated extraction of data from websites. A scraper sends a request to a web server. The server sends back HTML. The scraper reads that HTML and pulls out what it wants: product prices, article text, inventory counts, editorial content.
The trick is as old as the commercial web. What changed in 2026 is who scrapes, how much, and what they do with the data.
Why it matters now
People scrape for many reasons. They compare prices, do research, recruit, and study competitors. Now AI companies scrape the whole web to train their models.
AI changed the math. Old-style scraping grabs specific data from specific sites. AI scraping is bigger and different. Model builders need huge amounts of varied text. So they scrape wide and deep. They return to the same sites again and again.
In 2025 we measured 550 billion website visits. Of those, 9 billion were AI bot scrapes. And 2.9 billion of them bypassed robots.txt. On publisher sites, we saw a 1-to-31 ratio of AI bot visits to human visits. Two quarters earlier it was 1-to-50.
The law is messy and depends on where you are. In the US, the Computer Fraud and Abuse Act (CFAA) and copyright law offer some cover. Enforcement is spotty. The EU's Database Directive guards structured data more firmly. Lawsuits against OpenAI and Anthropic are testing whether AI training counts as fair use. The law is still catching up. Until it does, your infrastructure holds the line.
Types of scraping
Four kinds of scraper traffic show up on a live site. Each has its own technical signature and its own commercial goal.
Personal and research tools. A developer runs BeautifulSoup or Scrapy. A journalist pulls FOIA data. The scale is small. The user agents are easy to spot. You can usually rate-limit them.
Commercial scraping services. BrightData, Oxylabs, ScraperAPI, and others. They rotate residential proxies, automate browsers, and solve CAPTCHAs. Detection is much harder here. Each request looks like it comes from a different home ISP.
AI training crawlers. GPTBot, ClaudeBot, Google-Extended, Bytespider, Applebot-Extended, CCBot. Most name themselves. Most respect robots.txt. The real volume hides in the long tail of unlabeled crawlers.
Adversarial scrapers. They go after paywalled content, competitor pricing, and proprietary data. They use patched Chromium builds, curl-impersonate, or custom TLS libraries. The goal is to copy a real browser's handshake byte for byte.
How it works
A scraper is a program. It downloads web pages and pulls data from the HTML. Modern scrapers do more. They render JavaScript, solve CAPTCHAs, and rotate through proxy networks. They even copy real browser behavior, down to mouse moves and scroll patterns.
The request itself is plain HTTP. The scraper controls the user agent, the TLS handshake, the HTTP/2 settings, the cookies, and the timing. It cannot fully control how these layers line up. A Python HTTP library that claims to be Chrome still sends a TLS fingerprint. That fingerprint gives away the lie on the first byte. A headless browser running a script has a rhythm. It is steady and exact. It skips the pauses a human makes. That rhythm sets it apart from a real reader.
How to spot scraping on your site
Four signals point to scraping. No single one proves it. Together they do.
Traffic anomalies show up first. Requests to product or article pages jump, often outside business hours. You see a spike in 429 or 403 responses. At the same time, your origin cache hit rate drops.
User agent honesty is the next check. A Chrome user agent from a cloud ASN looks wrong. A GPTBot from an IP outside OpenAI's published ranges is a fake.
Rate patterns tell bots from readers. Humans linger, scroll, and pause. Bots hit pages at a steady pace or in matched bursts. Revisit times that fit a clock too neatly, every 6 hours or every 24 hours, come from a scheduler, not a reader.
Origin hops close the case. One session touches twenty pages from twenty different ISPs in the same city. That is one scraper wearing twenty masks.
How to stop unwanted scraping
Good anti-scraping needs layers: rate limiting, IP reputation, TLS fingerprinting, behavioral analysis, and crawler identification. No single method is enough. Scrapers adapt. The goal is not to block every request. The goal is to make scraping cost more than the data is worth. Then attackers move to easier targets.
Start at the edge, before the request reaches your origin. Match the TLS fingerprint and the HTTP/2 SETTINGS frame against a known library signature. Check the user agent for internal consistency. For the long tail, check the signals against a fingerprint database. Centinel tracks 1,600+ of them. Then decide per request whether to block, challenge, or watchlist.
Pair this with a legal stance. Post a public scraping policy. Set a clear access policy for training crawlers. Add a clear terms-of-service clause. These give the technical layer something to point at when a scraper escalates.
Key takeaways
- In 2026, AI training and retrieval crawlers drive most scraping, not classic price scrapers. Our data shows 9 billion AI bot scrapes across 550 billion visits. 2.9 billion of them bypassed
robots.txt. - Four classes of scraper each need different handling: personal tools, commercial services, AI training crawlers, and adversarial scrapers.
- Detection spans layers. User agents, TLS fingerprints,
HTTP/2settings, and behavior together catch what any single signal misses. - Prevention is about cost, not perfection. Make scraping cost more than the data is worth, and attackers move to softer targets.