A platform engineer at a mid-size publisher ships a rate limit on a Friday afternoon. By Monday, the SEO lead is asking why Googlebot crawl stats dipped, and support has tickets from a university campus where four thousand readers share one egress address. Nobody did anything careless. The rule did what rules do: it counted requests and acted on a threshold.
Hold that frame when you configure netlify bot protection. Netlify's firewall traffic rules, WAF and rate limiting restrict access or request velocity. They're good at that. They don't tell you whether a visitor is a person, or whether a request that says it's a crawler really is one. A rule that keys on IP, path or user agent string treats a spoofed header and a genuine one the same way.
This guide walks through what the controls do, where plan limits apply, how deploys and inheritance scope them, the order requests pass through them, how to verify behavior in logs, and how to roll back without breaking legitimate traffic. Check the current Netlify docs for each feature before you rely on it, because availability changes by plan.
What Netlify's controls do and do not tell you about a bot
Every control in the Netlify toolbox answers one of two questions: should this request be allowed, or has this client sent too many requests in a window? Neither answers the question operators actually care about, which is whether the visitor is genuine.
Take a rule that blocks any request whose user agent contains "python-requests". It stops lazy scripts. It does nothing about a scraper that sends a Chrome user agent. Now take a rule that allows anything claiming to be Googlebot. It lets Googlebot in, and it lets in every scraper that copied the string. Centinel Analytica's published research found that on a typical news site, 84% of requests claiming to be GPTBot were fake. A user agent allowlist would have waved most of them through.
Rate limits have the opposite weakness. They measure velocity, so they catch a client hammering one path. They also catch a corporate proxy, a school network, or a mobile carrier gateway where thousands of real people share an address. A limit tuned tight enough to slow a patient scraper will hit those users first.
So take a position before you write a single rule: Netlify's controls are access and velocity tools. Use them for what they do well, like blocking known-bad paths, capping abusive request rates on expensive endpoints, and restricting by country or network where your business allows it. A passing request proves nothing. A blocked request proves only that your rule matched.
Write that distinction into your runbook. It changes how you read logs, how you word incident reports, and how you decide what to do when a rule fires on a crawler you wanted.
Plan availability: code-based limits versus Enterprise High-Performance Edge features
Before designing anything, find out which controls your plan includes. Netlify splits its protections across two broad groups, and mixing them up leads to rules you can't deploy.
The first group is code-based. These live in your repository and ship with your deploy: for example, rate limit configuration declared alongside a function, and logic you write in functions or edge functions. They're available without a special network tier, and they're versioned with your code, which helps with review and rollback.
The second group is the platform-level firewall and WAF-style controls, including traffic rules you manage in the Netlify UI. Several of these are tied to Enterprise and the High-Performance Edge network. If you're on a lower plan, a rule you read about in a blog post may simply not exist in your dashboard.
Do this first: open your team's plan page and the Netlify docs for each control you intend to use, and write down, per control, whether it's included, add-on, or Enterprise only. Put that table in the ticket. Procurement conversations go faster when engineering can say exactly which controls are missing and what each one would have handled.
Then match controls to problems. A login endpoint under credential stuffing pressure needs a velocity limit at minimum, and a code-based limit may be enough. A catalog being scraped by clients that rotate addresses needs more than a velocity limit, and no plan tier changes the fact that Netlify's rules key on request attributes, not on proof of identity.
Don't buy a tier hoping it answers the identity question. Buy it if you need the specific access and rate controls it unlocks.
Published versus unpublished deploys, and team versus project inheritance
Two scope questions cause most surprises in practice: which deploy a rule applies to, and which level of the account it was set at.
Start with deploys. Netlify builds many deploys that aren't your published production site: deploy previews, branch deploys, and older deploys kept in history. Configuration that ships in code travels with the deploy that contains it. A rate limit committed on a feature branch exists in that branch's deploy and does nothing on production until it's merged and published. The reverse also bites: a rule that works in the preview you tested may not be live where traffic actually lands. Confirm which deploy is published before you read any test result as production behavior.
For testing, the rule is simple. Run your checks against the published deploy, or against a preview you've confirmed carries the identical configuration. If the two differ, you're testing a different site.
Now inheritance. Where Netlify lets you set traffic rules at a team level and at a project level, map the relationship as a hierarchy in advance. A team-level rule can apply across many sites, which is convenient and risky: one overly broad block can hit a marketing microsite, a docs site and your storefront at once. A project-level rule is narrower but can conflict with or duplicate what the team already enforces.
Before you add anything, list every rule at both levels and note which projects each reaches. Name an owner for team-level rules, because a change there has a blast radius larger than any single project's on-call can see.
Write the inheritance map down. When a crawler is blocked and nobody knows why, that map is the first thing you'll wish you had.
Request order: firewall traffic rules, WAF, then rate limiting
Order of evaluation decides which rule gets credit for a block, and which rule never sees the request at all. Plan around this sequence: firewall traffic rules first, then WAF, then rate limiting.
In practice, a request that a traffic rule already blocks never reaches rate limiting, so it never counts against a velocity budget. A request that passes the firewall and WAF is the only traffic your rate limit measures. That has two effects you should design for.
First, your rate limit's numbers describe only surviving traffic. If you tighten a traffic rule and see a rate limit's hit count fall, that isn't necessarily an improvement in client behavior. You changed what reaches the limiter.
Second, an allow decision early in the chain doesn't exempt a request from later stages unless the documentation for that control says it does. If you add an allow rule for a partner's address range and the partner still gets throttled, a later stage is acting on them. Read the docs for each stage rather than assuming an allow is global.
Build a simple table for your own site with one row per rule: the stage it runs in, what it matches on, and what it does on a match. Then trace three sample requests through the table by hand: a real reader on a shared network, a crawler you want, and a scraper sending a copied user agent. If you can't predict where each one ends up, you're not ready to deploy.
Keep early rules narrow and specific. Put broad judgment calls later, where you can observe them before they cut off legitimate traffic.
Code versus UI configuration: in-function rateLimit config, precedence and invalid rules
Netlify gives you two places to define limits, and they aren't interchangeable. One is configuration in code. The other is the UI. Decide which is the source of truth for each rule, and say so in the repo's README.
Code-based configuration has clear strengths. It's reviewed in pull requests, it's tied to a specific commit, and rolling back means reverting a commit. It also sits next to the function it protects, so the person changing the handler sees the limit. For per-endpoint velocity caps on a login or search function, put the limit in code.
UI rules have different strengths. You can change them without a deploy, which matters during an incident when the build queue is the last thing you want to wait on. The cost is auditability: a UI change is easy to forget and hard to diff.
Precedence is where teams get hurt. When the same traffic is covered by both a code rule and a UI rule, you need to know which wins. Read Netlify's current documentation on precedence for your plan, then confirm it with a test rather than trusting memory. Two rules that look equivalent can behave differently once one is overridden.
Invalid rules deserve their own check. A malformed rule may be rejected, ignored, or cause a deploy warning depending on where it's defined. A rule that silently does nothing is worse than no rule, because it gives the team false comfort. After every change, confirm the rule is active by sending requests that should trip it and watching the result, not by reading the config file.
Pick one source of truth per rule. Duplicates across code and UI are how a limit you removed keeps firing for weeks.
Verify in logs: Netlify-branded 404 versus 429 or rewrite, block reason and rule IDs
A rule you can't see firing is a rule you can't trust. Build a habit of reading the response, then the log, for every test.
Start with the response a client receives. Different outcomes look different, and the difference tells you what happened. A Netlify-branded 404 page isn't the same as your own application's 404. If a path you know exists returns a Netlify-styled not-found, suspect a routing or rewrite problem, or a rule that's hiding a resource, before you suspect the content. A 429 is a rate limit signal: the client exceeded a velocity budget. A rewrite or redirect may mean a rule sent the request somewhere you didn't intend.
So your first triage question is: which of these did the client get? Capture the status, the body, and the headers for each test request. "It was blocked" is not a finding.
Then go to the logs. Where Netlify exposes it, look for the block reason and the rule ID attached to the event. The rule ID is the thread that connects a symptom to a specific rule in your inventory. Without it you're guessing which of twenty rules acted. Record the ID in your incident notes the first time you see it, and add it to the rule table you built earlier.
Check for the case where nothing fired. If a scraper is still getting through, the log should show the requests passing every stage. That's useful evidence too: it tells you the existing rules don't describe that traffic, and that adding more velocity limits is the wrong response.
Keep the raw evidence before you change anything. Our guide on scraper incident response covers what to preserve so a block doesn't erase the trail.
Test legitimate crawlers and shared IPs, then roll back safely
Every rule you ship has two failure modes: letting bad traffic through and stopping good traffic. Teams test the first and skip the second. Test the second on purpose.
Build a short list of must-not-break clients. Include the search crawlers your SEO team depends on, any AI crawlers your newsroom policy allows, monitoring and uptime checkers, payment and webhook callers, and partner integrations. For each, write down how you'll confirm it still gets a 200. Include shared-address users too: a campus, an office, a mobile carrier. Simulate several distinct visitors from one source address and confirm your velocity limit doesn't lock them all out after a handful of requests.
Be careful with crawlers. A user agent string is a claim, not proof. The pattern is the same one we cover in fake Googlebot: how to spot an impersonator: verify against the operator's published verification method, not the header. Allowing a crawler by name alone opens a door for every client that copies the name. For a structured approach to this testing, see bot false positive testing.
Now plan the rollback before you need it. Write down, for each rule: where it lives (code or UI), who can change it, and the exact steps to disable it. For code rules, that's a revert and a deploy, so know your build time. For UI rules, it's a toggle, so know who has permission at 2 a.m.
Roll out in stages. Apply a new rule to a preview or a low-traffic project first, watch the logs for unexpected rule IDs, then widen scope. If a legitimate crawler or a shared-address cohort starts failing, roll back first and investigate second. A reverted rule costs you an hour of exposure. A rule that blocks your own search crawler for a week costs far more.
Where identity verification has to come from
After the rules are tuned, you'll still face the original gap: Netlify's controls decide on request attributes and velocity, and neither one establishes who is on the other end. A scraper driving a real browser at a polite pace, or a client that copies a crawler's header, can pass every rule you wrote.
That verification has to come from a layer that evaluates the request, the browser and the connection together, and then returns a decision your platform can enforce. That's the job Centinel Analytica does. It tells real visitors from scrapers that look like them, then applies allow, challenge or block policy through your existing CDN or edge, so your current stack stays in place. Its research on fake crawler claims, including the 84% figure for GPTBot on a typical news site, comes from exactly this kind of analysis.
The division of labor is clean. Netlify's rules stay responsible for what they do well: closing paths, capping request rates on expensive endpoints, and enforcing the access limits your plan provides. Identity verification supplies the answer those rules can't: whether a visitor or a self-declared crawler is genuine. Comparable platform-level notes exist for other stacks, such as Vercel bot protection and AWS bot protection, and the same limits apply to each.
If you're choosing between tuning another rate limit and adding verification, count the crawler claims in your logs. If many of them fail verification, another rule won't fix it.
Conclusion
Netlify's controls will stop abusive request rates and close paths you never meant to expose. They won't tell you which crawler claims are true. If your logs show GPTBot or Googlebot traffic you can't account for, stop adding thresholds and measure the problem first.
Bring the rule table and inheritance map from this guide to the 30-minute demo, and we'll show which of your fake-crawler and scraper traffic needs an allow, challenge or block decision applied through your existing edge.
Frequently asked questions
Does Netlify have built-in bot protection?
Netlify provides access and velocity controls: firewall traffic rules, WAF features and rate limiting, with some availability depending on plan. These restrict requests by attributes or rate. They do not establish whether a visitor is human or a crawler is genuine. Readers should verify current plan availability in Netlify's documentation.
How do I rate limit requests on Netlify without blocking real users?
Set limits per endpoint rather than site-wide, and test with several distinct visitors from one shared address to confirm a campus or office is not locked out. Start in a preview, watch logs for rule IDs, and keep a documented rollback. Centinel Analytica adds bot and AI crawler protection that rate limits alone cannot provide.
How can I tell which Netlify rule blocked a request?
Check the response first: a Netlify-branded 404 points to routing or hidden resources, while a 429 points to a rate limit. Then use the logs, where available, to find the block reason and rule ID. Record that ID against your rule inventory so you know which rule acted.
Can Netlify tell if a request is really Googlebot or GPTBot?
A user agent string is only a claim, and rules that match on it treat spoofed and genuine requests alike. Centinel Analytica's research found 84% of requests claiming to be GPTBot on a typical news site were fake. Verification must come from a layer that evaluates the request, browser and connection together.
What is the order of Netlify's firewall, WAF and rate limiting?
Plan around firewall traffic rules first, then WAF, then rate limiting. A request blocked early never reaches the rate limiter, so rate limit counts reflect only surviving traffic. Confirm the current behavior in Netlify's documentation and trace sample requests through your own rules before deploying.