Wordfence bot protection counts requests per visitor at the PHP layer and throttles or blocks whoever crosses your thresholds. Set the crawler and human page-view limits, the 404 limit and the action from your own logs, and exempt only verified Google crawlers, never any client that merely says it is Googlebot.
That is also where the plugin stops. Requests answered from a CDN cache never reach WordPress, so they are never counted. This guide covers each setting, how to choose values for your site, and how to decide whether PHP-layer limits are enough or whether enforcement needs to sit upstream. Every number below is a placeholder to test, not a safe default.
Short answer: what Wordfence rate limiting can and cannot do for crawlers
Wordfence rate limiting can slow or stop crawlers that hit your WordPress origin too fast. It cannot see traffic that never reaches WordPress. It runs inside the plugin's firewall, so it evaluates only the requests PHP actually handles. The Wordfence rate limiting help page is the reference for what each setting does. Read the current version before changing anything, because option names and defaults change between releases.
Three jobs fit well. It can cap runaway crawlers that walk your archive pages at high speed. It can catch scanners that probe for files that do not exist and generate floods of 404s. And it can keep a misbehaving but non-malicious bot from exhausting PHP workers.
Three jobs fit badly. It cannot tell a real crawler from a copy of its user agent without a verification step. It cannot see cache hits served by a CDN or page cache in front of PHP. And it is not a scraping defense: a patient scraper that stays under your thresholds and rotates addresses will pass.
Treat the plugin as one layer. For login and XML-RPC specifically, see the WordPress bot protection guide. This post stays on page views, 404s and crawler handling.
Crawler versus human page-view limits and 404 thresholds
Wordfence splits traffic into two buckets, and you should set the two limits differently. One limit applies to anything the firewall identifies as a crawler, the other to visitors it treats as human. Humans click, read, and click again, so their limit can be loose. Crawlers fetch pages in tight bursts, so their limit is where a mistake costs you search visibility.
Start with the human page-view limit, because it is the safer one to get right. A reader rarely requests dozens of pages a minute. But a reader on a slow connection with several tabs, or a visitor behind a shared office or mobile carrier address, can look busy. Set the limit above the busiest legitimate pattern in your logs, not above your average.
The crawler limit needs more care. Search engines and monitoring services are crawlers too, and a limit tight enough to stop an abusive bot can also slow Google. That is why the next two sections cover action choice and verification before you tighten anything.
The 404 threshold is the most useful of the three for catching probing. A visitor who requests many missing URLs in a short window is usually scanning for old plugin paths, backup files or admin panels. Real users hit the occasional dead link. Before you set it, check how many 404s your own site legitimately serves. A site that recently migrated or changed its URL structure will have crawlers and old inbound links producing bursts of 404s that look hostile and are not.
One rule applies to all three. Pick a starting value from your log data, apply it with the most lenient action, and watch what it catches. If you cannot name the traffic a limit would have stopped, you don't yet have a reason for that number.
Throttle or block: choosing the action and block duration
Throttle first, block second, and keep block durations short until the log shows you are catching the right traffic. Throttling slows a client that crossed the line. Blocking denies it for a set period. The two fail differently.
A throttled legitimate crawler recovers on its own. It gets slowed, backs off, and returns to normal. A blocked legitimate crawler may mark your site unreachable, and recovery takes as long as the crawler's own retry schedule, which you don't control. For anything that might be a search engine, that asymmetry is the whole argument for throttling.
Blocking earns its place for traffic you have already judged hostile, such as a client producing hundreds of 404s against paths no legitimate visitor requests. Even then, start with a short duration. A long block applied to a shared address, like a university network or mobile carrier gateway, also locks out innocent users behind it. Lengthen the duration only after you have seen repeat offenders return from the same sources.
A practical order of operations:
- Set the 404 threshold with a throttle action and review what it catches for a week.
- Promote to a short block only for patterns that are clearly scanning, such as repeated requests for nonexistent file types.
- Keep the crawler page-view limit on throttle until verification is working, covered next.
- Revisit durations after you have seen how often blocked addresses return.
Write down what each setting was before you changed it. Rollback is the cheapest safety feature you have.
Verified Google crawlers versus anyone claiming Googlebot
Exempt Google only when the request is verified as Google, never because the user agent string says Googlebot. A user agent is a self-reported label. Any script can send it, and scrapers do, precisely because sites tend to wave Googlebot through.
That is the trap in any allowlist keyed on a name. If your rate limit says "skip Googlebot" and the check is a string match, you have built a free pass for every scraper willing to type that string. Wordfence's rate limiting settings include an option that separates real Google crawlers from clients that only claim to be Google. Check the current wording on the rate limiting help page and use the verified option rather than a user agent exemption.
Google documents how to verify its own crawlers independently: a reverse DNS lookup on the requesting address, a forward lookup on the resulting hostname, and confirmation that it resolves back to the same address. Google also publishes IP range lists for its crawlers. Use these to spot-check addresses in your logs, so you can see whether the traffic labelled Googlebot in your access log matches what the plugin treats as verified.
Keep the scope narrow. Verification methods are specific to each operator. What works for Google does not transfer to OpenAI, Anthropic, Perplexity or any other crawler, each of which publishes its own method, and some publish none a site can check. Don't apply "verified Google" logic to other bots, and don't assume an AI crawler named in a user agent is the real one. For one worked example of checking a named AI crawler, see What is ClaudeBot?.
For your settings, that means exempting verified Google crawlers from the aggressive limits if you want to protect search crawling, and leaving every unverified client subject to them, including the ones wearing Google's name.
Where PHP-layer limits stop: CDN cache hits, Live Traffic and upstream enforcement
Wordfence counts what reaches PHP, so a request answered by a CDN or a full-page cache never enters its count. Put a CDN in front of WordPress with caching on, and a crawler hammering cached pages may look quiet to Wordfence while it still eats bandwidth and copies your content. Whether a given request reaches the plugin depends on your cache configuration, so verify it for your own stack rather than assuming.
The test is simple. Request a cacheable page from a client you control, then look for it in the plugin's records. If a page you fetched does not appear, it was served before PHP ran, and no Wordfence limit applies to it.
Live Traffic is the plugin's view of visitors and crawlers hitting the site, and it has the same boundary. It shows what the plugin observed, not everything your CDN served. Use it to see which clients are tripping limits and what they request, and read its output as a record of origin traffic, not a full picture of crawler load.
The address problem sits on the same boundary. Behind a proxy or CDN, the connecting address Wordfence sees can be the proxy's, not the visitor's, unless the plugin is configured to read the forwarded client address correctly. If that is wrong, rate limits can lump many visitors together or exempt the wrong ones. Check the setting that tells Wordfence how to obtain the visitor's IP before trusting any per-client limit. The same real-address reasoning, applied at the web server, is covered in the Nginx bot protection guide.
That boundary is why PHP-layer limiting and edge enforcement are different jobs, not rival ones. The plugin protects the application that runs behind the cache. Anything that must be decided before the cache answers has to be decided upstream.
Inspect logs and test legitimate workflows before tightening
Look at a week or two of real traffic before you change a threshold, and test your own workflows after. Without that baseline, every number is a guess.
Start with the evidence you already have. From server access logs and Live Traffic, pull the busiest clients by request count, the clients with the most 404s, and the user agents claiming to be search crawlers. For each cluster, ask whether you can explain it. Your monitoring service, your own feed reader, a partner's integration and a search engine are all expected. A cluster you can't explain is the one to investigate before limiting.
Then list the legitimate workflows that generate bursts:
- Search engine crawling after a sitemap update or large content publish
- Uptime and performance monitors that poll on a schedule
- Editors previewing and saving many pages in one session
- Payment, shipping or newsletter services calling back into the site
- Your own staging or QA scripts
For each, work out whether the limit you are about to set would catch it. A monitor that checks every ten seconds is not abuse, and neither is a publishing day where an editor opens forty drafts.
After a change, retest. Browse as a normal reader. Run your monitor. Use Google's own tools to confirm a page can still be fetched. If blocked requests rise and you can't attribute them to hostile traffic, the limit is too tight. For a fuller method, bot false positive testing walks through verifying visitors before blocking them.
Keeping Wordfence resource use bounded
A security plugin that slows the site it protects has failed, so budget its overhead as deliberately as its rules. Wordfence documents the settings that affect its own consumption on its resource tuning page, and that page is the source to follow for current option names and recommendations.
Match the plugin's work to your hosting. Scans, logging and live monitoring all cost CPU, memory and database writes. On modest shared hosting, the heavy options can compete with page generation. Review what you have enabled, lengthen or reduce anything you don't act on, and don't turn on every logging option by default.
Logging deserves its own caution. Recording every request is useful during an investigation and expensive in steady state, because it grows the database and adds writes to each page load. Turn up detail when you are diagnosing a crawler problem, then turn it back down.
Also keep rate limiting from becoming the load. A flood of requests that the plugin has to evaluate one by one still uses PHP workers, even if each is denied quickly. If a crawler produces enough volume that denying it costs real capacity, move enforcement upstream rather than tuning PHP harder.
When to add edge enforcement beside your CDN
Add enforcement at the edge when the traffic you care about is decided before WordPress runs, or when volume itself is the problem. Three signs point that way: scraping that stays under your PHP thresholds, crawlers hitting cached pages the plugin never sees, and bots that claim a trusted identity you can't verify from inside the plugin.
Centinel Analytica is bot and AI crawler protection that tells real visitors from scrapers that look like them, then applies allow, challenge or block policy through your existing CDN or edge. It does not replace Wordfence or integrate with it. The plugin keeps handling what reaches WordPress, and edge policy handles what should be decided earlier.
For crawlers, identity is the problem. Published Centinel research found that on a typical news site, 84% of requests claiming to be GPTBot were fake. A name in a user agent is not proof. That is the same lesson as Googlebot above, applied to every crawler that matters to your content.
If you already run a CDN, the placement question is where checks sit relative to the cache. The Fastly bot protection guide covers pre-cache and post-cache decisions in detail, and the same reasoning applies to other CDNs.
Conclusion
Wordfence is a sound origin-side control when you set it from your own logs: loose human limits, careful crawler limits, a 404 threshold that catches probing, throttle before block, and exemptions only for verified Google crawlers. Its edge is the PHP boundary, and anything served from cache or decided before WordPress runs needs enforcement elsewhere.
Frequently asked questions
Does Wordfence block Googlebot by default?
Wordfence can slow or block any client that crosses your configured limits, including a real search crawler if the limits are too tight. Use the option that exempts verified Google crawlers, and keep crawler limits on throttle until you have checked your logs. Do not exempt by user agent string alone, because scrapers copy it.
How do I tell real Googlebot from a fake?
Do a reverse DNS lookup on the requesting IP, then a forward lookup on the hostname and confirm it returns the same address. Google also publishes crawler IP ranges you can compare against. A user agent saying Googlebot proves nothing on its own. Edge tools such as Centinel Analytica can apply the same distinction before traffic reaches WordPress.
Why doesn't Wordfence rate limiting see traffic served by my CDN?
Wordfence is PHP-based, but Extended Protection can run before WordPress loads and process PHP requests; Basic WordPress Protection runs after WordPress loads and may miss direct requests to files that bypass WordPress. Pages answered from a CDN or full-page cache never get that far. Test it by fetching a cacheable page and checking whether it appears in the plugin's records. For decisions before the cache, you need enforcement at the edge.
What rate limit values should I use in Wordfence?
There is no universal safe value. Pull a week or two of access logs and Live Traffic, find your busiest legitimate clients, and set limits above that pattern. Start with throttling, review what each limit catches, and only then consider short blocks. Retest search crawling and your own workflows after each change.
Is Wordfence enough to stop AI scrapers?
Not by itself. It limits request rates at the PHP layer, but a scraper that stays under your thresholds or hits cached pages will pass, and AI crawler names in user agents are easy to fake. Centinel Analytica adds bot and AI crawler protection through your existing CDN or edge, alongside Wordfence.