A team turns on a Fastly bot challenge for a checkout flow, watches the dashboard light up with challenge responses, and calls the job done. Two weeks later the scraper traffic on the same endpoints has not moved. Nothing was enforced. The challenge went out, the client ignored it, and the origin answered the next request anyway.
This is the most common failure in Fastly bot protection rollouts, and it comes from misunderstanding where Fastly looks at a request. Some checks run before the cache and judge who is asking. Others run after the cache and judge what that client does at a specific endpoint. They are different tools with different failure modes, and each needs its own handling of cookies, caching and hostnames.
This guide is for platform and security engineers who run Fastly services and have to decide where each check belongs. It covers the two inspection points, what each deployment requires, how to scope rules, the header details that quietly break challenges, and a test plan you can run before enforcing anything. Check product specifics against Fastly's current documentation, because plans and defaults change.
Pre-cache vs post-cache: when Fastly inspects the request
Fastly processes a request in stages, and the stage decides what a bot check can see and what it can protect. Pre-cache inspection happens on the way in, before Fastly decides whether a cached object can answer. Post-cache inspection happens after that decision, on requests that need to proceed toward the origin or an endpoint rule.
Pre-cache is where category and identity policy belongs. At this stage the question is coarse: is this a known crawler, an automation tool, or something unclassified, and is its claimed identity believable? The advantage is reach. A pre-cache decision applies to every request, including those a warm cache would have served without ever touching your origin. The cost is that the decision rests on request-level evidence alone, made before any challenge exchange.
Post-cache is where endpoint enforcement belongs. A login POST, a cart add, a search API: these are the actions you actually need to protect, and they need a stateful exchange. The client fetches a page, receives a challenge, solves it, and presents proof on the next request. That exchange cannot live in a pre-cache category rule, because a category rule has no concept of a solved challenge.
Take a position on this early. Don't protect an endpoint with a pre-cache rule, and don't classify bot categories with an endpoint rule. A news site that wants GPTBot handled differently from a spoofed GPTBot needs identity policy up front. A retailer that wants to stop credential stuffing needs token verification on the login POST. Most real deployments need both, configured as two separate decisions.
What each deployment requires (Bot Management, Edge WAF, per-service setup, no Compute)
Before you design rules, confirm what your contract and your services actually include. Fastly's bot capabilities are not a single switch, and the pieces you need depend on which inspection point you are using.
Three requirements come up in almost every scoping call:
- Entitlements. Pre-cache category and identity policy depends on having the appropriate features enabled for your account. Endpoint enforcement runs through the Edge WAF layer (Fastly's Next-Gen WAF edge deployment). Confirm both in writing with your account team. One does not imply the other.
- Per-service setup. Fastly Next-Gen WAF edge configuration is not exclusively per service: deployment and policy management also use account- and workspace-level resources, while enabling the WAF and configuring traffic inspection are tied to a specific CDN service. If you run separate services for www, api and a checkout subdomain, each one needs its own configuration, its own testing and its own rollback. A rule that works on one service does nothing for its neighbor.
- No Compute dependency. This pattern works on VCL-based Fastly CDN services. You don't need to rewrite anything for Fastly Compute to get pre-cache policy or post-cache endpoint rules. If someone on your team proposes a Compute migration as a prerequisite, push back and ask which specific capability requires it.
The per-service point causes the quietest failures. Teams validate on a staging service, copy the settings by hand to production, and miss one field. Keep the service configuration in version control or at least in a written checklist, and diff production against it after every change.
If you are comparing how other platforms split this work, the same pre-decision versus endpoint-decision pattern shows up elsewhere, for example in Akamai bot score policies and AWS WAF Bot Control levels. The names differ. The staging problem is identical.
Pre-cache policy: bot category, identity and enforcement mode
Pre-cache policy has three dials: which bot category a request falls into, whether its claimed identity holds up, and what the service does about it. Treat them as three separate decisions.
Category. Decide what you want per class: verified search crawlers, AI crawlers, monitoring tools, scraping and automation tools, and unclassified traffic. Write this down before touching configuration. A publisher might allow verified search crawlers, allow selected AI crawlers under a stated policy, and challenge or block automation tools. An ecommerce site might allow price comparison partners by agreement and treat everyone else as untrusted.
Identity. A user agent string is a claim, not proof. Centinel's own research on a typical news site found 84% of requests claiming to be GPTBot were fake. Any policy that says "allow GPTBot" and matches on the string alone is allowing the impersonators. Your allow rules must depend on verified identity, not the header. The fake Googlebot guide walks through the verification methods.
Enforcement mode. Start in a non-blocking mode that records what would have happened. Run it for at least a full traffic cycle that includes your weekly peak and any scheduled partner jobs. Review the would-block list by hand. You are looking for your own uptime monitors, your SEO tooling, payment webhooks and partner integrations. Move to enforcement one category at a time, beginning with the category you are most certain about.
Don't flip the whole policy to block on the same day you enable detection. Detection tells you what Fastly sees. It does not tell you what you are about to break. Keep a documented rollback path (a single setting change you can make in minutes) before you enforce anything.
Post-cache endpoint enforcement: challenge on GET, verify token on POST, block invalid tokens
Endpoint enforcement is a three-part contract, and a challenge alone delivers one-third of it. The parts: serve a challenge on a GET, verify the resulting token on the POST, and block POSTs that arrive without a valid token.
The flow for a login form looks like this:
- The client requests the login page with a GET. The endpoint rule serves the challenge as part of that response.
- The client's browser runs the challenge and receives a token, typically carried in a cookie.
- The client submits credentials with a POST. The rule verifies the token on that request.
- A POST with a missing, expired or invalid token is blocked before it reaches your origin.
Step four is the one teams skip. They configure the challenge, see challenge responses in the logs, and assume the endpoint is protected. But a scraper or credential-stuffing script that never renders the page never sees the challenge. It just sends the POST directly. If nothing verifies the token, the origin accepts the request. The challenge was a suggestion, not a gate.
The rule of thumb: sending a challenge is not enforcement. Rejecting requests that fail verification is enforcement. Audit this by replaying a bare POST with no cookies from a command line. If your origin processes it, you have a challenge and no enforcement.
This split also explains why GET and POST need different rules. The GET rule is permissive by design, because it has to let a new visitor receive the challenge. The POST rule is strict, because it guards the action you are protecting. One rule covering both either blocks first-time visitors before they can solve anything, or lets unverified POSTs through. Keep them separate and name them clearly in your configuration so the next engineer can tell which is which.
Scope rules by hostname, method and path (and why a www solve does not validate the api hostname)
Every endpoint rule should be scoped on three axes: hostname, HTTP method and path. Leave any of them open and the rule ends up either too broad, hitting traffic you never meant to touch, or too narrow, leaving the real target unprotected.
Hostname is the axis that surprises people. A challenge solved on www.example.com produces a token tied to that hostname. If your single-page app then calls api.example.com, the browser does not carry the www token to the api hostname, because cookies are scoped to the host that set them unless a parent domain was explicitly specified. The visitor solved the challenge, and the API still sees an unverified request. Teams respond by loosening the API rule to let it through, which removes the protection they wanted.
Handle it deliberately. You have three workable options:
- Run the challenge flow on the api hostname itself, with its own GET that serves the challenge and its own POST verification.
- Set the verification cookie on a shared parent domain, and confirm in testing that the api hostname actually receives and accepts it. A broader cookie scope is a broader attack surface, so know what you are trading.
- Keep sensitive actions on the same hostname as the page that serves the challenge.
Whichever you choose, test it from a real browser session across both hostnames. A curl script you have manually fed a token proves nothing.
For method and path, write the narrowest rule that covers the action. A POST to /login and /account/password-reset is a specific list. "All POSTs" will eventually catch a webhook from your payment provider or a partner callback. Maintain an explicit exclusion list for machine-to-machine paths, and review it whenever a new integration ships. Document each exclusion with an owner and a reason.
Preserve Cookie and Set-Cookie headers and keep challenge responses uncached (private, no-store)
Two header problems account for a large share of broken Fastly challenge rollouts: stripped cookies and cached challenges. Both are invisible in a quick smoke test and both fail loudly in production.
Cookies must survive the trip. The challenge flow depends on the Cookie request header reaching the verification step, and on the Set-Cookie response header reaching the client. Plenty of existing Fastly configurations remove cookies to improve cache hit rates. Stripping the Cookie header on requests, or removing Set-Cookie on responses, for a path you are now trying to protect will break the exchange. The client solves the challenge, the token never lands, and every subsequent POST fails verification. Real users get blocked while scrapers shrug.
Audit your existing VCL and any header rules for cookie handling on the protected paths. Preserve both directions for those paths specifically. Leave your cookie stripping in place for static assets, where it still helps.
Challenge responses must never be cached. A challenge page is a per-client response. If Fastly caches it and serves one visitor's challenge to the next, you get stale or mismatched challenges, and in the worst case one person's token response gets handed to someone else. Mark challenge responses with Cache-Control: private, no-store so neither the shared edge cache nor any downstream cache stores them. Then verify it: request the protected path twice from two different clients and compare the responses and the cache status headers.
A concrete before and after. Before: a login GET is cacheable for 60 seconds and the first visitor's challenge is served to the next 200 visitors, none of whom can complete it. After: the same path returns private, no-store, every visitor gets a fresh challenge and the hit rate on the login page drops to zero, which is correct. A login page was never a good caching candidate anyway.
Vendor quick-start guides skip these details. Check them first when something works in testing and fails in production.
Private Access Token limits, dynamic vs selected challenge types, and a safe test plan
Challenge type is a real configuration decision, and it interacts with the limits of device-level attestation.
Private Access Tokens (PATs). PATs let an attesting device vouch for a client without showing the user a puzzle. They are useful, but not universal. Support depends on the client's operating system, browser and vendor ecosystem, and many automation setups and older devices will not present one. Don't build a flow that assumes every legitimate visitor can produce a PAT. Keep a fallback path for clients that cannot, and check Fastly's current documentation for which clients and conditions apply to your service.
Dynamic versus selected challenge types. A dynamic setting lets the platform choose the challenge it considers suitable per request, which cuts your maintenance burden and adapts as evasion tooling changes. A selected setting fixes a specific challenge type, which gives you predictability and a clean test surface. Pick dynamic if you want less tuning and can tolerate variation in what visitors see. Pick selected if you need a reproducible flow for accessibility review or compliance sign-off.
A safe test plan. Run these in order, and don't skip the early steps:
- Deploy to a single low-risk path in non-blocking mode and log decisions for a week.
- Replay a bare POST with no cookies and confirm it would be rejected once enforcement is on.
- Solve the challenge in a real browser on each hostname you scoped, and confirm the token is accepted on each.
- Test from a mobile browser, a privacy-hardened browser and a screen reader.
- Confirm challenge responses carry
private, no-store. - Enable enforcement on that one path, watch conversion and support tickets, then expand.
Use the bot false positive testing guide to structure step four. Keep a rollback you can execute in minutes.
Where a layer beside Fastly fits when browser-like scrapers pass these checks
Pre-cache policy and endpoint challenges stop a lot of traffic, and they still leave one class of visitor standing: scrapers that run real browsers, solve challenges, and pull content at a pace that looks like reading. A challenge proves a browser executed some code. It does not prove the browser belongs to a person.
This matters most for publishers and ecommerce teams with unique content. Article text, product catalogs and pricing are retrievable from public GET pages, and the high-value scrapers are the ones built to pass exactly the checks described above. They hold valid tokens, they present plausible browser details, and they arrive over residential connections. If you suspect this is your problem, the residential proxy scraper detection guide shows how to test it against your own site rather than a vendor demo.
This is the gap Centinel Analytica is built for. It tells real visitors from scrapers and applies allow, challenge or block policy through your existing CDN or edge. Fastly stays in place, along with your pre-cache policy and endpoint rules.
In practice, Fastly keeps doing its job at the edge, and Centinel's verdict becomes an extra input for the traffic that clears the first two layers. Start by finding out how much of that traffic you have.
Conclusion
Treat Fastly bot protection as two decisions, not one. Pre-cache policy decides who is asking, endpoint rules decide what a client may do, and neither is working until your origin rejects a bare POST with no token. Fix the cookie and no-store details first, scope by hostname so a www solve doesn't leave the api hostname open, and enforce one path at a time.
Then measure what still gets through. If browser-like scrapers are passing your Fastly checks, evaluate what gets through before deciding whether a layer beside Fastly is worth adding. Or book the 30-minute demo to walk through your Fastly setup with the team.
Frequently asked questions
What is the difference between pre-cache and post-cache bot checks on Fastly?
Pre-cache checks run before Fastly decides whether a cached object can answer, so they apply to every request and suit bot category and identity policy. Post-cache checks run on requests proceeding toward an endpoint and suit challenge and token verification on actions like login. Most deployments need both, configured separately.
Why does a bot challenge not stop scrapers by itself?
A challenge only works if something verifies the result. Scripts that skip the page and send a POST directly never see the challenge, so unless a rule rejects POSTs without a valid token, the origin still accepts them. Test by replaying a bare POST with no cookies and confirming it is blocked.
Why does a challenge solved on www not work on my api hostname?
Cookies are scoped to the host that set them unless a parent domain was explicitly specified. A token earned on www is not automatically sent to the api hostname. Run the challenge flow on the api hostname, share a parent-domain cookie after careful testing, or keep sensitive actions on one hostname.
Do I need Fastly Compute to run bot protection?
The pattern in this guide targets VCL-based Fastly CDN services and does not require a Compute migration. Confirm your Bot Management and Edge WAF entitlements with Fastly, set up each service individually, and verify current requirements in Fastly's documentation before scoping the project.
What should I do if scrapers still get through Fastly's bot checks?
Browser-based scrapers can solve challenges and hold valid tokens, so a pass does not prove a human visitor. Centinel Analytica tells real visitors from scrapers and applies allow, challenge or block policy through your existing CDN, including Fastly.