When a wave of aggressive scrapers hits your origin, HAProxy is usually the first line of defense that doesn't fall over. Its event-driven core can track millions of connections without breaking a sweat, so it's tempting to push all bot mitigation logic straight into your load balancer config.

Community HAProxy has fast primitives for tracking abusive behavior. With stick tables, access control lists, and runtime maps, you can stop blunt denial-of-service attempts and aggressive content scraping in microseconds. But treating HAProxy like a full bot management engine is how production outages happen.

Effective haproxy bot protection requires understanding exact sliding window mechanics, key storage, and network boundaries. It also requires clarity on what simple rate counting can never catch. That clarity saves your engineering team from maintaining fragile config scripts every time attackers change tactics.

What community HAProxy can and cannot do about bots (and how HAProxy Enterprise bot management differs)

Community HAProxy is an excellent traffic director and layer 7 policy engine. It is not a behavioral security platform. In the open-source edition, bot defense relies on deterministic traffic inspection: request paths, HTTP headers, request rates over time, TCP connection velocity, and IP reputations loaded into memory. You define strict thresholds, and HAProxy drops or tarpits traffic that violates them.

HAProxy Enterprise includes specialized security modules designed to classify automated clients. The commercial tier also bundles managed bot detection feeds and dedicated web application firewall layers that inspect payloads dynamically.

Community HAProxy focuses on request inspection rather than client-side browser challenges. It cannot execute JavaScript puzzles or inspect canvas rendering anomalies. If a scraper rotates residential proxy IPs and mimics regular user navigation cadences, open-source stick tables can't tell that scraper apart from a real customer.

What community HAProxy does well is high-throughput, rule-based policing. It drops misbehaving scrapers, caps burst traffic at the network edge, and defends authentication routes against volumetric brute-force attacks. Open-source HAProxy works on explicit request telemetry, not behavioral client profiling. Once you accept that, building stable policies gets much easier.

Stick-table anatomy: type, size, expire and the stored counters

Every rate-limiting policy in HAProxy begins with the stick-table directive. A stick table is an in-memory key-value store built for concurrent, non-blocking lookups inside HAProxy worker processes. Misconfigure its parameters and you'll either exhaust system memory or evict active threat records too early.

A typical stick table definition requires four core parameters: name, key type, maximum size, and expiration time:

backend st_per_ip stick-table type ip size 1m expire 10m store http_req_rate(10s),http_err_rate(10s),conn_cur

The key type determines the indexing mechanism. type ip uses a 4-byte address key to store IPv4 addresses only, meaning it cannot represent IPv6 addresses. type ipv6 uses a 16-byte address key and can store IPv6 addresses or IPv4-mapped IPv6 addresses. For tracking tokens or session cookies, type string is available, though it requires setting a maximum key length.

The size parameter caps total stored records. size 1m reserves room for one million concurrent keys. If your site gets traffic from millions of distinct IP addresses, the table uses an LRU eviction strategy: older entries drop out to make room for new ones, which clears counters for slow-moving scrapers. Plan for roughly 64 to 128 bytes per entry depending on the counters stored. A table with one million entries takes roughly 100 megabytes of RAM.

The expire setting controls how long an idle key stays in memory after its last tracking update. Set it too short (say, 30 seconds) and a persistent scraper just pauses between bursts to clear its record. Set it too long and benign client IPs hog table capacity long after their sessions end. A window between 5 and 15 minutes is typical for rate tracking.

track-sc0 and sc_http_req_rate: how the window actually behaves

To track a visitor, HAProxy ties incoming requests to a stick table slot using tracking counters, historically named track-sc0, track-sc1, and track-sc2 (or modern numbered slots like track-sc(0)). Tracking has to happen before you evaluate rate limits, typically in an http-request track-sc0 directive within your frontend or backend.

Many engineers assume http_req_rate(10s) calculates a rigid ten-second bucket that resets to zero every ten seconds. It doesn't. HAProxy uses an internal sliding window algorithm based on two adjacent periods. It tracks the count in the current period and blends it with the count from the preceding period, weighting the average by elapsed time.

This smoothing prevents the classic rate-limit hole where an attacker sends twice the allowed requests right at the border of a fixed window. It also means sc_http_req_rate(0) is an estimate, not an absolute integer tick count. If an IP sends 100 requests in one second and stops, its measured rate decays smoothly across the configured window instead of dropping off instantly.

frontend fe_main bind :443 ssl crt /etc/ssl/site.pem http-request track-sc0 src table st_per_ip http-request deny status 429 if { sc_http_req_rate(0) gt 50 } default_backend be_app

Tracking increments before evaluation happens. If a rule denies requests above 50, the 51st request increments the table counter and triggers the rejection rule immediately.

Per-route thresholds with maps and the correct 429 deny

A single static rate limit across an entire application causes real usability problems. An interactive catalog with image carousels and API calls naturally produces higher burst rates than an account creation endpoint or a checkout submission. Hardcoding dozens of ACLs across multiple route paths makes HAProxy configs unmaintainable.

The clean pattern is pairing stick tables with HAProxy runtime map files. A map file stores key-value pairs that HAProxy searches using prefix, exact, or regex matching.

Create a map file at /etc/haproxy/maps/rates.map containing your sensitive paths and their corresponding limits:

/api/v1/auth/login 5 /checkout/pay 3 /catalog/search 40 /products 80

In your HAProxy config, look up the target path in the map. If the path matches, compare the client's current request rate against the retrieved numeric value. You can read more about coordinating endpoint rules across proxies in our guide on Nginx bot protection.

frontend fe_web bind :443 ssl crt /etc/ssl/cert.pem http-request track-sc0 src table st_per_ip

Map lookup returns integer limit

http-request set-var(req.rate_limit) path,map_beg(/etc/haproxy/maps/rates.map,0)

Check if route has an explicit limit and client exceeds it

acl route_limited var(req.rate_limit) -m int gt 0 acl rate_exceeded var(req.rate_limit),sub(sc_http_req_rate(0)) -m int lt 0

http-request deny status 429 default-errorfiles if route_limited rate_exceeded

When rejecting bots, return HTTP 429 (Too Many Requests) instead of a generic 403 or 500. Add a standard Retry-After response header when possible. A 429 tells well-behaved automated clients to back off and keeps your metrics clean in upstream log aggregators.

Trusted client identity: PROXY protocol, X-Forwarded-For and shared-IP false positives

Tracking client IP addresses via src breaks the moment HAProxy sits behind an edge CDN, cloud load balancer, or reverse proxy. In that setup, src evaluates to the upstream CDN edge node, so HAProxy rate-limits the CDN and takes down your whole site.

To identify the true client IP, make sure your edge provider speaks PROXY protocol v2 to HAProxy. The PROXY protocol prepends connection metadata before the TLS or HTTP handshake, so HAProxy can set its internal src primitive directly to the client IP without inspecting HTTP headers.

If your architecture can't use PROXY protocol, extract the real IP from the X-Forwarded-For header. Never trust the raw header blindly, because any client can spoof arbitrary values. Tell HAProxy to accept X-Forwarded-For only from verified upstream proxy subnets:

acl from_trusted_cdn src 192.0.2.0/24 198.51.100.0/24 http-request set-var(req.client_ip) req.fhdr(x-forwarded-for,-1) if from_trusted_cdn http-request track-sc0 var(req.client_ip) table st_client_ips if from_trusted_cdn

Even with accurate IP extraction, relying only on IP addresses causes false positives on shared network infrastructure. University campuses, corporate offices, mobile cell towers, and public Wi-Fi hotspots route hundreds of real users through single carrier-grade NAT (CGNAT) gateways. Low per-IP rate limits on catalog browsing will lock out genuine buyers.

Before dropping traffic outright, check whether the client claims to be a known search engine. Spurious web bots frequently forge user-agent strings. To avoid locking out real indexing bots while stopping impersonators, pair your stick-table limits with fake Googlebot verification.

Inspecting tables and the reload and peers caveats

Debugging stick tables during an active incident means using the HAProxy Runtime API via the administrative UNIX domain socket. With socat, you can inspect active entries, dump rate metrics, and drop offending keys without restarting the process:

echo "show table st_per_ip" | sudo socat stdio /var/run/haproxy.sock echo "show table st_per_ip data.http_req_rate gt 30" | sudo socat stdio /var/run/haproxy.sock echo "clear table st_per_ip key 203.0.113.45" | sudo socat stdio /var/run/haproxy.sock

Stick tables carry a nasty hazard during service reloads. When HAProxy runs a hitless reload (systemctl reload haproxy), new worker processes spawn to handle new connections while old workers finish existing tasks. By default, stick-table state in community HAProxy lives in process memory and disappears when workers terminate.

To persist stick-table state across seamless reloads and share tracking counters across multiple HAProxy instances in an active-active cluster, configure a peers section:

peers local_cluster peer lb1 127.0.0.1:10240 peer lb2 10.0.1.12:10240

backend st_per_ip stick-table type ipv6 size 1m expire 15m peers local_cluster store http_req_rate(10s)

Connecting HAProxy to local peer instances syncs table state over a dedicated TCP stream. When an old worker terminates after a reload, the incoming worker pulls historical counters directly from the peer process. Scrapers can't reset their rate penalty windows just because you pushed a config change.

Where per-IP limits stop: low-rate distributed scraping and a rollout and test plan

Per-IP rate limits assume the attacker generates disproportionately high traffic volume from a concentrated pool of network addresses. Modern content scraping skips that assumption entirely.

Scraping frameworks rent pools of millions of residential proxy IPs. An attacker crawling an entire e-commerce catalog can rotate IP addresses on every single HTTP request. To your HAProxy stick tables, this looks like millions of distinct visitors making one request every five minutes. The counter for http_req_rate(10s) never exceeds 1, so table thresholds do nothing. If you're struggling with low-volume crawling, review our analysis on price scraping protection.

Scrapers also run headless Chromium or custom automation tools to mimic human navigation patterns. When basic IP throttling fails against distributed bots, you need behavioral classification and client-side proof-of-work challenges. That's where teams deploy dedicated edge protection like Centinel Analytica.

Centinel Analytica works alongside Cloudflare, CloudFront, Akamai, and Fastly, so your existing CDN and HAProxy stack stays intact while it identifies scrapers and AI crawlers that fly beneath per-IP thresholds. On a typical news site, 84% of requests claiming to be GPTBot were fake (Centinel Analytica, 2026). Dropping those unauthorized bots means verifying authenticity before they hit origin infrastructure.

Before enforcing aggressive HAProxy rules, run a phased rollout:

Audit your baseline traffic logs for peak per-route request velocities.
Deploy stick tables with http-request track-sc0, but use http-request capture or log flags to record would-be rejections instead of denying traffic.
Validate the rules against common false-positive vectors like corporate NATs and search engine crawlers.
Shift the action from logging to http-request deny status 429 on sensitive endpoints first.

Conclusion

HAProxy stick tables remain the right tool for baseline rate limiting, volumetric DDoS mitigation, and origin shielding. But when sophisticated scrapers spread their requests across massive proxy swarms, static IP limits hit a wall.

If distributed AI crawlers and automated scraping tools are draining your origin resources and bypassing your HAProxy rules, see how Centinel Analytica handles low-rate bots. It plugs into your existing edge setup to tell real visitors from scrapers, and it enforces allow, challenge, or block policies without making you rip out your load balancer.

Frequently asked questions

Can community HAProxy block bots using CAPTCHA challenges?

Community HAProxy focuses on direct traffic inspection and control. It handles deterministic actions like deny, silent drop, tarpit, or redirect. To challenge visitors without replacing your load balancer, integrate edge solutions like Centinel Analytica alongside your existing CDN or WAF.

How do stick tables handle IPv6 client addresses?

Stick tables defined with `type ip` store IPv4 address keys only. To track IPv6 clients, configure your table with `type ipv6`. HAProxy automatically stores both native IPv6 addresses and IPv4-mapped IPv6 addresses in this format, preventing memory corruption or missed detections from modern dual-stack networks.

Why do my stick-table rate limits reset when HAProxy reloads?

When HAProxy performs a reload, new worker processes start with empty tables. To retain tracking state across reloads and sync counters across clusters, define a `peers` section in your configuration and bind your stick tables to the local peer group.

How can I test HAProxy rate limits before blocking real traffic?

Rather than issuing an immediate `http-request deny`, configure an ACL that matches your rate-limit condition and set an HTTP transaction variable or log flag using `http-request set-var`. Monitor your access logs to evaluate how many legitimate visitors would have been impacted before switching to active rejections.