What is Crawl4AI? The open-source AI crawler

Crawl4AI is an open-source Python library that crawls websites and converts pages into clean Markdown or structured data for large language models. It usually drives a real browser through Playwright, and it runs wherever its user installs it, so there is no single Crawl4AI network or service behind the traffic.

What Crawl4AI does

The project repository describes Crawl4AI as an "LLM Friendly Web Crawler & Scraper" and publishes it under the Apache License 2.0. A developer installs it with pip, and the setup step installs Playwright and its browsers. The browser configuration reference lists Chromium, Firefox, and WebKit as browser types, with Chromium as the default.

The output is the point. The Markdown generation docs say the library was built to keep a page's actual content and discard boilerplate. Content filters score text density, link density, and page structure to drop navigation and footers, which produces what the project calls "fit markdown." The same page can also come back as HTML, JSON from an extraction strategy, a screenshot, or a PDF.

Crawl4AI does not always need a browser. Version 0.5.0 added an HTTP-only crawler strategy for pages that do not need JavaScript rendering. The same tool can therefore reach your site as a full browser session or as plain HTTP requests, depending on how the operator configured it.

How people use it

The common pattern is a pipeline: crawl a list of URLs or a whole site, turn each page into Markdown, and feed the result into a retrieval index, an agent, or a dataset. Crawl4AI can also run as a self-hosted server. The installation page describes a Docker image that exposes the crawler over HTTP, which lets other applications send it jobs.

Unlike a hosted scraping API such as Firecrawl, Crawl4AI has no vendor-operated fetch fleet. The requests your site receives come from whatever machine, cloud account, or proxy the operator chose. An IP list for "Crawl4AI" does not exist in any useful sense.

Settings that matter to defenders

Several documented options change what a target website sees:

User agent. The operator can set any user-agent string or set user_agent_mode to "random", which the parameter reference describes as a way to randomize the user agent. The string your logs record is whatever the operator picked.
Stealth mode. Setting enable_stealth=True applies playwright-stealth. The undetected browser page says stealth mode removes the navigator.webdriver flag and modifies browser fingerprints. A separate undetected browser adapter applies deeper patches aimed at bot detection.
Simulated interaction. The run configuration includes simulate_user, documented as simulating mouse movements to avoid bot detection, and override_navigator, which overrides navigator properties in JavaScript.
Proxies. The proxy docs cover authenticated HTTP, HTTPS, and SOCKS5 proxies, plus a round-robin strategy that rotates proxies per request.
Persistent identity. Identity-based crawling stores cookies, local storage, and other session data in a browser profile so a crawl can reuse a logged-in session without repeating the login.

The project is candid about the limits of these modes. The undetected browser page says evasion is "Not 100% Guaranteed" and that headless mode can be detected more easily. It suggests running with a visible browser window instead.

Crawl rate and robots.txt

Crawl4AI includes pacing controls, but the operator chooses them. The multi-URL crawling docs describe a rate limiter that waits a random one to three seconds between requests to the same domain by default and backs off when the site returns 429 or 503. Dispatchers cap concurrent tasks, with documented defaults of 10 or 20 depending on the dispatcher.

robots.txt is opt-in. The parameter reference lists check_robots_txt with a default of False. When it is enabled, the crawler checks robots.txt rules before each URL. Without that flag, a default Crawl4AI run does not consult your robots.txt at all. robots.txt is a voluntary policy in any case, so it tells you what a polite crawler should do, not what a given client will do.

What a target website can observe

Because the operator controls the user agent, network path, and browser patches, one field rarely settles the question. What you can observe depends on the mode:

HTTP-only runs send requests without executing JavaScript. They load no scripts or images and leave no browser signals to inspect. A browser user agent on such a request is a claim you can check against its headers and TLS handshake, which come from an HTTP client rather than a browser.
Browser runs execute your pages in Playwright-driven Chromium, Firefox, or WebKit. Stealth patches change some JavaScript-visible properties. Consistency checks still apply: does the claimed browser match the TLS fingerprint, header order, rendering behavior, and platform details? A randomized user agent on top of an unchanged browser build is a common source of mismatches. How to detect browser automation walks through that kind of evidence.
Request patterns are often the clearest signal. A Markdown pipeline typically fetches documents, skips assets it does not need, follows links systematically, and moves through a section faster and more evenly than a reader does.

How to assess Crawl4AI traffic on your site

The question is less "is this Crawl4AI?" and more "what is this client doing, and is that allowed?" When reviewing a suspected crawl:

Which routes did it request, and were any behind a login or a paywall?
Did it reuse cookies from an authenticated session?
Does the same traversal order repeat across changing IPs or user agents?
Did it request robots.txt, and did it follow the rules it found?
Is there an approved integration, partner, or internal team running a crawler?

Then pick a response that fits the resource. A public documentation page may be fine to crawl at a modest rate. A search endpoint, a pricing page that competitors track, or a signed-in workflow may warrant rate limits, a challenge, or a block. How to prevent web scraping covers those options in more depth.