What is CCBot?

How often is CCBot impersonated?

For the median organisation, 89% of requests claiming to be CCBot failed verification.

Measured between 14 July 2026 and 10 October 2026 across organisations protected by Centinel. We only publish a share when at least five organisations each saw enough of these requests. Pooled across all of their requests, the share was 43%: a few large organisations carry most of the traffic, so the median organisation is the better guide to what yours will see.

This directory measures requests between 14 July 2026 and 10 October 2026. The fake crawler report covers its own, earlier window (late June to 21 September 2026), so its figures differ from the ones here. Read it for the method.

Have a request that claims to be CCBot? Check its IP address.

Should you block CCBot?

Allow it if

you are content for your pages to be in a public dataset others reuse.

Block it if

you do not want your content in an open dataset that third parties can use for any purpose, including model training.

What blocking changes

Your pages stop being added to future Common Crawl archives. Copies already in past archives stay there.

To block it, add this group to your robots.txt. Common Crawl says CCBot obeys robots.txt and Crawl-delay.

User-agent: CCBot
Disallow: /

robots.txt only asks. It does nothing against a client that ignores it or only pretends to be CCBot. Stopping those takes verification at your edge.