CCBot user agent and robots.txt rules

CCBot is Common Crawl’s crawler. It builds an open archive of the web that is free to download and widely used as AI training data.

AI training Follows robots.txt Common Crawl

User agent

As documented by Common Crawl. Version numbers change; match on the token.

User agent
CCBot/2.0 (https://commoncrawl.org/faq/)

robots.txt rules for CCBot

Paste one block into the robots.txt at the root of your domain. Changes can take up to a day to be read.

Block CCBot

Disallows every path for CCBot. That keeps your future pages out of what Common Crawl may use for training. It does not change search results.

robots.txt
User-agent: CCBot
Disallow: /

Allow CCBot

Lets CCBot fetch every path, whatever “User-agent: *” says.

robots.txt
User-agent: CCBot
Allow: /

Good to know

Others sometimes fake its user agent. Common Crawl publishes its IP ranges and supports reverse DNS under crawl.commoncrawl.org.

CCBot FAQ

What is CCBot?

CCBot is Common Crawl’s crawler. It builds an open archive of the web that is free to download and widely used as AI training data.

How do I block CCBot?

Add “User-agent: CCBot” and “Disallow: /” to your robots.txt. That keeps your future pages out of what Common Crawl may use for training. It does not change search results.

Does CCBot respect robots.txt?

Yes. Common Crawl says CCBot follows robots.txt rules written for it.

What is the CCBot user agent?

Common Crawl documents it as “CCBot/2.0 (https://commoncrawl.org/faq/)”. Version numbers change, so match on “CCBot”.

How can I verify CCBot requests?

Match the request IP against Common Crawl’s published list (https://index.commoncrawl.org/ccbot.json). The user agent alone can be faked.

Source: commoncrawl.org, checked Nov 17, 2026. AI Crawler Directory

Files are the start. Answers are the result.

Your buyers’ questions on eight AI engines, on your schedule, with history. Free for 7 days.