ClaudeBot user agent and robots.txt rules

ClaudeBot is Anthropic’s crawler for web content that may be used to train its models. Blocking it keeps your future content out of Anthropic’s training data.

AI training Follows robots.txt Anthropic

User agent

Anthropic documents the name only: requests contain “ClaudeBot”.

robots.txt rules for ClaudeBot

Paste one block into the robots.txt at the root of your domain. Changes can take up to a day to be read.

Block ClaudeBot

Disallows every path for ClaudeBot. That keeps your future pages out of what Anthropic may use for training. It does not change search results.

robots.txt
User-agent: ClaudeBot
Disallow: /

Allow ClaudeBot

Lets ClaudeBot fetch every path, whatever “User-agent: *” says.

robots.txt
User-agent: ClaudeBot
Allow: /

Good to know

ClaudeBot also honours the non-standard Crawl-delay directive. Anthropic advises against IP blocking to opt out, since it can stop the bot from reading your robots.txt.

ClaudeBot FAQ

What is ClaudeBot?

ClaudeBot is Anthropic’s crawler for web content that may be used to train its models. Blocking it keeps your future content out of Anthropic’s training data.

How do I block ClaudeBot?

Add “User-agent: ClaudeBot” and “Disallow: /” to your robots.txt. That keeps your future pages out of what Anthropic may use for training. It does not change search results.

Does ClaudeBot respect robots.txt?

Yes. Anthropic says ClaudeBot follows robots.txt rules written for it.

What is the ClaudeBot user agent?

Anthropic documents the name “ClaudeBot”, which appears in its requests.

How can I verify ClaudeBot requests?

Match the request IP against Anthropic’s published list (https://claude.com/crawling/bots.json). The user agent alone can be faked.

Source: support.claude.com, checked Nov 17, 2026. AI Crawler Directory

Files are the start. Answers are the result.

Your buyers’ questions on eight AI engines, on your schedule, with history. Free for 7 days.