Claude-SearchBot user agent and robots.txt rules

Claude-SearchBot crawls the web to improve the relevance and accuracy of Claude’s search answers. Blocking it keeps your pages out of that index.

AI search Follows robots.txt Anthropic

User agent

Anthropic documents the name only: requests contain “Claude-SearchBot”.

robots.txt rules for Claude-SearchBot

Paste one block into the robots.txt at the root of your domain. Changes can take up to a day to be read.

Block Claude-SearchBot

Disallows every path for Claude-SearchBot. That keeps your pages out of Anthropic’s AI search index, so its answers can’t cite them.

robots.txt
User-agent: Claude-SearchBot
Disallow: /

Allow Claude-SearchBot

Lets Claude-SearchBot fetch every path, whatever “User-agent: *” says.

robots.txt
User-agent: Claude-SearchBot
Allow: /

Good to know

Anthropic notes that blocking it may reduce your visibility in user search results.

Claude-SearchBot FAQ

What is Claude-SearchBot?

Claude-SearchBot crawls the web to improve the relevance and accuracy of Claude’s search answers. Blocking it keeps your pages out of that index.

How do I block Claude-SearchBot?

Add “User-agent: Claude-SearchBot” and “Disallow: /” to your robots.txt. That keeps your pages out of Anthropic’s AI search index, so its answers can’t cite them.

Does Claude-SearchBot respect robots.txt?

Yes. Anthropic says Claude-SearchBot follows robots.txt rules written for it.

What is the Claude-SearchBot user agent?

Anthropic documents the name “Claude-SearchBot”, which appears in its requests.

How can I verify Claude-SearchBot requests?

Match the request IP against Anthropic’s published list (https://claude.com/crawling/bots.json). The user agent alone can be faked.

Source: support.claude.com, checked Nov 17, 2026. AI Crawler Directory

Files are the start. Answers are the result.

Your buyers’ questions on eight AI engines, on your schedule, with history. Free for 7 days.