Claude-User user agent and robots.txt rules

Claude-User fetches a page when a Claude user asks a question that needs it. Blocking it stops Claude from retrieving your content for those questions.

User-triggered Follows robots.txt Anthropic

User agent

Anthropic documents the name only: requests contain “Claude-User”.

robots.txt rules for Claude-User

Paste one block into the robots.txt at the root of your domain. Changes can take up to a day to be read.

Block Claude-User

Disallows every path for Claude-User. That asks Anthropic not to fetch your pages when its users ask, though Anthropic says user-triggered fetches may not always follow robots.txt.

robots.txt
User-agent: Claude-User
Disallow: /

Allow Claude-User

Lets Claude-User fetch every path, whatever “User-agent: *” says.

robots.txt
User-agent: Claude-User
Allow: /

Good to know

Anthropic says its bots honour robots.txt, including this one.

Claude-User FAQ

What is Claude-User?

Claude-User fetches a page when a Claude user asks a question that needs it. Blocking it stops Claude from retrieving your content for those questions.

How do I block Claude-User?

Add “User-agent: Claude-User” and “Disallow: /” to your robots.txt. That asks Anthropic not to fetch your pages when its users ask, though Anthropic says user-triggered fetches may not always follow robots.txt.

Does Claude-User respect robots.txt?

Yes. Anthropic says Claude-User follows robots.txt rules written for it.

What is the Claude-User user agent?

Anthropic documents the name “Claude-User”, which appears in its requests.

How can I verify Claude-User requests?

Match the request IP against Anthropic’s published list (https://claude.com/crawling/bots.json). The user agent alone can be faked.

Source: support.claude.com, checked Nov 17, 2026. AI Crawler Directory

Files are the start. Answers are the result.

Your buyers’ questions on eight AI engines, on your schedule, with history. Free for 7 days.