AI Crawler Directory

28 AI crawlers from OpenAI, Anthropic, Google, Perplexity and others: what each one does, its user agent and the robots.txt rule to allow or block it.

Checked against each operator’s documentation on Nov 17, 2026

Crawler Used for robots.txt robots.txt token
OAI-AdsBotOpenAI Other Not documented OAI-AdsBot
Google-CloudVertexBotGoogle Other Follows robots.txt Google-CloudVertexBot

Training, search and user-triggered crawlers

Most AI companies run separate crawlers. Block the kind you don’t want and keep the rest.

GPTBot ClaudeBot Google-Extended

AI training

Collect pages that may train models. Blocking them keeps future content out of training sets.

9 crawlers
OAI-SearchBot Claude-SearchBot Googlebot

AI search

Build the index AI answers search and cite. Blocking them can keep you out of those answers.

9 crawlers
ChatGPT-User Claude-User Google-GeminiNotebook

User-triggered

Fetch one page when a person asks. Operators say robots.txt may not always apply.

8 crawlers

Block AI training, stay in AI search

The rule set most sites ask for. It disallows every training crawler and control token in this directory and leaves search and user-triggered fetchers alone.

Check Which AI Bots Can Crawl Your Site

robots.txt
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: Amazonbot
User-agent: MistralAI-Training
User-agent: CCBot
User-agent: AI2Bot
Disallow: /

AI crawlers FAQ

What the bots are and how to control them.

What are AI crawlers?

Bots run by AI companies that fetch web pages. Some collect pages that may train models (GPTBot, ClaudeBot), some build the index AI search answers cite (OAI-SearchBot, PerplexityBot), and some fetch a page when a person asks (ChatGPT-User).

How do I block AI crawlers?

With robots.txt: a “User-agent:” line with the bot’s token and “Disallow: /”. Each bot page has the block and allow rules to copy, and the AI crawler checker shows what your current file does.

Do AI crawlers respect robots.txt?

The operators listed here say their crawlers follow robots.txt. Several say their user-triggered fetchers may not, because a person asked for the page. Each bot page quotes what its operator documents.

What are Google-Extended and Applebot-Extended?

Control tokens, not crawlers. They never visit your site; they decide whether pages Googlebot or Applebot already fetched may be used for AI training (and, for Google-Extended, grounding in Gemini). They don’t affect search results.

How do I verify a request really comes from an AI crawler?

User agents can be faked. Most operators publish IP ranges (linked on each bot page) or support reverse DNS checks. Match the request IP against them before trusting the user agent.

How often is this list updated?

Every bot is checked against its operator’s own documentation, with the date shown, and the list is reviewed every quarter. Bots without a primary source are not listed.

Files are the start. Answers are the result.

Your buyers’ questions on eight AI engines, on your schedule, with history. Free for 7 days.