AI training
Collect pages that may train models. Blocking them keeps future content out of training sets.
9 crawlers28 AI crawlers from OpenAI, Anthropic, Google, Perplexity and others: what each one does, its user agent and the robots.txt rule to allow or block it.
Checked against each operator’s documentation on Nov 17, 2026
| Crawler | Used for |
|---|---|
| ChatGPT-UserOpenAI | User-triggered |
| User-triggered | |
| User-triggered | |
| User-triggered | |
| User-triggered | |
| User-triggered | |
| User-triggered | |
| User-triggered |
Most AI companies run separate crawlers. Block the kind you don’t want and keep the rest.
Collect pages that may train models. Blocking them keeps future content out of training sets.
9 crawlersBuild the index AI answers search and cite. Blocking them can keep you out of those answers.
9 crawlersFetch one page when a person asks. Operators say robots.txt may not always apply.
8 crawlersThe rule set most sites ask for. It disallows every training crawler and control token in this directory and leaves search and user-triggered fetchers alone.
User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: meta-externalagent User-agent: Amazonbot User-agent: MistralAI-Training User-agent: CCBot User-agent: AI2Bot Disallow: /
What the bots are and how to control them.
Bots run by AI companies that fetch web pages. Some collect pages that may train models (GPTBot, ClaudeBot), some build the index AI search answers cite (OAI-SearchBot, PerplexityBot), and some fetch a page when a person asks (ChatGPT-User).
With robots.txt: a “User-agent:” line with the bot’s token and “Disallow: /”. Each bot page has the block and allow rules to copy, and the AI crawler checker shows what your current file does.
The operators listed here say their crawlers follow robots.txt. Several say their user-triggered fetchers may not, because a person asked for the page. Each bot page quotes what its operator documents.
Control tokens, not crawlers. They never visit your site; they decide whether pages Googlebot or Applebot already fetched may be used for AI training (and, for Google-Extended, grounding in Gemini). They don’t affect search results.
User agents can be faked. Most operators publish IP ranges (linked on each bot page) or support reverse DNS checks. Match the request IP against them before trusting the user agent.
Every bot is checked against its operator’s own documentation, with the date shown, and the list is reviewed every quarter. Bots without a primary source are not listed.
See whether GPTBot, ClaudeBot and others can reach your site.
Free toolValidate any site’s llms.txt against the format.
GuideWhat OpenAI’s crawlers do and how to control them.
TrackerSee whether Perplexity names and cites you.
Your buyers’ questions on eight AI engines, on your schedule, with history. Free for 7 days.