Check Which AI Bots Can Crawl Your Site

Enter your domain and we read its robots.txt to show which AI crawlers, like GPTBot, ClaudeBot and PerplexityBot, are allowed or blocked. Free, no sign-up.

Free · 60 checks a day · reads robots.txt only, no AI

What an AI crawler check returns

An example for brightlane.com, a sample invoicing brand that blocks AI training but not AI search.

brightlane.comrobots.txt · 412 bytes · Nov 16 robots.txt read

22 of 28 AI crawlers can reach /

CrawlerStatusRule that decides
GPTBotAI training Blocked Disallow: /
OAI-SearchBotAI search Allowed No matching rule
ClaudeBotAI training Blocked Disallow: /
PerplexityBotAI search Allowed No matching rule
  1. 1

    One line per crawler

    Allowed, partly blocked or blocked, for the exact path you entered, with what each bot is used for.

  2. 2

    The rule that decides

    The robots.txt line that wins for that bot, with its line number, so you know what to edit.

  3. 3

    Grouped by purpose

    Training crawlers, AI search crawlers and user-triggered fetchers, so you can block one kind and keep the others.

How it works

We read one public file and apply the Robots Exclusion Protocol the way the crawlers do.

  1. 1

    Enter a domain or URL

    A bare domain checks the home page; a full URL checks that path, like /blog/post.

    brightlane.com/blog
  2. 2

    We fetch robots.txt

    As AnswerHawkBot, from https://domain/robots.txt, following up to five redirects like RFC 9309 says.

    GET /robots.txt · 200
  3. 3

    We match each crawler

    Its own User-agent group if there is one, else “*”. The longest matching rule wins.

    User-agent: GPTBot
  4. 4

    You see who gets in

    Every crawler from our directory, with its status, the deciding line and what it is used for.

    Allowed Blocked

What to do with the result

Blocking is a business choice. These are the three usual situations.

All allowed AI training AI search User-triggered

Every AI crawler can get in

Search crawlers can index you and training crawlers can collect your pages.

What to do If you’re fine with training, nothing to do. If not, block training bots and keep the search ones.

See every AI crawler
Mixed AI training AI search User-triggered

Training blocked, search allowed

The common middle ground: GPTBot or ClaudeBot out, OAI-SearchBot and PerplexityBot in.

What to do Check that the search crawlers you care about are allowed, then track whether AI answers cite you.

Track ChatGPT answers
All blocked AI training AI search User-triggered

AI search crawlers are blocked

OAI-SearchBot, PerplexityBot or Claude-SearchBot can’t fetch your pages, so their answers can’t cite them.

What to do If that wasn’t on purpose, remove the rule or a “User-agent: *” Disallow, then check again.

Read about GPTBot

robots.txt rules for AI crawlers

Three common choices. Paste the block into your robots.txt, then check again.

Block AI training, stay in AI search

Training crawlers and control tokens are disallowed. Search and user-triggered fetchers keep access, so AI answers can still cite you.

robots.txt
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: Amazonbot
User-agent: MistralAI-Training
User-agent: CCBot
User-agent: AI2Bot
Disallow: /

Allow all AI crawlers

Every AI crawler in our directory may fetch everything. Same as having no rule for them.

robots.txt
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: OAI-AdsBot
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: Google-Extended
User-agent: Google-CloudVertexBot
User-agent: Google-GeminiNotebook
User-agent: Google-Agent
User-agent: Applebot
User-agent: Applebot-Extended
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: meta-externalagent
User-agent: meta-webindexer
User-agent: meta-externalfetcher
User-agent: Amazonbot
User-agent: Amzn-SearchBot
User-agent: Amzn-User
User-agent: MistralAI-Training
User-agent: MistralAI-Index
User-agent: MistralAI-User
User-agent: CCBot
User-agent: DuckAssistBot
User-agent: AI2Bot
Allow: /

Block all AI crawlers

Every AI crawler in our directory is disallowed, except Googlebot (it also runs Google Search).

robots.txt
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: OAI-AdsBot
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: Google-Extended
User-agent: Google-CloudVertexBot
User-agent: Google-GeminiNotebook
User-agent: Google-Agent
User-agent: Applebot
User-agent: Applebot-Extended
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: meta-externalagent
User-agent: meta-webindexer
User-agent: meta-externalfetcher
User-agent: Amazonbot
User-agent: Amzn-SearchBot
User-agent: Amzn-User
User-agent: MistralAI-Training
User-agent: MistralAI-Index
User-agent: MistralAI-User
User-agent: CCBot
User-agent: DuckAssistBot
User-agent: AI2Bot
Disallow: /

What is GPTBot, and which AI crawlers matter?

GPTBot is OpenAI’s crawler for collecting pages that may train its models. OpenAI runs two more: OAI-SearchBot, which surfaces sites in ChatGPT search, and ChatGPT-User, which fetches a page when a person asks.

Most AI companies now split crawlers the same way: one for training, one for search, one for user requests. Blocking the training crawler does not block the search one, and the search crawler is the one that lets answers cite you.

robots.txt is a request, not a wall: the operators below say their crawlers follow it, and some user-triggered fetchers say rules may not apply. Firewalls and CDNs can block bots too, which robots.txt can’t show. Browse the AI crawler directory.

Three kinds of AI crawler
AI training
Collect pages that may train models. Blocking them keeps future content out of training sets. GPTBot, ClaudeBot, Google-Extended
AI search
Build the index AI answers search and cite. Blocking them can keep you out of those answers. OAI-SearchBot, PerplexityBot, Claude-SearchBot
User-triggered
Fetch one page when a person asks. Operators say robots.txt may not always apply. ChatGPT-User, Claude-User, Perplexity-User

AI crawler checker FAQ

robots.txt, GPTBot and the other AI bots.

How do I know if GPTBot is blocked on my site?

Enter your domain above. The checker reads your robots.txt and shows whether GPTBot, and every other AI crawler in our directory, is allowed or blocked for that path, with the line that decides it.

What is the GPTBot user agent?

OpenAI documents it as “Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot”. The version number changes; in robots.txt you only write “User-agent: GPTBot”.

How do I block AI crawlers in robots.txt?

Add a group per crawler: “User-agent: GPTBot” and “Disallow: /” on the next line. Copy one of the rule sets on this page: block training only, or block all AI crawlers.

If I block GPTBot, will I disappear from ChatGPT?

Not from ChatGPT search: OpenAI says GPTBot is for training and OAI-SearchBot is the one that surfaces sites in ChatGPT search. Block GPTBot and keep OAI-SearchBot to stay out of training but in search.

Why does the checker show “allowed” but bots still can’t reach my site?

robots.txt is only one layer. A firewall, CDN bot protection or a login wall can block crawlers whatever robots.txt says. The checker reads robots.txt only.

Does this checker call AI?

No. It reads one public file and applies the Robots Exclusion Protocol (RFC 9309). It is free, 60 checks a day, and results are saved for 24 hours at a shareable link.

Files are the start. Answers are the result.

Your buyers’ questions on eight AI engines, on your schedule, with history. Free for 7 days.