Every AI crawler can get in
Search crawlers can index you and training crawlers can collect your pages.
What to do If you’re fine with training, nothing to do. If not, block training bots and keep the search ones.
See every AI crawlerEnter your domain and we read its robots.txt to show which AI crawlers, like GPTBot, ClaudeBot and PerplexityBot, are allowed or blocked. Free, no sign-up.
Free · 60 checks a day · reads robots.txt only, no AI
An example for brightlane.com, a sample invoicing brand that blocks AI training but not AI search.
22 of 28 AI crawlers can reach /
Disallow: /
No matching rule
Disallow: /
No matching rule
Allowed, partly blocked or blocked, for the exact path you entered, with what each bot is used for.
The robots.txt line that wins for that bot, with its line number, so you know what to edit.
Training crawlers, AI search crawlers and user-triggered fetchers, so you can block one kind and keep the others.
We read one public file and apply the Robots Exclusion Protocol the way the crawlers do.
A bare domain checks the home page; a full URL checks that path, like /blog/post.
brightlane.com/blogAs AnswerHawkBot, from https://domain/robots.txt, following up to five redirects like RFC 9309 says.
GET /robots.txt · 200Its own User-agent group if there is one, else “*”. The longest matching rule wins.
User-agent: GPTBotEvery crawler from our directory, with its status, the deciding line and what it is used for.
Allowed BlockedBlocking is a business choice. These are the three usual situations.
Search crawlers can index you and training crawlers can collect your pages.
What to do If you’re fine with training, nothing to do. If not, block training bots and keep the search ones.
See every AI crawlerThe common middle ground: GPTBot or ClaudeBot out, OAI-SearchBot and PerplexityBot in.
What to do Check that the search crawlers you care about are allowed, then track whether AI answers cite you.
Track ChatGPT answersOAI-SearchBot, PerplexityBot or Claude-SearchBot can’t fetch your pages, so their answers can’t cite them.
What to do If that wasn’t on purpose, remove the rule or a “User-agent: *” Disallow, then check again.
Read about GPTBotThree common choices. Paste the block into your robots.txt, then check again.
Training crawlers and control tokens are disallowed. Search and user-triggered fetchers keep access, so AI answers can still cite you.
User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: meta-externalagent User-agent: Amazonbot User-agent: MistralAI-Training User-agent: CCBot User-agent: AI2Bot Disallow: /
Every AI crawler in our directory may fetch everything. Same as having no rule for them.
User-agent: GPTBot User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: OAI-AdsBot User-agent: ClaudeBot User-agent: Claude-SearchBot User-agent: Claude-User User-agent: Google-Extended User-agent: Google-CloudVertexBot User-agent: Google-GeminiNotebook User-agent: Google-Agent User-agent: Applebot User-agent: Applebot-Extended User-agent: PerplexityBot User-agent: Perplexity-User User-agent: meta-externalagent User-agent: meta-webindexer User-agent: meta-externalfetcher User-agent: Amazonbot User-agent: Amzn-SearchBot User-agent: Amzn-User User-agent: MistralAI-Training User-agent: MistralAI-Index User-agent: MistralAI-User User-agent: CCBot User-agent: DuckAssistBot User-agent: AI2Bot Allow: /
Every AI crawler in our directory is disallowed, except Googlebot (it also runs Google Search).
User-agent: GPTBot User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: OAI-AdsBot User-agent: ClaudeBot User-agent: Claude-SearchBot User-agent: Claude-User User-agent: Google-Extended User-agent: Google-CloudVertexBot User-agent: Google-GeminiNotebook User-agent: Google-Agent User-agent: Applebot User-agent: Applebot-Extended User-agent: PerplexityBot User-agent: Perplexity-User User-agent: meta-externalagent User-agent: meta-webindexer User-agent: meta-externalfetcher User-agent: Amazonbot User-agent: Amzn-SearchBot User-agent: Amzn-User User-agent: MistralAI-Training User-agent: MistralAI-Index User-agent: MistralAI-User User-agent: CCBot User-agent: DuckAssistBot User-agent: AI2Bot Disallow: /
GPTBot is OpenAI’s crawler for collecting pages that may train its models. OpenAI runs two more: OAI-SearchBot, which surfaces sites in ChatGPT search, and ChatGPT-User, which fetches a page when a person asks.
Most AI companies now split crawlers the same way: one for training, one for search, one for user requests. Blocking the training crawler does not block the search one, and the search crawler is the one that lets answers cite you.
robots.txt is a request, not a wall: the operators below say their crawlers follow it, and some user-triggered fetchers say rules may not apply. Firewalls and CDNs can block bots too, which robots.txt can’t show. Browse the AI crawler directory.
robots.txt, GPTBot and the other AI bots.
Enter your domain above. The checker reads your robots.txt and shows whether GPTBot, and every other AI crawler in our directory, is allowed or blocked for that path, with the line that decides it.
OpenAI documents it as “Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot”. The version number changes; in robots.txt you only write “User-agent: GPTBot”.
Add a group per crawler: “User-agent: GPTBot” and “Disallow: /” on the next line. Copy one of the rule sets on this page: block training only, or block all AI crawlers.
Not from ChatGPT search: OpenAI says GPTBot is for training and OAI-SearchBot is the one that surfaces sites in ChatGPT search. Block GPTBot and keep OAI-SearchBot to stay out of training but in search.
robots.txt is only one layer. A firewall, CDN bot protection or a login wall can block crawlers whatever robots.txt says. The checker reads robots.txt only.
No. It reads one public file and applies the Robots Exclusion Protocol (RFC 9309). It is free, 60 checks a day, and results are saved for 24 hours at a shareable link.
Your buyers’ questions on eight AI engines, on your schedule, with history. Free for 7 days.