GPTBot is the web crawler OpenAI uses to gather content that may be used to train its generative AI models (OpenAI). It is one of three OpenAI user agents, and each one needs its own decision in robots.txt. Check whether GPTBot can crawl your site before you change any rule.
GPTBot, OAI-SearchBot and ChatGPT-User
| User agent | What OpenAI uses it for | robots.txt | IP ranges |
|---|---|---|---|
GPTBot |
Crawls content that may be used to train OpenAI's foundation models | Honoured | gptbot.json |
OAI-SearchBot |
Surfaces websites in ChatGPT's search features | Honoured | searchbot.json |
ChatGPT-User |
Certain actions a user starts in ChatGPT and custom GPTs; not automatic crawling | "robots.txt rules may not apply" | chatgpt-user.json |
Source: OpenAI, Overview of OpenAI crawlers.
robots.txt rules for GPTBot
Block training, allow ChatGPT search:
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
Allow everything from OpenAI:
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
Block GPTBot from one folder only:
User-agent: GPTBot
Disallow: /private/
Rules apply per user agent, so a Disallow for GPTBot doesn't affect OAI-SearchBot. Remember that a CDN or firewall "block AI bots" switch can override what your robots.txt says.
How to verify GPTBot traffic
A user agent string is easy to fake. OpenAI publishes the IP ranges of each crawler as JSON (linked in the table above). Match the IP of a request against the right list before you trust it, and review your logs after any rule change.
Does GPTBot affect visibility in ChatGPT?
Visibility in ChatGPT's search answers depends on the search crawler, not on GPTBot: OpenAI describes OAI-SearchBot as the agent used to surface websites in ChatGPT's search features (OpenAI). Answers that come from the model alone reflect what the model learned in training, which is where GPTBot's role sits. If AI visibility matters to you, measure it on a fixed set of prompts before and after changing these rules.
Sources
- Overview of OpenAI crawlers · OpenAI
- GPTBot IP ranges · OpenAI
- OAI-SearchBot IP ranges · OpenAI
- ChatGPT-User IP ranges · OpenAI