GPTBot and robots.txt: allowing or blocking OpenAI's crawlers
How to write robots.txt rules for OpenAI's crawlers, the configurations that make sense, and why blocking everything is usually accidental.
This article covers how to control OpenAI's crawlers through robots.txt, which configurations make sense for different goals, and the common mistakes that leave sites invisible without anyone intending it.
The rules
OpenAI's agents respect standard robots.txt directives, matched by user-agent token. At time of writing the tokens are GPTBot for training collection, OAI-SearchBot for search indexing, and ChatGPT-User for user-initiated page fetches. Verify against OpenAI's published documentation before changing anything, since vendors do add agents.
To block training collection while staying visible in ChatGPT's search answers.
- Disallow all paths for user-agent GPTBot.
- Allow user-agent OAI-SearchBot.
- Allow user-agent ChatGPT-User.
To allow everything, no rules are needed — absence of a disallow is permission.
To block OpenAI entirely, disallow all three. Understand that this removes you from ChatGPT's answers, not only from training.
To block training on part of your site, disallow specific paths for GPTBot rather than the whole site. This is a reasonable middle position for sites with both marketing content they want discovered and premium content they do not want reused.
The mistake that costs the most
A blanket rule blocking every user agent whose name contains "GPT", or a wildcard disallow applied to all bots, removes OAI-SearchBot along with GPTBot. The site then disappears from ChatGPT's search-based answers, which is almost never what the person writing the rule intended.
This happened at scale. A large number of sites added broad AI crawler blocks before the search and training agents were separated, and many have not revisited the configuration since. If your brand is absent from ChatGPT and you have not checked robots.txt recently, check it before investigating anything else.
What robots.txt does not do
- It does not enforce anything. Compliant crawlers respect it; crawlers that ignore it are unaffected. It is a request, not a control.
- It does not remove content already collected. Blocking today does not affect a model already trained.
- It does not override your edge. If your CDN or WAF challenges a crawler, an allow rule in robots.txt changes nothing. This is a frequent cause of confusion — the file says yes and the firewall says no.
- It does not control other vendors. Each operator uses its own tokens, and blocking OpenAI has no effect on Anthropic, Perplexity, Google, or anyone else.
How to verify your configuration works
- Fetch your robots.txt and read it in full, including any rules inherited from a platform default or a CDN-injected file.
- Check for wildcard rules that might catch the agents you meant to allow.
- Look at server and edge logs for each AI crawler token, and check the status codes returned. Requests receiving 403s or challenges indicate an edge problem, not a robots problem.
- Ask ChatGPT directly to read one of your URLs and summarise it. If it cannot access the page, something in the chain is blocking it.
- Re-check after any CDN, WAF, or hosting platform change, since these frequently reset or override bot rules.
Deciding what to allow
The practical position most commercial sites land on: allow the search and user-fetch agents from every major vendor, because those determine whether you appear in answers, and make a considered decision about training agents based on how you feel about content reuse. Blocking search agents to protect content is self-defeating — it removes the visibility while leaving the content just as available to non-compliant crawlers.
Common questions about GPTBot and robots.txt
- Where do I find OpenAI's robots.txt guidance? OpenAI publishes the current agent names and IP ranges in its own documentation, and that is the source to check before making changes — the agent list has changed as training and search were separated.
- Does the rule need to be exact? Yes. User-agent tokens are matched by name, so a rule for one agent has no effect on another. A rule aimed at GPTBot does nothing to OAI-SearchBot.
- What happens if I have no robots.txt at all? Everything is permitted by default. Absence of a file is permission, not a block.
- Why are crawlers still hitting pages I disallowed? Either the crawler ignores robots.txt, or the user agent is spoofed, or the rule is not matching the token you think it is. Check your logs for the exact agent string and verify against published IP ranges.
- Can I allow crawling but block training another way? Some vendors offer separate training opt-outs — Google-Extended and Applebot-Extended are examples — that leave ordinary crawling and ranking untouched. These are the right control when you want visibility without training reuse.
- Should I use noindex instead? No. Noindex governs search indexing and is a much blunter instrument here; it will remove you from conventional search results, which most AI engines retrieve through.
Tip: add a comment in your robots.txt next to every AI crawler rule saying who added it, when, and why. This field changes fast enough that an unexplained rule from eighteen months ago is the single most common cause of unexplained invisibility.
Still stuck? We typically reply within 1 business day.
Contact support