What is GPTBot?
GPTBot is OpenAI's crawler for collecting training data. It is not the crawler that fetches pages for ChatGPT's search answers — that distinction matters a great deal.
GPTBot is OpenAI's web crawler. This article explains what it does, how it differs from OpenAI's other crawlers, how to identify it in your logs, and why confusing the three is one of the most costly mistakes in AI search.
What GPTBot is for
GPTBot collects publicly available web content that may be used to train OpenAI's models. It was introduced so site owners had a documented way to opt out of that collection, which is why it is controllable separately from OpenAI's other agents.
The critical point is what GPTBot is not. It is not the crawler that fetches pages when ChatGPT searches the web to answer a question. Blocking GPTBot does not remove you from ChatGPT's live answers, and allowing it does not put you in them.
OpenAI's three crawlers
OpenAI operates separate agents for separate purposes, and they respect separate robots.txt directives. At time of writing these are:
- GPTBot — collects content that may be used for model training.
- OAI-SearchBot — crawls and indexes pages so ChatGPT can surface and link them in search-based answers. This is the one that affects whether you appear when ChatGPT searches.
- ChatGPT-User — fetches a specific page when a user asks ChatGPT to visit or read a URL during a conversation.
Because vendors do add and rename agents, verify the current list against OpenAI's own published documentation before making a robots.txt change.
The practical consequence is that a site wanting to be visible in ChatGPT's answers while staying out of training data should allow OAI-SearchBot and ChatGPT-User while disallowing GPTBot. A site that blocks all three is invisible to ChatGPT entirely, which is a decision worth making deliberately rather than by accident.
How to identify it in your logs
GPTBot identifies itself with a user-agent string containing "GPTBot" and a link to OpenAI's documentation. Because user agents are trivially spoofed, verify significant traffic against the IP ranges OpenAI publishes rather than trusting the string alone. Any crawler claiming to be GPTBot from an IP outside the published ranges is something else.
What to look for in your own logs
- Which of the three OpenAI agents are actually visiting, and at what rate.
- Which URLs they fetch — the pattern tells you what OpenAI considers your significant content.
- What status codes they receive. A high rate of 403s or challenge responses means your edge protection is blocking them regardless of what robots.txt says.
- Whether crawl volume is trending up or down, since retrieval-side visibility tends to follow crawl activity.
Should you block it
This is a genuine judgement call rather than a technical question, and reasonable organisations decide differently.
Arguments for blocking GPTBot: you do not want your content used to train a commercial model without compensation, you publish licensed or proprietary material, or you have a legal or contractual obligation to restrict reuse.
Arguments against blocking: content in training data is part of what makes a model able to name and recommend your brand from memory, which is one of the two paths to being mentioned at all. Blocking removes that path while providing no protection against the many crawlers that ignore robots.txt entirely.
What is rarely a good idea is blocking OAI-SearchBot. That directly removes you from ChatGPT's search-based answers, which is the fastest and most controllable route to visibility.
Common questions about GPTBot
- What is the GPTBot user agent string? It identifies itself with a user agent containing "GPTBot", a version, and a link to OpenAI's documentation page. Because user agents are trivially spoofed, verify significant traffic against OpenAI's published IP ranges rather than trusting the string.
- Does blocking GPTBot remove me from ChatGPT? No, and this is the most consequential misunderstanding in the area. GPTBot governs training collection. OAI-SearchBot governs whether you appear in ChatGPT's search-based answers. Blocking the first leaves the second untouched; blocking the second is what makes you disappear.
- How many websites block GPTBot? A substantial number of large publishers added blocks, particularly news and reference sites, and adoption was highest in the first year after GPTBot was introduced. Many of those rules pre-date the split between training and search agents and have not been revisited since.
- Does blocking GPTBot remove my content from models already trained? No. A block applies to future collection only. Content already gathered is not withdrawn by adding a rule now.
- Will blocking GPTBot protect my content from scraping? Only from crawlers that respect robots.txt. It is a request rather than an enforcement mechanism, and many crawlers ignore it entirely.
- Should a commercial site block GPTBot? It is a business judgement about content reuse rather than a performance question. The one choice that is usually wrong is blocking OAI-SearchBot, since that removes visibility without protecting anything.
Tip: check your robots.txt today rather than assuming. A large number of sites added blanket AI crawler blocks during an earlier wave of concern, before the search and training agents were separated, and never revisited the decision — which means they are invisible in ChatGPT for a reason nobody currently remembers.
Still stuck? We typically reply within 1 business day.
Contact support