Help · AI crawlers
AI crawlers
GPTBot and the other bots that read your site for AI answers.
3 articles
- What is GPTBot?
GPTBot is OpenAI's crawler for collecting training data. It is not the crawler that fetches pages for ChatGPT's search answers — that distinction matters a great deal.
- AI web crawlers explained
The bots reading your site for AI engines — who operates them, how training crawlers differ from retrieval crawlers, and what to allow.
- GPTBot and robots.txt: allowing or blocking OpenAI's crawlers
How to write robots.txt rules for OpenAI's crawlers, the configurations that make sense, and why blocking everything is usually accidental.