AI web crawlers explained
The bots reading your site for AI engines — who operates them, how training crawlers differ from retrieval crawlers, and what to allow.
AI web crawlers fetch your pages on behalf of AI companies. This article explains the main operators, the crucial distinction between crawler types, how to monitor them, and how to decide what to allow.
Two kinds of crawler
This distinction determines everything else, and most confusion in this area comes from missing it.
Training crawlers collect content that may be used to train models. Their effect is slow and indirect: content they gather may influence what a model knows about your brand months later, when a new version ships. Blocking them has no immediate effect on your visibility.
Retrieval crawlers fetch pages so an engine can cite them in a live answer. Some build a search index ahead of time; others fetch a page at the moment a user's question needs it. Their effect is immediate and direct: block them and you disappear from that engine's search-based answers.
Several vendors now operate both, under separate names with separate robots.txt tokens, precisely so site owners can make different decisions about each. Treating them identically is the most common and most expensive configuration error in AI search.
Who operates them
The major operators, with the tokens current at time of writing. Verify against each vendor's own documentation before changing configuration, since these do change:
- OpenAI — GPTBot for training, OAI-SearchBot for search indexing, ChatGPT-User for user-initiated fetches.
- Anthropic — ClaudeBot for training, Claude-SearchBot for search indexing, Claude-User for user-initiated fetches. Older tokens such as anthropic-ai still appear in some logs.
- Perplexity — PerplexityBot for indexing, Perplexity-User for user-initiated fetches.
- Google — Googlebot handles the crawling behind AI Overviews and AI Mode, so it cannot be separated from ordinary search. Google-Agent fetches pages on a user's behalf. Google-Extended is not a crawler but a robots.txt token controlling whether Gemini may use your content, and it does not affect search rankings.
- Microsoft — Bingbot. Bing's index also feeds Microsoft Copilot and ChatGPT search, so blocking it costs more than Bing traffic.
- Apple — Applebot for Siri and Spotlight, with Applebot-Extended as the training opt-out.
- Meta, ByteDance, Amazon, and Common Crawl each run their own agents; Common Crawl's CCBot is notable because its archive feeds many other organisations' training sets.
Crawlers you run yourself
A related category worth knowing about: crawl-and-convert tools that companies run themselves to turn web pages into clean text for their own AI applications, such as retrieval-augmented generation, competitive research or data pipelines. These are crawlers you operate, not crawlers that visit you on behalf of a major engine.
They matter here for two reasons. If you build with them, respect the same robots.txt conventions you would want respected on your own site. And if you see unfamiliar crawler traffic, some of it will be other companies running tools like these rather than a major vendor.
Monitoring crawler activity
Your server logs are the only fully reliable source, and four things are worth tracking:
- Which agents visit, and at what volume. Absence of a major retrieval crawler is a finding that explains a lot.
- Status codes returned. A high rate of 403s, 429s, or challenge responses means your edge is blocking crawlers your robots.txt permits.
- Which URLs are fetched, which tells you what each engine considers your significant content.
- Trend over time. Crawl growth generally precedes citation growth, making it the earliest available leading indicator.
Verify significant traffic against published IP ranges rather than trusting user-agent strings, which are trivially spoofed. Several operators publish their ranges for exactly this purpose.
In WriteWorks, AI crawlers counts real crawler visits from your own traffic once your host is connected, split by purpose, and AI discoverability checks your robots.txt against each tracked bot and times responses for AI user agents, so you can spot a firewall treating them differently.
What to allow
The position most commercial sites reach after thinking it through.
- Allow retrieval and user-initiated agents from every vendor whose answers your buyers use. These determine whether you appear at all, and blocking them is pure loss.
- Decide about training agents on principle rather than performance grounds. There is a real argument either way, and it is a business decision about content reuse rather than a technical one.
- Do not block Googlebot or Bingbot to avoid AI answers. You lose your search rankings along with the AI features, which is strictly worse.
- Check your edge configuration, not just robots.txt. Bot protection blocking a crawler you have explicitly allowed is common and invisible unless you look at logs.
Common questions about AI web crawlers
- What is an AI crawler? A bot that fetches web pages on behalf of an AI company, either to collect training data or to retrieve sources for a live answer. The two purposes are usually served by separate agents with separate robots.txt tokens.
- How do I identify AI crawler traffic in my logs? Match on the published user-agent tokens for each vendor, then verify against their published IP ranges. User agents are trivially spoofed, so string matching alone will overstate legitimate crawler traffic.
- Are AI crawlers hurting my server performance? Occasionally. If crawl volume is a genuine load problem, rate-limit rather than block — a 429 with a sensible retry is far better than removing yourself from answers entirely.
- Should I block AI crawlers? Allow the retrieval and user-initiated agents from vendors whose answers your buyers use, because blocking them is pure loss. Treat training agents as a separate business judgement about content reuse.
- Do AI crawlers execute JavaScript? Most do not, and this is one of the most common reasons a site is invisible despite having no blocking rules. Always check what your pages return as raw HTML.
- What are crawl-and-convert tools? Crawlers you run yourself, used to turn web pages into clean text for your own AI applications. They are tools you operate, not bots that visit you.
Tip: audit crawler access once a quarter and after every CDN or platform change. Hosting providers and firewall vendors update their default bot rules regularly, and a default that changed under you is a far more common cause of lost visibility than anything you did to your content.
Still stuck? We typically reply within 1 business day.
Contact support