AI web crawlers explained
The bots reading your site for AI engines — who operates them, how training crawlers differ from retrieval crawlers, and what to allow.
AI web crawlers fetch your pages on behalf of AI companies. This article explains the main operators, the crucial distinction between crawler types, how to monitor them, and how to decide what to allow.
Two kinds of crawler
This distinction determines everything else, and most confusion in this area comes from missing it.
Training crawlers collect content that may be used to train models. Their effect is slow and indirect: content they gather may influence what a model knows about your brand months later, when a new version ships. Blocking them has no immediate effect on your visibility.
Retrieval crawlers fetch pages so an engine can cite them in a live answer. Their effect is immediate and direct: block them and you disappear from that engine's search-based answers within days.
Several vendors now operate both, under separate names with separate robots.txt tokens, precisely so site owners can make different decisions about each. Treating them identically is the most common and most expensive configuration error in AI search.
Who operates them
The major operators, with the tokens current at time of writing. Verify against each vendor's own documentation before changing configuration, since these do change:
- OpenAI — GPTBot for training, OAI-SearchBot for search indexing, ChatGPT-User for user-initiated fetches.
- Anthropic — ClaudeBot and anthropic-ai for collection, with separate user-initiated and search agents.
- Perplexity — PerplexityBot for indexing, and a separate user-initiated agent.
- Google — Googlebot handles the crawling behind AI Overviews, so it cannot be separated from ordinary search. Google-Extended is a separate control governing use in Gemini and related grounding, and it does not affect search rankings.
- Apple — Applebot, with Applebot-Extended as the training opt-out.
- Meta, ByteDance, Amazon, and Common Crawl each run their own agents; Common Crawl's CCBot is notable because its archive feeds many other organisations' training sets.
Crawlers you run yourself
A related category worth knowing about, because the terms come up constantly: tools like Firecrawl and Crawl4AI are crawlers you operate, not crawlers that visit you. They fetch and convert web pages into clean, structured text for use in your own AI applications — retrieval-augmented generation, competitive research, or data pipelines.
They matter here for two reasons. If you build with them, respect the same robots.txt conventions you would want respected on your own site. And if you see unfamiliar crawler traffic, some of it will be other companies running these tools rather than a major vendor.
Monitoring crawler activity
Your server logs are the only fully reliable source, and four things are worth tracking:
- Which agents visit, and at what volume. Absence of a major retrieval crawler is a finding that explains a lot.
- Status codes returned. A high rate of 403s, 429s, or challenge responses means your edge is blocking crawlers your robots.txt permits.
- Which URLs are fetched, which tells you what each engine considers your significant content.
- Trend over time. Crawl growth generally precedes citation growth, making it the earliest available leading indicator.
Verify significant traffic against published IP ranges rather than trusting user-agent strings, which are trivially spoofed. Several operators publish their ranges for exactly this purpose.
What to allow
The position most commercial sites reach after thinking it through.
- Allow retrieval and user-initiated agents from every vendor whose answers your buyers use. These determine whether you appear at all, and blocking them is pure loss.
- Decide about training agents on principle rather than performance grounds. There is a real argument either way, and it is a business decision about content reuse rather than a technical one.
- Do not block Googlebot to avoid AI Overviews. You lose your search rankings along with the overview, which is strictly worse.
- Check your edge configuration, not just robots.txt. Bot protection blocking a crawler you have explicitly allowed is common and invisible unless you look at logs.
Common questions about AI web crawlers
- What is an AI crawler? A bot that fetches web pages on behalf of an AI company, either to collect training data or to retrieve sources for a live answer. The two purposes are usually served by separate agents with separate robots.txt tokens.
- How do I identify AI crawler traffic in my logs? Match on the published user-agent tokens for each vendor, then verify against their published IP ranges. User agents are trivially spoofed, so string matching alone will overstate legitimate crawler traffic.
- Are AI crawlers hurting my server performance? Occasionally. If crawl volume is a genuine load problem, rate-limit rather than block — a 429 with a sensible retry is far better than removing yourself from answers entirely.
- Should I block AI crawlers? Allow the retrieval and user-initiated agents from vendors whose answers your buyers use, because blocking them is pure loss. Treat training agents as a separate business judgement about content reuse.
- Do AI crawlers execute JavaScript? Several do not, and this is one of the most common reasons a site is invisible despite having no blocking rules. Always check what your pages return as raw HTML.
- What are Firecrawl and Crawl4AI? Crawlers you run yourself, used to turn web pages into clean text for your own AI applications. They are tools you operate, not bots that visit you.
Tip: audit crawler access once a quarter and after every CDN or platform change. Hosting providers and firewall vendors update their default bot rules regularly, and a default that changed under you is a far more common cause of lost visibility than anything you did to your content.
Still stuck? We typically reply within 1 business day.
Contact support