AI bot access and robots.txt
See which AI bots your robots.txt lets in, why blocking search bots costs score while blocking training bots never does, and how to check each bot.
AI search tools can only cite pages their bots are allowed to fetch. The Bots view in AI discoverability shows, for every AI bot you track, how much of your site your robots.txt lets it fetch, how that has changed across crawls and, if your hosting is connected, how often it actually visits.
Bots are grouped by what they do. AI search indexes, such as OAI-SearchBot, Claude-SearchBot and PerplexityBot, crawl ahead of time to build the index an AI search tool cites from. Live answer fetchers, such as ChatGPT-User, Claude-User, Perplexity-User and Google-Agent, fetch a page at the moment a user's question needs it. Search engines feeding AI, such as Googlebot, Bingbot, Applebot and Amazonbot, are classic crawlers whose index also feeds AI answers. Model training bots, such as GPTBot, ClaudeBot, CCBot, Google-Extended and Bytespider, collect public text to train AI models.
Blocking any of the first three groups costs Bot access score, because blocked pages can't appear in those tools' answers. A page blocked from every AI search bot scores 0 for Bot access, and if over half your indexable pages are in that state the whole score is capped at 40. Blocking training bots is a policy choice and never costs score. WriteWorks only raises a notice if your training bot policy is inconsistent.
WriteWorks reads robots.txt the way crawlers do. A bot follows the group that names it and falls back to the * group only when none does. The longest matching rule wins, and Allow beats Disallow on a tie. If robots.txt returns a 4xx error, every bot is allowed. If the server can't be reached, bots treat everything as blocked.
How to check a bot's access
- Open AI discoverability and choose Bots.
- Review Tracked bots by role and the Pages allowed figure for each bot.
- Use the role menus at the top, or click a bot, to open its detail page.
- Check Blocked pages, listed most important first with the robots.txt rule that blocks each one, and the group and rules the bot obeys.
- Read the response check. WriteWorks requests a share of pages with the bot's user agent and times the reply, so slow or refused responses can reveal a CDN or firewall treating AI agents differently.
- Update robots.txt, then run a crawl to confirm the change.
- Choose which bots count in Settings, under Tracked bots. An empty list tracks every default bot.
- Connect your host in AI crawlers to see each bot's visits over the last 90 days, when it was last seen and the paths it requested.
- Our crawler identifies itself as WriteWorksBot. Allow it through your firewall so crawls see what AI bots see.
Tip: you can block training bots and still allow the AI search and live answer bots. That keeps your pages citable in AI answers without offering your text for model training.
Still stuck? We typically reply within 1 business day.
Contact support