Scheduled and manual crawls
Scheduled AI discoverability crawls are free and run weekly by default; manual crawls use AI tokens. How to run, track and configure crawls.
AI discoverability scores your site from a crawl of your own pages. A crawl reads your robots.txt, finds pages from your sitemap, your Search Console data and your homepage, fetches and checks each page, then scores the results. It takes a few minutes for a small site and about a minute per 60 pages after that.
You start your first crawl yourself. After that, a scheduled crawl runs automatically, weekly by default. Scheduled crawls are free, run at most once a week, and drive the score trend and the Change columns. You can also start an extra manual crawl at any time, for example after shipping a fix. Manual crawls use AI tokens from your plan as they run, mainly for JavaScript rendering, so you need tokens available to start one. Beside the New crawl button, WriteWorks shows an estimate of the tokens a manual crawl will use, based on your latest crawl. If you run out of tokens, your scheduled crawl still runs.
While a crawl runs, a progress panel shows each step: Reading robots.txt, Finding pages, Fetching and checking pages, and Scoring, along with the pages just checked and which bots' rules were applied. Only one crawl can run at a time.
How to run and set up crawls
- Open AI discoverability from the Measure section of the sidebar.
- On your first visit, click Start crawl. If you see Add your website URL instead, add your site in Context first.
- To crawl again, click New crawl and check the token estimate beside it.
- Open Settings and choose a Schedule: Weekly, Monthly or Manual only.
- Set Max pages if you need to. Your plan sets the most pages per crawl, and partial crawls skew site averages.
- Add Include paths or Exclude paths, and turn Execute JavaScript on or off.
- Click Save settings.
- Execute JavaScript renders each page to measure content that only appears after scripts run. It is free on scheduled crawls.
- Scoring settings, such as SaaS mode, page type overrides and tracked bots, re-score the latest crawl straight away, with no new crawl needed.
- IndexNow, also in Settings, tells participating search engines when your pages change. Click Set up IndexNow, host the key file, click Verify key file, then turn on Auto-submit after each crawl.
After each scheduled crawl, members of your organisation can receive the Weekly AI discoverability report email, with your score, what improved and what to fix next. Turn it on or off in Settings, Notifications, or use the one-click unsubscribe link in the email itself.
Tip: leave the schedule on Weekly. It costs nothing and gives you a clean trend line that shows the effect of each fix.
Still stuck? We typically reply within 1 business day.
Contact support