AI crawler visits: add your server log or read Cloudflare

What it shows

Under What reached your website on a check, AI crawler visits lists every AI crawler found in your log: how many requests it made, how many different pages it read and how many ended in an error. Crawlers are split into two groups:

• Read pages for answers. Search crawlers and the fetchers that read a page because a person asked an assistant about it. These decide what an assistant can find and quote when it answers.
• Collect pages to train future models. Reported for completeness. Blocking them is a legitimate choice and does not keep you out of live answers.

Did they reach the pages that matter? follows your own pages the assistants read most in that check, plus your homepage, pricing, about, FAQ and contact pages (also under a country or language prefix, such as /ie/en/pricing). A page counts as reached only when it answered: a "not found" is not a read. When a crawler that reads your site regularly never reached one of those pages, the card says so.

Where to find your log

Download the access log for the period you want (a week or a month is plenty), plain or compressed (.gz):

• Apache or Nginx: usually /var/log/apache2/access.log or /var/log/nginx/access.log, with older days in numbered .gz files.
• cPanel hosting: Metrics > Raw Access, then download the archive for your domain.
• Managed WordPress hosts (WP Engine, Kinsta and others): the access logs section of your hosting dashboard.
• IIS: the W3C log files, usually under C:\inetpub\logs\LogFiles.
• AWS CloudFront or a load balancer: the access log files your distribution writes to S3.
• Cloudflare Logpush, Vercel or Netlify log drains: JSON lines with the user agent, path and status of each request.

Then, on your check, choose Add a server log. A CSV works too when it has a user agent column and a path column.

Your log stays on your computer

Access logs hold your visitors' IP addresses, so your browser reads the file itself, line by line, and sends PollyX only the counts: visits per crawler, the pages each crawler read most, response codes and dates. The log is never uploaded, and nothing about your human visitors is sent. A large log can take a minute to read; the button shows the progress.

Or read it from Cloudflare

If your website runs behind Cloudflare (its DNS records set to Proxied, the orange cloud), PollyX can read the same counts from your Cloudflare account instead of a log. First make a read-only token:

1. 1Open API Tokens in your Cloudflare profile and choose Create Token, then Get started next to Create Custom Token.
2. 2Add two permissions: Zone > Analytics > Read and Zone > Zone > Read.
3. 3Under Zone Resources, choose Include, Specific zone and your domain.
4. 4Leave the start date under TTL empty: a token with a start date in the future is refused until that date. An end date of tomorrow is a good idea, so the token switches itself off.
5. 5Choose Continue to summary, then Create Token, and copy the token. Cloudflare shows it only once.

Then, on your check, choose Read from Cloudflare, enter your domain (or the zone ID from the zone's Overview page in Cloudflare), pick 1, 7 or 30 days, paste the token and choose Read my crawler visits. PollyX uses the token for that one read and does not keep it, so you can delete it in Cloudflare straight after. You get the same counts as from a log: per crawler and page, never your visitors' IP addresses.

• Cloudflare counts only requests that pass through it. A domain set to DNS only (the grey cloud) shows no visits.
• Each Cloudflare plan keeps request data for a limited time, and the read stops at your plan's limit, so 30 days can come back shorter.
• If Cloudflare refuses the token, PollyX says why: a start date in the future, an expired token, a token switched off, or a missing permission.

The crawlers recognised

OAI-SearchBot (OpenAI), ChatGPT-User (OpenAI), GPTBot (OpenAI), Claude-SearchBot (Anthropic), Claude-User (Anthropic), ClaudeBot (Anthropic), Perplexity-User (Perplexity), PerplexityBot (Perplexity), Googlebot (Google), GoogleOther (Google), Google-CloudVertexBot (Google), Bingbot (Microsoft), Applebot (Apple), Meta-ExternalAgent (Meta), Meta-ExternalFetcher (Meta), MistralAI-User (Mistral), Amazonbot (Amazon), DuckAssistBot (DuckDuckGo), YouBot (You.com), Bytespider (ByteDance), CCBot (Common Crawl), cohere-ai (Cohere).

Good to know

• Crawlers are recognised by the user agent names their companies publish. A bot pretending to be one of them is counted as that bot, and a crawler that does not say who it is cannot be counted.
• If a CDN or cache answers requests before they reach your server, your server's log will not show them: use the CDN's own log instead.
• Each new log is kept below the previous one, so you can compare periods, and Remove deletes one for good.