How to find AI crawlers in your server logs

Your robots.txt says what you permit. Your access log says what actually happened. Those two things disagree more often than anyone expects, and the log is the one that is right.

A firewall rule nobody remembers writing, a CDN bot-fight setting, a rate limit, a redirect chain that ends in a 404. None of that appears in robots.txt. All of it appears in the log.

This guide is the token list, dated and sourced, and the commands to pull it apart.

The user-agent reference table

Every token below was read from the vendor's own documentation on 2026-08-06. These lists change, so treat this as a starting point and re-check the linked pages before relying on it.

Two entries in that table are the most commonly misread things in this whole topic, so they get their own sentence. Google-Extended and Applebot-Extended are robots.txt control tokens with no crawler behind them. Searching your logs for either will return nothing forever, and that is not evidence of anything.

Find the log

On nginx, /var/log/nginx/access.log and its rotated .gz siblings. On Apache, /var/log/apache2/access.log or /var/log/httpd/access_log. On shared hosting, the control panel usually has a raw access logs download.

One structural warning that invalidates the whole exercise if you miss it. If your site sits behind a CDN, the origin log only records cache misses. A crawler served entirely from cache never appears. Read the CDN's request logs instead: Cloudflare, Fastly, CloudFront, and the rest all expose bot traffic, and that is your real log.

And if you are on a hosted platform that gives you no log at all, which covers Shopify, Wix, Squarespace and most page builders, this method is closed to you unless there is a CDN in front. Skip to the last section.

Do the task

1. One pass across every token

grep -aiE 'GPTBot|OAI-SearchBot|ChatGPT-User|OAI-AdsBot|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|GoogleOther|Google-CloudVertexBot|Applebot|meta-webindexer|meta-externalagent|meta-externalfetcher' /var/log/nginx/access.log

Run this first for the shape of it. What you are looking for is not the volume, it is which names are entirely missing.

2. Count per crawler

for ua in GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Claude-SearchBot Claude-User PerplexityBot Perplexity-User Applebot GoogleOther meta-webindexer; do
  printf '%-20s %s\n' "$ua" "$(grep -aic "$ua" /var/log/nginx/access.log)"
done

A crawler at zero across a month is the finding. A search crawler at zero, OAI-SearchBot or Claude-SearchBot or PerplexityBot, means that engine is not fetching your pages at all, and the reasons are a short list: a robots.txt rule, a firewall or WAF rule, an IP block, or nothing on the web linking to you.

3. Status codes, which is where the real signal is

grep -ai 'OAI-SearchBot' /var/log/nginx/access.log | awk '{print $9}' | sort | uniq -c | sort -rn

That is the combined log format, where field 9 is the status code and field 7 is the path.

A wall of 200s is fine. A wall of 403s means something between the crawler and your content is refusing it, and it is almost never robots.txt, because a crawler honouring robots.txt would not have made the request. It is a WAF rule, a bot-fight mode, or a country block. A run of 429s means you are rate limiting a crawler you probably want. 5xx means your server fell over during a crawl and the engine now has an error where a page should be.

4. Which pages they are actually fetching

grep -ai 'PerplexityBot' /var/log/nginx/access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head -20

Compare that list against the pages you care about. Crawlers frequently hammer a paginated archive and never touch the product or service pages that matter, which is a discovery problem, not a permission problem. An llms.txt file is one way to point at your best pages rather than only stating what is off limits.

5. Verify the name before you trust it

The user agent is a request header the client writes itself. Anything can claim to be GPTBot, and things do.

Every vendor in the table publishes a way to check. Each of the files below was fetched on 2026-08-06 and returned JSON.

• OpenAI: openai.com/gptbot.json, openai.com/searchbot.json, openai.com/chatgpt-user.json. There is no equivalent file published for OAI-AdsBot.
• Anthropic: claude.com/crawling/bots.json.
• Perplexity: www.perplexity.ai/perplexitybot.json and www.perplexity.ai/perplexity-user.json. Note the domain. These are published on perplexity.ai, and the perplexity.com equivalents redirect there rather than serving their own copy.
• Apple: search.developer.apple.com/applebot.json. Apple's documentation gives both methods, the CIDR file and reverse DNS under applebot.apple.com.
• Google: an IP range file for its common crawlers, plus a documented reverse-DNS convention.

grep -ai 'GPTBot' /var/log/nginx/access.log | awk '{print $1}' | sort -u | head

Take those addresses and check them. If a number is going in a report, this step is not optional.

6. Separate the scheduled from the user-triggered

ChatGPT-User, Claude-User, Perplexity-User, and meta-externalfetcher are a different category from the rest. They fire because a person asked a question that touched your page, right then.

They look different in a log too: irregular, bursty, often a single URL rather than a sweep. That pattern is closer to a demand signal than a crawl statistic. It is also the category where the vendors are most explicit that robots.txt may not apply, because a user requested the fetch, which is documented at OpenAI, Perplexity, and Meta.

What a crawl hit does and does not mean

It means a client identifying itself as that crawler requested that URL, and your server returned that status code. It means nothing is blocking the request at the network, firewall, or robots layer. That is a real and useful thing to confirm.

It does not mean the content parsed. It does not mean anything was retained. It does not mean an engine will use the page, and it certainly does not mean you will be cited. Fetching is upstream of everything that matters and predicts none of it.

The asymmetry is worth internalizing. Presence is weak evidence. Absence is strong evidence. A search crawler that has never touched your site across a month tells you something definite, and it tells you where to look next: your robots.txt rules, then your firewall, then whether anything links to you at all.

The damaging admission

Logs are lagging, noisy, and easy to over-read. You are counting requests from clients that describe themselves, in a format that predates all of this, and drawing conclusions about systems that publish almost nothing about how they choose sources.

We are also not going to pretend crawler counts are a metric worth optimizing. Chasing more GPTBot hits is chasing the wrong number. The only question the log answers well is binary: can they reach it, and did anything break when they tried.

And a large share of the people reading this cannot run any of it, because their platform does not expose a log. That is a genuine limit of this method, not a reason to switch hosts.

Check reachability the other way round

If you have logs, start with the missing names and the non-200 status codes. If you do not, check reachability from the other direction: ask what the engines can actually retrieve and parse from your pages right now.

Run a free scan to see how ChatGPT, Perplexity, Gemini, and Claude read your key pages today. The scan diagnoses any site, including platforms that never give you a log line. Automated apply through the connected Citedon plugin is WordPress only, and every command in this guide runs on any server you control.