AI crawler
An automated client operated by an AI company that fetches web pages, either to build training data, to index the web for an assistant's search, or to retrieve a page in the moment a user asks a question.
Something has been fetching your pages that is not Googlebot. It may be building training data, it may be indexing you for an assistant's search, or it may be reading one page right now because someone asked a question about you.
Those are three different jobs, done by three different clients, and they are controlled separately.
What an AI crawler is
An AI crawler is an automated client operated by an AI company that requests pages from your server. The name covers more than one behavior, which is exactly where most confusion starts.
Broadly there are three purposes, and the vendors document them as distinct agents:
- Training collection. Fetching content that may be used to train or improve models.
- Search indexing. Fetching and indexing pages so an assistant's search feature can surface them.
- User-initiated retrieval. Fetching one page in the moment, because a user asked something that requires reading it.
The third is the one most site owners underestimate. It is not a scheduled sweep. It is a live read of your page while an answer is being written.
How AI crawlers differ from search crawlers
A classic search crawler has one job: fetch pages, build an index, feed a ranking system that returns links. One agent, one pipeline, one outcome you recognize as a search result.
AI crawlers split that into separate agents with separate consent, and the output is a composed answer rather than a link. That changes two practical things.
First, control is per purpose. Allowing training collection and allowing search indexing are different decisions, and each vendor gives you a different token for each.
Second, the fetch that decides whether you appear in an answer may happen after the question is asked, not weeks earlier. A page that is slow, script-dependent, or gated can fail that live read even though a scheduled crawler indexed it fine.
The crawlers you are most likely to see
The table below reflects each vendor's own published documentation, checked on 2026-07-30. Treat it as a snapshot: these lists get new entries, and the only current source is the vendor page itself.
| Vendor | Agents | What each is documented to do |
|---|---|---|
| OpenAI | GPTBot, OAI-SearchBot, ChatGPT-User, OAI-AdsBot | GPTBot crawls content that may be used in training foundation models. OAI-SearchBot surfaces sites in ChatGPT's search features. ChatGPT-User visits a page for a user-initiated action. OAI-AdsBot checks pages submitted as ads. |
| Anthropic | ClaudeBot, Claude-SearchBot, Claude-User | ClaudeBot collects web content that may contribute to model training. Claude-SearchBot indexes content to improve search result quality. Claude-User retrieves a page in response to a user's question. |
| Perplexity | PerplexityBot, Perplexity-User | PerplexityBot surfaces and links sites in Perplexity search results and is documented as not being used to crawl content for foundation models. Perplexity-User visits a page to answer a specific user request. |
| Google-Extended | A robots.txt control token rather than a separate user agent. It governs whether content Google crawls may be used for training Gemini models and for grounding in Gemini apps. Google states it does not affect inclusion or ranking in Google Search. |
Sources: OpenAI's crawler documentation at developers.openai.com; Anthropic's help center article on web crawling, dated April 7, 2026; Perplexity's crawler documentation at docs.perplexity.ai; Google's list of common crawlers, last updated 2026-07-14. All checked 2026-07-30.
One detail worth reading twice: both OpenAI and Perplexity document that their user-initiated fetchers may not apply robots.txt rules, on the reasoning that a person explicitly asked for that page. So a robots.txt entry is not a complete switch for every kind of AI access.
How they show up in your logs
Three signals, in order of how much they prove.
The user-agent string is the first and weakest. It is self-reported and trivially spoofed, so it tells you what a client claims to be.
Published IP ranges are the check that matters. OpenAI, Anthropic, and Perplexity each publish JSON files of the addresses their crawlers operate from, and verifying against those is how you confirm a request is genuine. Google documents both its own IP range files and a reverse DNS pattern for its common crawlers.
Then there is response status. A crawler that reached you and got a 403, a redirect loop, or a timeout is a crawler that saw nothing, and that failure looks identical in an AI answer to never having been fetched at all.
How robots.txt controls them
For scheduled crawlers, control works the way it always has: name the user-agent token in a group and set the rules for it. Each vendor publishes the exact token to use, and matching one token is enough for that group to apply.
Two cautions that are easy to get wrong.
A blanket disallow addressed at every AI agent also removes you from the assistant search surfaces, not just from training. Those are separate tokens on purpose, and treating them as one decision usually costs more visibility than it protects.
And the tokens change. Vendors add agents, rename them, and split existing ones. A robots.txt written against last year's list is a guess about this year's crawlers.
If your intent is the opposite, inviting rather than restricting, that is what an llms.txt is for. robots.txt restricts, llms.txt points to what you most want read.
The honest limit
Being reachable by an AI crawler is the floor, not the finish. Citedon measures whether engines can reach and read a page and, on WordPress, helps build the machine-readable layer that is often missing. It cannot make an engine choose you, and it does not claim to.
A crawlable page with a buried answer and no labels is still a poor read. Access gets you into the room. Being legible once inside is the separate job described under crawlability and agent-readiness.
How to check yours
Open your robots.txt and read it against the current vendor tokens rather than from memory. Then take the page you would most want an engine to use and confirm a crawler can actually fetch its text.
Run a free scan on that URL to see which engines can reach and read it today, or work through the guide on how to fix crawlability issues.