GPTBot

GPTBot is the name most site owners learn first and the one they most often use wrongly. It is a training crawler. It is not the agent that decides whether ChatGPT can surface your pages, and treating those as one decision is the single most common self-inflicted mistake in this area.

What GPTBot is

GPTBot is the user agent OpenAI documents for crawling content that may be used in training its generative AI foundation models (OpenAI crawlers, checked 2026-08-06).

That is the whole job. It is not fetching your page to answer a question, and it is not building the index that ChatGPT's search feature reads.

The other three, which matter more than you think

OpenAI documents four agents, and the distinction between them is the entire point of this page.

Read the second row again. OAI-SearchBot is the one attached to visibility inside ChatGPT's search answers. GPTBot is attached to training.

So a site that disallows GPTBot and allows OAI-SearchBot has opted out of training while remaining available to ChatGPT search. A site that disallows both has done something quite different, and a lot of sites have done the second while intending the first, usually through a plugin toggle labeled "block AI bots" that treats every agent as one switch.

OpenAI states the settings work independently and that "it can take ~24 hours from a site's robots.txt update for our systems to adjust."

Why the separation exists at all

Because they are different bargains.

Training collection takes your content into a model's weights, with no link back and no ongoing relationship. Search indexing makes your content findable inside an assistant, with your URL attached. Those are not the same trade, and reasonable site owners land differently on each.

The vendors have built the controls to match. Anthropic documents ClaudeBot, Claude-SearchBot and Claude-User separately. Perplexity documents PerplexityBot and Perplexity-User separately. Every one of them is a distinct token. A robots.txt written as a single anti-AI block collapses distinctions the vendors deliberately built, and usually costs more visibility than it protects. AI crawler covers the full landscape.

The user-initiated exception

ChatGPT-User is a different category from the rest, and it is where blanket-block reasoning breaks.

It fires because a person asked a question that touched your page, right then, and OpenAI notes that because the action is initiated by a user, robots.txt rules may not apply. Perplexity says the same about Perplexity-User. Meta says the same about meta-externalfetcher.

So robots.txt is not a complete switch for every kind of AI access, and anyone who tells you a disallow line makes your site invisible to assistants is describing a simpler world than the documented one.

In a log, this traffic also looks different: irregular, bursty, often a single URL rather than a sweep. That pattern is closer to a demand signal than a crawl statistic.

Verifying a GPTBot hit

The user agent is a request header the client writes itself. Anything can claim to be GPTBot, and things do.

OpenAI publishes IP address files for verification, and each of these was fetched on 2026-08-06 and returned JSON:

• openai.com/gptbot.json
• openai.com/searchbot.json
• openai.com/chatgpt-user.json

OAI-AdsBot has one too, at openai.com/adsbot.json (note the path: oai-adsbot.json returns 404).

If a crawler number is going into a report, take the source addresses from your log and check them against those files first. How to find AI crawlers in your server logs has the commands.

What a GPTBot hit does and does not prove

It proves that a client identifying itself as GPTBot requested a URL and your server returned a status code. Nothing at the network, firewall, or robots layer refused it. That is a real thing to confirm.

It does not prove that the content parsed, that anything was retained, or that any engine will use the page. Fetching is upstream of everything that matters and predicts none of it.

The asymmetry is worth internalizing. Presence is weak evidence. Absence is strong evidence, and absence of a search crawler is stronger still. OAI-SearchBot at zero across a month is a finding with a short list of causes: a robots.txt rule, a firewall or WAF rule, an IP block, or nothing on the web linking to you.

And a hit count is not a metric to optimize. Chasing more GPTBot visits is chasing the wrong number entirely, since GPTBot is the training crawler and training is not the surface you are trying to appear on.

The common failure, stated once

A site owner reads about AI scraping, enables a "block AI crawlers" setting, and six months later cannot work out why an assistant never surfaces their pages.

The block did exactly what it said. The setting just covered more agents than the owner meant, and the search crawlers were among them.

Before you change anything, read your current robots.txt against each vendor's current token list, agent by agent, and decide each one separately. Allow or block AI crawlers in robots.txt walks that decision.

The honest limit

Being crawlable is the floor, not the finish. A page GPTBot and OAI-SearchBot can both reach, with a buried answer and no structure, is still a poor read.

Citedon measures whether engines can reach and read your pages, and on WordPress helps build the machine-readable layer that is usually missing, applied through a connected plugin with a preview and a per-fix approval. It cannot make an engine choose you, and it does not claim to. On other platforms you get the diagnosis and apply it yourself.

Vendor token lists also change without notice. Every name on this page was read from OpenAI's documentation on 2026-08-06, and a robots.txt written against last year's list is a guess about this year's crawlers.

How to check yours

Open your robots.txt and read it against the four agents above by name rather than from memory. Then confirm that a crawler can actually fetch the text of the page you would most want an engine to use.

Run a free scan on that URL to see which engines can reach and read it today.