How to allow or block AI crawlers in robots.txt

Most sites have never made a decision about AI crawlers. They have a robots.txt file written for Googlebot in 2019, possibly with a wildcard rule someone added to stop a scraper, and whatever that file happens to say is now the policy.

That is worth ten minutes of your attention, in either direction. If you want AI engines reading your pages, a stale rule can be blocking them. If you do not want your content in a training set, the tokens to say so exist and are published.

This guide lists the current names, sourced, and is honest about what allowing and blocking actually buy you.

The tokens, and where each one comes from

Every name below was checked against the vendor's own documentation on 2026-07-30. These lists change, so treat this as a starting point and verify against the linked pages before you publish.

OpenAI

Source: OpenAI crawlers documentation, checked 2026-07-30.

OpenAI publishes four user agents and states that each robots.txt setting is independent of the others.

• GPTBot crawls content that may be used in training OpenAI's generative AI foundation models. Disallowing it signals your content should not be used for that training.
• OAI-SearchBot is the search crawler. OpenAI's documentation says sites opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links, and recommends allowing it if you want to appear in those results.
• ChatGPT-User is used for user-initiated fetches, such as when someone asks ChatGPT a question and it visits a page. OpenAI's own page notes that because these actions are initiated by a user, robots.txt rules may not apply.
• OAI-AdsBot visits landing pages submitted as ads on ChatGPT.

OpenAI also notes it can take roughly 24 hours after a robots.txt change for search behavior to adjust.

Anthropic

Source: Anthropic crawler support article, page dated April 7, 2026, checked 2026-07-30.

Anthropic publishes three robots and states they honor standard robots.txt directives.

• ClaudeBot collects web content that could contribute to model training. Restricting it signals that future material should be excluded from training datasets.
• Claude-SearchBot crawls to improve search result quality. Anthropic's page says disabling it prevents indexing for search, which may reduce your visibility and accuracy in user search results.
• Claude-User handles user-initiated retrieval when someone asks Claude a question. Anthropic says disabling it prevents retrieval of your content in response to a user query.

Anthropic also supports the non-standard Crawl-delay extension for its bots.

Perplexity

Source: Perplexity crawlers documentation, checked 2026-07-30.

• PerplexityBot surfaces and links websites in Perplexity search results. Perplexity's documentation states it is not used to crawl content for AI foundation models, and recommends allowing it if you want to appear in results.
• Perplexity-User handles user-initiated visits. Perplexity's page states that because a user requested the fetch, this fetcher generally ignores robots.txt rules.

Google

Source: Google's common crawlers list, page last updated 2026-07-14, checked 2026-07-30.

• Google-Extended is the one most people mean. Google describes it as a standalone control token with no separate user-agent string of its own: it governs whether content Google crawls may be used for training future Gemini models and for grounding in Gemini apps. Google's page also states that Google-Extended does not affect a site's inclusion in Google Search and is not used as a ranking signal.
• Googlebot remains the crawler for Google Search, including its search features. If you block Googlebot you are making a Search decision, not an AI-only one.
• Google-CloudVertexBot covers crawls that site owners request for building Vertex AI agents, and GoogleOther is a generic crawler used across product teams.

Write the rules

A robots.txt group is a User-agent line and the rules beneath it. Tokens are matched individually, so each crawler you care about gets its own group.

To allow the search and answer crawlers while declining training use:

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: ClaudeBot
Disallow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Disallow: /

To simply allow everything, the cleanest move is usually to remove the blanket rule that is blocking them rather than to add named allow groups, since most crawlers default to permitted.

Two mechanical traps. A misspelled token does nothing at all and fails silently, so check the spelling against the source pages. And a wildcard group with a Disallow: / can be what is actually stopping these crawlers, regardless of what your named groups say, depending on how the file is written.

The honest tradeoff

A blocked crawler cannot read the page. That is the whole tradeoff, and it does not have a clever version.

If you disallow a search or answer crawler, that engine is not weighing your content against a competitor's and losing. It is not seeing it. Whatever the engine says about your category, it will say without you in the room.

The reverse is also true and we will not dress it up: allowing every AI crawler does not get you cited. It removes one reason you cannot be. Whether an engine then uses your page depends on the page, the question, and everyone else's page, and it changes as models change.

There are real reasons to block. Not wanting your writing in a training corpus is a legitimate position, and the vendors publish separate tokens precisely so you can hold it without also removing yourself from search surfaces. Just make that a decision rather than an inheritance from an old file.

One more honest note: robots.txt is a request. The four vendors above document that their automated crawlers honor it, and two of them state plainly that their user-triggered fetchers may not apply those rules, because a person asked for the page. Anything outside these vendors is outside this guide.

Robots.txt is the floor, not the work

Getting the file right means an engine is permitted to fetch the page. It says nothing about whether the page is readable once fetched, which is a different problem covered in fix crawlability issues for AI engines and, for the content itself, in the structure guides.

The complementary move is pointing engines at your best pages rather than only telling them what they may not touch. That is what an llms.txt file is for, and how to write an llms.txt file covers it. It is a proposed convention, not a rule any engine must obey.

Check what is actually reachable

Read your robots.txt, decide training and search separately, then confirm the pages you meant to keep open are genuinely reachable and readable.

Run a free scan to see how ChatGPT, Perplexity, Gemini, and Claude read your key pages today. The scan diagnoses any site. The automated apply through the connected Citedon plugin is WordPress only, and a robots.txt file is a text file you can edit yourself on any platform in about a minute.