llms.txt and AI-Crawler Rules Across 500 Small-Business Websites

Most writing about llms.txt argues about whether the file is worth publishing. Very little of it says how many sites actually have one.

So on 2026-08-06 we fetched three files from 500 small-business websites: /robots.txt, /llms.txt, and /llms-full.txt. Nothing else. No crawling, no page parsing, three requests per site.

The number that changed how we read the results was not the adoption rate. It was that a third of the llms.txt files we found were signed by an SEO plugin.

What we measured

Every percentage below is out of the 455 sites that answered. 45 of the 500 never returned an HTTP response on any of the four attempts, so they are excluded from every rate and counted here instead of being quietly dropped.

A separate 7 sites returned their robots.txt when we asked for /llms.txt, which is a catch-all server response rather than a published file. Those are counted as not having an llms.txt, and the exclusion is recorded in the results file.

A third of the llms.txt files were signed by an SEO plugin

This is the finding we did not expect. Of the 101 llms.txt files, 36 carried a generator signature in their first 400 characters:

Several of these files open with a line the plugin writes itself, along the lines of "Generated by All in One SEO, this is an llms.txt file, used by LLMs to index the site". A plumber in Denver did not weigh the llmstxt.org spec and decide. The file appeared when the plugin updated.

That matters for how you read the 22.2%. Some of it is owners acting. Some of it is a WordPress plugin release rolling out a default across its install base. The 65 files with no detectable signature are not proof of hand-authorship either, since a generator can ship without signing its output.

Sizes varied by four orders of magnitude. The median llms.txt was 3,357 bytes. The smallest was 111 bytes, a barbershop file consisting of little more than its own domain name as a heading. The largest was 1,389,162 bytes on a car dealership, an inventory dump of link lines rather than the short curated index the spec asks for.

llms-full.txt barely exists here

8 sites out of 455 served an llms-full.txt, and all 8 also served an llms.txt. Nobody in this sample published the concatenated file without the index file.

That tracks with what the specification actually says: llms-full.txt is not part of it. It is a convention that documentation platforms popularised, and small-business sites are not documentation platforms.

What these sites tell AI crawlers

362 sites had a robots.txt. Most of them say nothing at all about AI.

Here is every site that named one of the four headline tokens, and what the rule did. "Full block" means a Disallow: / in that user-agent's own group. "Partial" means the group disallows some paths but not the whole site, which on WordPress is usually /wp-admin/. "Allowed" means the group exists and blocks nothing.

Google-Extended is worth reading carefully, because it is not a crawler. Google documents it as a control over whether already-crawled content is used in Gemini and AI answers, not as a user-agent that fetches pages. It was still the most-named token in the sample.

PerplexityBot is the outlier in the other direction. It was named by only 13 sites, and 10 of those allowed it. Whatever is driving the blocking behavior, it is not being applied evenly across engines.

Vendors also publish separate names for training crawlers and for the fetchers behind live answers. We parsed those from the same files at no extra cost:

The shape here is consistent. The training and scraping crawlers get named and blocked. The search-side fetchers, the ones that go and read a page because someone asked a question right now, are named far less often and blocked far less often when they are. On the evidence of these files, the sites that made a decision mostly made it about training.

By business category

Sample sizes per category are small, between 29 and 51 reachable sites, so treat the spread as a description of this sample rather than a ranking of industries.

The last column counts a site that named any of the fourteen AI tokens parsed in this study, not only the four headline ones, so it sums to 55 rather than the 52 in the first table.

Methodology

Sample source. The sample is drawn from OpenStreetMap, a public dataset under the ODbL license, via the Overpass API. We queried a fixed list of twelve US metro boxes, each exactly 0.30 degrees of latitude by 0.36 of longitude (roughly 33 km by 30 km) centered on the metro's downtown coordinates: Los Angeles, Dallas, New York, Miami, Chicago, Philadelphia, Columbus, Atlanta, Charlotte, Detroit, Seattle, and Denver. Within those boxes we pulled every record carrying one of ten fixed small-business tags: office=lawyer, amenity=dentist, amenity=veterinary, office=accountant, office=insurance, office=estate_agent, craft=plumber, craft=electrician, shop=hairdresser, and shop=car_repair.

That query is the complete sampling frame. It returned 14,649 OSM records, of which 4,146 carried a usable website tag.

Selection. Website values were normalised to a host. 85 records pointing at a social network, booking platform, or directory profile rather than a domain the business controls were excluded, using a list fixed in advance, because robots.txt and llms.txt are properties of a site you control. After deduplicating by host, 3,471 unique domains remained. Those were ordered by sha256(seed + host) using a fixed seed and taken round-robin across the ten categories until we had 500. That ordering is a reproducible shuffle computed before anything was measured, so no site was chosen for what it turned out to contain.

Measurement. For each domain we requested https://<domain>/robots.txt, /llms.txt, and /llms-full.txt, following redirects, with a 15-second timeout and a self-identifying user-agent. Four workers ran at a time, so no site received more than one request at once and each received three in total. A domain counted as reachable if any request, or a single fallback request to /, returned an HTTP status of any kind.

A file counted as present only on HTTP 200 with a non-empty body that did not look like an HTML document, because a large number of sites answer any unknown path with a 200 HTML page. Bodies whose first line was a robots.txt directive were also excluded, as described above.

Robots parsing. Consecutive User-agent lines share the rule block that follows them, per the robots.txt convention. A token counted as "named" only when it had its own group. The counts are what the file says, not what any crawler actually did.

Dates. Files were fetched on 2026-08-06. The OpenStreetMap snapshot behind the frame is timestamped 2026-08-06T15:03:01Z.

The script, the selected sample, and the full per-domain results are committed in the repository at scripts/studies/. Re-running the harness against the same sample will reproduce these counts, allowing for sites that change between runs.

What this does not tell you

The limits are as load-bearing as the findings, so they get their own list.

This is not a measurement of the web, or of US small business. It is 500 sites from a specific OpenStreetMap query over twelve metro cores in ten trades. Businesses with no website, or with a website not recorded in OSM, cannot appear. Businesses whose OSM entry points at a Facebook page were removed by design. Urban cores are over-represented relative to where US small businesses actually are.

OpenStreetMap coverage is uneven and volunteer-maintained, and its category tags can be wrong. We spot-checked one record tagged office=lawyer and found an accounting firm whose own homepage describes it as "Experienced Accountants". Some entries are stale, some are mis-tagged, and a domain in OSM may no longer belong to the business. That noise lands squarely on the per-category table and is a reason to read it as descriptive rather than precise.

Presence is not quality. We measured that a file exists at a path. We did not check whether an llms.txt conforms to the llmstxt.org spec, whether its links resolve, or whether its content matches the site. At least one file we counted was a set of robots-style permission directives under a markdown heading rather than the curated index the spec describes.

A robots.txt rule is a statement, not an outcome. These counts describe what the files say. They do not show whether any crawler obeyed, and they do not show what a site's owner intended. A Disallow may have arrived through a plugin, a host default, or a developer years ago.

Percentages on the category table rest on 29 to 51 sites each. The spread between plumbers at 39.6% and veterinary practices at 11.8% is real in this sample and is not a reliable industry ranking.

Nothing here says whether any of it works. We did not measure citations, traffic, or whether an engine read any of these files. No engine vendor we have found publishes a commitment to fetch or honor llms.txt. The value of this study is the state of the files, not a claimed effect.

What we take from it

Two things.

The first is that the interesting number is not 22.2%. It is 36 of 101. When a measurement of adoption is substantially a measurement of plugin defaults, the honest reading is that the tooling moved before the owners did. If you are deciding whether to publish an llms.txt because "a fifth of sites already have one", you are partly looking at your competitors' plugin update logs.

The second is the silence. 362 sites had a robots.txt and 55 sites named any AI user-agent in it, 52 of them one of the four headline tokens. That leaves the large majority making no statement either way. Not a block, not an allow, just a file written for search engines that predates the question.

You can find out where your own site sits on both counts in about a minute. The scan checks whether engines can read your pages, and it works on any site.

Reproduce it

Both scripts are in the repository:

node scripts/studies/fetch-osm-sample.mjs   # rebuilds the 500-domain sample
node scripts/studies/llms-txt-audit.mjs     # re-runs the three fetches per site

The first re-queries OpenStreetMap live, so it will build its frame from a newer snapshot than ours and select a different 500. To reproduce these exact counts, skip it and run only the second script against the committed smb-sample.json.

The first writes scripts/studies/data/smb-sample.json, including the per-metro and per-category frame counts. The second writes scripts/studies/data/llms-txt-adoption-results.json, including the classification for every domain, so any number above can be traced back to the site it came from.