Dmytro Galko · open data
llms.txt conformance across the 1,500 most visited sites · 2026-09-19
How many sites publish an llms.txt has been counted several
times this year, and the answers agree. This is the next question, which
nobody seems to have asked: of the files that exist, how many actually
work — the right shape, live links inside, and a robots.txt that lets the
reader in.
An llms.txt is a map an assistant follows to your content.
A map with dead streets on it is worse than no map: the assistant spends
its one visit on a 404 and leaves. Nothing tells you when that happens.
13 of the 123 sites publishing a usable
llms.txt — 10.6% — also name at least one AI crawler in
their robots.txt and forbid it outright. They wrote a guide
for a reader they are not letting through the door.
This is not the wildcard rule being read as a ban. Only a group naming the
crawler explicitly counts here, and only Disallow: / counts as
a block, because anything looser inflates the number.
Publish an llms.txt | 123 | 16.1% of 765 answered |
|---|---|---|
| Have the documented shape | 107 | 87% |
| No heading | 3 | |
| No sections | 7 | |
| No links at all | 12 |
A file with no links in it is a file that points an assistant nowhere. It exists, it would be counted as adoption by every published figure, and it does nothing.
| Sites with at least one dead link | 15 | 13.5% of 111 |
|---|---|---|
| Links checked | 864 | |
| Resolved | 797 | |
| Dead | 36 | 4.2% |
| Could not verify | 31 | not counted as dead |
At most eight links per site were followed, spaced out, so this is a floor and not a ceiling: a site with one dead link among its first eight may have more further down.
| Crawler | Blocked outright | Share |
|---|---|---|
GPTBot | 106 | 15.7% |
ClaudeBot | 109 | 16.1% |
Google-Extended | 94 | 13.9% |
CCBot | 129 | 19.1% |
PerplexityBot | 89 | 13.2% |
Bytespider | 114 | 16.9% |
Applebot-Extended | 75 | 11.1% |
Out of 675 sites that publish a robots.txt at all. A crawler
that is never mentioned is counted as not mentioned, never as allowed and
never as blocked.
Population: Tranco list V3YPN, top 1500 domains. Run on 2026-09-19 with
well-known-audit/survey/llms-conformance.mjs.
Politeness: 4 domains at a time, 250ms between batches, 300ms between link checks, at most 8 links per site, no retries, honest User-Agent. A refusal is recorded as a refusal and never retried.
Of 1,500 domains, 735 could not be asked at all: 418 never answered — most of these are CDN and infrastructure names that carry no website — 133 refused the request, and 184 answer 200 to a path that cannot exist, so nothing they return is evidence of anything. None of the three is inside any percentage on this page.
each domain is first asked for a path that cannot exist; a 200 there means the site answers 200 to anything, and it is excluded from every rate.
only a group naming the crawler explicitly counts; the wildcard group is not evidence about a named bot, and only Disallow: / counts as a full block.
What this cannot tell you: whether any assistant actually read these
files, and whether a site meant to publish one — a large share of
llms.txt files on the web are written by a platform or a
plugin rather than chosen by the publisher.
Want to know what your own domains publish?
npx well-known-audit yourdomain.com, or write to
hello@dkautomation.dev.