ai-visibility-audit

What it does

• Reads robots.txt as the assistants' crawlers read it, per named agent, so a blanket rule and a rule aimed at one bot are told apart.
• Parses llms.txt and follows its links: a guide that points at pages that are gone is worse than no guide, and that is the common case.
• Judges each page on what is in the HTML - word count, who published it, usable structured data, whether the copy answers a question anyone asks.
• Every check runs offline in the self-test, so a change to the scoring cannot pass unnoticed.

How it is proved

31 of 31 offline self-checks pass, each pinning one rule, and the same audit found the survey finding published on this site: of the top 1,500 sites, one in ten that publish an llms.txt also ban a crawler that would read it.

Pythonrequests31 offline self-checksper-crawler robots.txt

People find this looking for

can chatgpt read my website, gptbot blocked robots.txt check, claudebot crawler access audit, llms.txt validator, why is my site not cited by AI, AI visibility checker.

If that is your week and you would rather someone else did it, write to hello@dkautomation.dev or open an issue.