Search & AI Discovery Readiness Auditor
Enter any domain to see how its robots.txt, llms.txt and XML sitemap work together for Google and AI answer engines. You get a readiness score, an AI bot access matrix and copy-ready fixes.
What the audit checks
The score combines four pillars. Each one deducts points for real problems, weighted by how much they matter for being found and cited.
1. Cross-file contradictions (30%). These are the expensive mistakes: publishing an llms.txt while robots.txt blocks the AI search bots that would use it, listing URLs in your sitemap or llms.txt that robots.txt forbids, and sitemaps that robots.txt never declares or that return errors.
2. robots.txt posture and crawl efficiency (30%). The robots.txt parser follows RFC 9309, including how specific bot groups override the * group and how the longest matching rule wins. It classifies your AI posture (Smart Split, Wide Open, AI Blackout or Misconfigured), checks for crawl-trap rules that protect crawl budget on large sites, and flags deprecated or invalid lines.
3. llms.txt agentic map (15%). It catches the soft 404, where a site returns its HTML app shell with a 200 status for /llms.txt, then checks the file against the spec: one H1 title, a blockquote summary, H2 sections, descriptive links and a sensible size.
4. XML sitemap signal hygiene (25%). It detects index versus flat sitemaps and samples real URLs. It flags synthetic lastmod dates (when a CMS stamps every URL with the same build time), dead-weight priority and changefreq tags, and URLs with the wrong protocol, the wrong host or tracking parameters.
How it works
Your browser asks this site to fetch the four public discovery files from the domain you enter. The fetch runs at the edge with strict limits: 5-second timeouts, a 1.5 MB cap per file, and results cached for 5 minutes. Every check then runs in your browser. No AI or paid APIs are involved, and the audit results aren't stored.
Each visitor gets 10 free audits per day, which reset at midnight UTC. The tool keeps an anonymous daily count for that limit and never stores your IP address.
Large sites often use bot protection that blocks automated requests. If a file shows as unreachable, check it in your own browser before acting on the result.
FAQ
What does the Discovery Readiness Auditor check?
It fetches a domain's robots.txt, llms.txt, llms-full.txt and XML sitemap, then checks four things. It looks for contradictions between those files, grades robots.txt posture for traditional and AI crawlers, lints llms.txt against the spec, and audits sitemap signals like lastmod accuracy and URL hygiene.
Does this tool use AI?
No. Every check is deterministic code that follows the published standards (RFC 9309 for robots.txt, the llms.txt proposal and the sitemaps protocol). The same input always produces the same result.
What is the Smart Split AI posture?
Smart Split means a site lets AI search and citation bots like OAI-SearchBot, PerplexityBot and Claude-SearchBot crawl it, so it can be cited in AI answers, while blocking AI training scrapers like GPTBot, CCBot and Google-Extended. It's the posture most enterprise brands should consider.
Why does the audit say my robots.txt is unreachable?
Either your server returned an error, or a CDN or bot-protection service blocked the auditor's request. Under RFC 9309, crawlers that can't fetch robots.txt because of a server error must treat the whole site as disallowed, so it's worth checking either way.
Want your brand to be the answer?
Talk to John about search, AI visibility, or building an earned and owned strategy that lasts.