Robots.txt Tester

Test whether any crawler can access any path on your site — and read the rules that decide it.

What this tool does

The Robots.txt Tester fetches a site's robots.txt file and gives you a clear verdict: can this crawler access this path — allowed or blocked? Choose from Googlebot, Bingbot, AI crawlers like GPTBot, ClaudeBot, and PerplexityBot, or the wildcard agent, and test any path. You also get the file's full contents, its rule counts, and any sitemap declarations it contains.

Robots.txt is a tiny file with catastrophic failure modes. A single misplaced 'Disallow: /' — often left over from a staging environment — can silently deindex an entire site, and syntax subtleties around wildcards and rule precedence mean rules frequently don't do what their author intended. The rise of AI crawlers adds a fresh decision: whether GPTBot and friends may use your content, and whether your file actually says what you've decided.

Test your money pages against Googlebot today, and test them again after every deploy that touches site config — accidental blocking is a classic post-launch bug. Once crawlers can reach your content, SirWritesALot helps you make it worth ranking.

More free tools