snag.page

Every check ยท AI and agents

AI crawlers

agents/ai-crawlers

What it checks

What robots.txt says to GPTBot, ClaudeBot, Google-Extended, CCBot and other AI crawlers, and any Content-Signal line.

Why it matters

It is the site's choice whether AI crawlers may read it; this says which choice it has made.

How to fix it

  1. Add a group to robots.txt for each crawler to block, with Disallow: /, or leave them out to allow them.
/robots.txt
User-agent: GPTBot
Disallow: /

Check a site for this and everything else: snag.page. More on ai and agents: llms.txt