Fix: robots.txt declares AI-crawler rules (RFC 9309)

Web-audit fix skill for the robots-ai-rules check (Bot & crawl policy, SHOULD).

Goal

State your AI-crawler policy in robots.txt with explicit User-agent rules.

Fix

Add explicit User-agent rules for AI crawlers (GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot, and others) to robots.txt, or a wildcard policy. Stating your stance (RFC 9309) removes ambiguity about whether AI systems may fetch your content.

Resources

• RFC 9309

Copy-paste prompt

Paste this into your coding agent, replacing the Issue line with the Result line from your audit:

Verify

Re-run the audit at https://anc.dev/web-audit or call the audit_website MCP tool; the robots-ai-rules check should report pass.