# Fix: robots.txt declares AI-crawler rules (RFC 9309)

> Web-audit fix skill for the `robots-ai-rules` check (Bot & crawl policy, SHOULD).

## Goal

State your AI-crawler policy in robots.txt with explicit User-agent rules.

## Fix

Add explicit `User-agent` rules for AI crawlers (GPTBot, ClaudeBot, Google-Extended,
PerplexityBot, CCBot, and others) to `robots.txt`, or a wildcard policy. Stating your stance
(RFC 9309) removes ambiguity about whether AI systems may fetch your content.

## Resources

- [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309)

## Copy-paste prompt

Paste this into your coding agent, replacing the Issue line with the Result line from [your audit](https://anc.dev/web-audit):

```text
Goal: State your AI-crawler policy in robots.txt with explicit User-agent rules
Issue: <the audit's finding for this check>
Fix: Add explicit `User-agent` rules for AI crawlers (GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot, and others) to `robots.txt`, or a wildcard policy. Stating your stance (RFC 9309) removes ambiguity about whether AI systems may fetch your content.
Skill: https://anc.dev/web-audit/skill/robots-ai-rules
Docs: https://www.rfc-editor.org/rfc/rfc9309
```

## Verify

Re-run the audit at [https://anc.dev/web-audit](https://anc.dev/web-audit) or call the `audit_website` MCP tool; the `robots-ai-rules` check should report `pass`.