User-agent: * Allow: / # AI Bots User-agent: GPTBot Disallow: / User-agent: Claude-Web Disallow: / User-agent: PerplexityBot Disallow: / Sitemap: https://example.com/sitemap.xml
Worked Example
Here is a worked example for the robots txt generator.
Frequently Asked Questions
What is the robots txt generator?