Robots.txt Generator for WordPress: Custom WP Robots Tool
Create a WordPress robots.txt file with editable rules for selected paths, WooCommerce, sitemap URLs, and named search or AI crawlers. Robots.txt is not a privacy or security control.
Custom Robots Txt Generator Tool
Configure robots txt generator for WordPress CMS & WooCommerce
Core WordPress Crawl Directives
WordPress robots.txt Output
Place in the root public_html/ directory
# WordPress Robots.txt Configuration # Generated via SearchLearningTools.com WordPress Generator User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Disallow: /wp-includes/ Allow: /wp-includes/js/ Allow: /wp-includes/images/ Allow: /wp-includes/css/ # WooCommerce Crawl Budget Protection Disallow: /cart/ Disallow: /checkout/ Disallow: /my-account/ # Block Internal WP Search Indexing Disallow: /?s= Disallow: /search/ Disallow: /trackback/ Disallow: /*/feed/ Disallow: /*/comments/ # Generative AI Search & Citation Crawlers (Explicit Allow) User-agent: GPTBot Allow: / User-agent: OAI-SearchBot Allow: / User-agent: PerplexityBot Allow: / User-agent: ClaudeBot Allow: / Sitemap: https://example.com/sitemap_index.xml
Worked Example Output & WordPress Robots Directives
Sample Input Scenario: WordPress WooCommerce site configuring crawler directives for public and utility paths.
Worked Industry Example Output:
User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Disallow: /wp-includes/ Allow: /wp-includes/js/ Allow: /wp-includes/css/ Disallow: /cart/ Disallow: /checkout/ Disallow: /my-account/ Disallow: /?s= User-agent: GPTBot Allow: / User-agent: PerplexityBot Allow: / Sitemap: https://mystore.example.com/sitemap_index.xmlThis worked example demonstrates how admin-ajax.php remains accessible for script rendering while private checkout pages and internal queries are shielded from indexation.
Frequently Asked Questions About WordPress Robots.txt
A WordPress robots.txt file lets site owners publish crawl directives for compliant bots. Rules can help manage crawling, but they are not access controls: do not use them to protect private URLs or confidential data.
Some WordPress themes and plugins rely on admin-ajax.php for functionality. Review how your site works before blocking it; robots.txt rules can affect crawling of URLs but are not a reliable way to secure or hide content.
Upload the robots.txt file directly to your website root directory (public_html) using SFTP or cPanel File Manager. Alternatively, paste the rules into SEO plugins like Yoast SEO or Rank Math via their File Editor tool.
Choose crawler rules based on the provider and your publishing preferences. Search, user-triggered fetch, and model-training bots may have different names and uses; allowing a crawler does not guarantee citation, and robots.txt is a voluntary protocol rather than an access-control system.
Explore Companion Crawling & Technical SEO Tools
Optimize your crawl budget, test AI search bot directives, and accelerate indexation:
Guide: Creating a Robots Txt File
Deep-dive handbook on syntax, wildcards, and common crawl errors.
Website AI Crawler Checker
Verify if GPTBot, ClaudeBot, and CCBot are permitted or blocked.
Standard Robots.txt Generator
Universal robots.txt creator supporting Next.js, Shopify, and custom stacks.
Free AI Visibility Checker
Audit your brand presence across modern LLM engines and chat tools.
Why Use AI Search Monitoring
Learn what AI search monitoring can measure and where crawler access fits.
Help Google Discover New Page
Check crawl access and connect the page through an XML sitemap and internal links.