SearchLearningToolsBot Documentation
Official transparency documentation for the SearchLearningToolsBot automated research scanner.
1. User-Agent String
User-Agent: SearchLearningToolsBot/1.0 (+https://searchlearningtools.com/bot)
2. What SearchLearningToolsBot Does
SearchLearningToolsBot is an automated research scanner operated by Search Learning Tools to compile the weekly AI Crawler Access Index.
- Fetches Only
/robots.txt: The scanner never crawls HTML content, images, assets, or internal web pages. - Strict Timeout & Size Caps: Enforces a 5-second timeout, maximum 3 HTTP redirects, and 500KB response cap per domain.
- Rate-Limited Concurrency: Honors HTTP 429 Too Many Requests response headers with exponential backoff.
3. How to Opt-Out or Block the Scanner
If you prefer to exclude your website's /robots.txt from our weekly public research index, add the following disallow rule to your robots.txt file:
User-agent: SearchLearningToolsBot
Disallow: /
Disallow: /