Media & Analyst Resource Center
Press Briefing: AI Crawler Access Index
Key empirical findings, media graphics, ready-to-embed charts, and citation sources for journalists, AI policy researchers, and search analysts.
Key Empirical Findings (As of 2026-10-02)
Finding #1
Bytespider Most Blocked AI Bot
45.0% of top domains explicitly disallow Bytespider (ByteDance) in robots.txt.
Finding #2
3.5 AI Bots Blocked Per Site
Top web domains disallow an average of 3.5 out of 11 AI crawlers, reflecting targeted crawler policy rules.
Finding #3
Applebot-Extended Access Highest
Applebot-Extended (Apple) experiences the lowest block rate at 25.0%.
Embeddable Chart for Articles & Publications
Copy and paste this HTML snippet to embed an interactive, auto-updating chart directly into your news story or blog post:
<iframe src="https://searchlearningtools.com/embed/ai-crawler-index" width="100%" height="450" frameborder="0"></iframe>AI Crawler Block Rate Breakdown
Percentage of top 20 web domains blocking each AI bot via robots.txt
Blocked Allowed
Media & Research Inquiries
For custom data exports, interviews, or research questions, contact the engineering team:
hello@searchlearningtools.com