X-Robots-Tag Header Generator
This X-Robots-Tag header generator builds HTTP response header rules to control crawler indexing on non-HTML assets and dynamic URLs. Crafted for technical SEO auditors, backend developers, and site owners, the generator creates directives like noindex, nofollow, noarchive, and noimageindex for PDF documents, image files, video media, and paginated queries. Choose your target files and web server environment to generate ready-to-deploy configuration blocks for Nginx, Apache .htaccess, Cloudflare _headers, and Next.js next.config.ts headers.
X-Robots-Tag HTTP Header Directives
Control search indexing on non-HTML assets, PDFs, and filtered queries
location ~* \.(pdf)$ {
add_header X-Robots-Tag "noindex, noarchive" always;
}What This Tool Does Not Check
This tool outputs syntactically correct HTTP response headers. It does not inspect whether your robots.txt currently disallows crawler access to the target files (which would prevent search bots from reading the HTTP header).
Related Technical SEO Workflows
For dynamic filter parameters and e-commerce query strings, use our tool to:
configure faceted navigation robots rulesCombine crawler indexing directives with security headers using our tool to:
generate Cloudflare edge headersLearn how to prevent crawl budget waste on media assets in our guide on:
non-HTML document indexing strategiesEnsure your robots.txt disallows match your HTTP headers by visiting our tool to:
create compliant robots.txt filesFrequently Asked Questions
When should I use X-Robots-Tag instead of an HTML meta robots tag?
HTML meta robots tags only work on HTML web pages. Non-HTML assets—such as PDF downloads, DOCX files, raw images, and JSON endpoints—cannot contain HTML meta tags. An X-Robots-Tag HTTP response header is the only valid way to instruct Googlebot and Bingbot not to index these files.
Can I apply X-Robots-Tag headers to entire subdirectories or file extensions?
Yes. Web servers like Nginx and Apache allow regex matching on file extensions (e.g., location ~* \.(pdf)$) or directory locations to apply the X-Robots-Tag header to all matching files automatically.
Why does Google still index PDFs when robots.txt disallows them?
If robots.txt blocks Googlebot from crawling a PDF URL, Googlebot never fetches the file and therefore never sees any noindex directive. If other websites link to that PDF, Google may index the bare URL without snippets. To completely remove a PDF from search results, allow it in robots.txt and serve X-Robots-Tag: noindex.
What are the most common X-Robots-Tag directives?
Common directives include noindex (prevents search display), nofollow (prevents following outbound links), noarchive (prevents cached versions), nosnippet (prevents text preview snippets), and noimageindex (prevents images on the page from appearing in image search).
Upgrade to the Starter Plan ($29/mo)
Audit header response codes and crawl status across all non-HTML assets with the Starter Plan ($29/mo).