Next.js App Router

Next.js App Router robots.txt and sitemap.ts Generator

This Next.js App Router robots.txt and sitemap.ts generator creates clean, fully typed TypeScript files for Next.js 14, 15, and 16 applications. Purpose-built for Next.js developers, Jamstack agencies, and React engineers, the tool produces app/robots.ts and app/sitemap.ts definitions compliant with official Next.js MetadataRoute interfaces. Configure your base domain, crawler disallow paths, sitemap endpoints, change frequencies, and route priorities to generate type-safe files ready to drop straight into your App Router project directory.

Next.js App Router Configuration

Strongly typed metadata route handlers for Next.js 14, 15 & 16

Base Domain:

Disallow Paths (one per line):Block AI Crawlers

Static Routes (path,frequency,priority):

import type { MetadataRoute } from 'next'

export default function robots(): MetadataRoute.Robots {
  const baseUrl = 'https://searchlearningtools.com'

  return {
    rules: [
      {
        userAgent: '*',
        allow: '/',
        disallow: [ "/api/", "/admin/", "/checkout/", "/temp/" ],
      },
    ],
    sitemap: `${baseUrl}/sitemap.xml`,
  }
}

What This Tool Does Not Check

This utility produces valid Next.js App Router TypeScript code adhering to MetadataRoute types. It does not query your local database or CMS to verify dynamic slug existence, nor does it benchmark build-time generation speeds for large enterprise catalogs.

Related Technical SEO Workflows

For static websites or non-React stacks, you can:

generate plain robots.txt files

Check how major AI search bots read your robots rules with our tool to:

audit your site for AI crawler permissions

Explore server rendering and crawl performance optimizations in our guide on:

Next.js technical SEO architecture

Protect your Next.js application at the CDN layer using our tool to:

configure edge response headers

Frequently Asked Questions

Where do app/robots.ts and app/sitemap.ts belong in Next.js?

In the Next.js App Router, place robots.ts and sitemap.ts directly inside the root app/ folder (or app/(group)/). Next.js automatically executes these route handlers during build or on request to serve /robots.txt and /sitemap.xml endpoints.

How does Next.js handle sitemap pagination for sites with over 50,000 URLs?

Next.js supports generateSitemaps() which allows returning an array of IDs. The framework then renders individual sub-sitemaps at /sitemap/[id].xml and creates an automated sitemap index at /sitemap.xml.

Can I include dynamic API data inside app/sitemap.ts?

Yes. Because app/sitemap.ts is an async function, you can fetch product slugs, blog posts, or localized paths directly from your database, headless CMS, or ORM inside the handler before returning the MetadataRoute.Sitemap array.

Does this generator support AI crawler directives (GPTBot, ClaudeBot)?

Yes. The generator allows you to configure custom rules for AI crawlers like GPTBot, ClaudeBot, and CCBot, either granting, restricting, or denying access to specific subpaths while keeping standard search crawlers allowed.

Continuous SEO Intelligence

Upgrade to the Starter Plan ($29/mo)

Monitor dynamic sitemap indexation and crawler errors with the Starter Plan ($29/mo).

View Starter Plan