Generating sitemaps and robots.txt

In the world of search engines, content accessibility and its proper indexing form the foundation of effective SEO (Search Engine Optimization). Next.js offers advanced mechanisms and tools that facilitate the generation of sitemaps and robots.txt files - two key elements for search engines such as Google, Bing, or Yahoo.

The importance of sitemaps and the robots.txt file

Sitemap

A sitemap is an XML file that contains a list of all URLs of your page along with additional metadata, such as:

• Date of the last page modification
• Change frequency
• Priority of importance compared to other pages

A sitemap helps search engines in the following aspects:

1. Content discovery - informs about all pages on your site
2. Efficient indexing - helps understand the site structure
3. Prioritization - indicates which pages are most important
4. Recency - informs when pages were last modified

The robots.txt file

Robots.txt is a simple text file that contains instructions for search engine robots (crawlers). The main functions of robots.txt are:

1. Access control - determines which pages can be indexed and which cannot
2. Bot traffic management - prevents server overload by limiting access
3. Indicating the sitemap location - informs bots about the location of the sitemap.xml file

Manual sitemap generation in Next.js

Let's start with the simplest approach - manually creating a sitemap as a static XML file:

1// app/sitemap.xml/route.ts
2import { BASE_URL } from '@/lib/constants';
3import { getAllPostSlugs } from '@/lib/api';
4
5export async function GET() {
6  // Fetch dynamic data, e.g., URLs of all blog posts
7  const postSlugs = await getAllPostSlugs();
8  
9  // Current date for lastmod
10  const date = new Date().toISOString();
11  
12  // Creating the XML content
13  const sitemap = `<?xml version="1.0" encoding="UTF-8"?>
14    <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
15      <url>
16        <loc>${BASE_URL}</loc>
17        <lastmod>${date}</lastmod>
18        <changefreq>daily</changefreq>
19        <priority>1.0</priority>
20      </url>
21      <url>
22        <loc>${BASE_URL}/about</loc>
23        <lastmod>${date}</lastmod>
24        <changefreq>monthly</changefreq>
25        <priority>0.8</priority>
26      </url>
27      <url>
28        <loc>${BASE_URL}/products</loc>
29        <lastmod>${date}</lastmod>
30        <changefreq>weekly</changefreq>
31        <priority>0.9</priority>
32      </url>
33      ${postSlugs.map(slug => `
34        <url>
35          <loc>${BASE_URL}/blog/${slug}</loc>
36          <lastmod>${date}</lastmod>
37          <changefreq>weekly</changefreq>
38          <priority>0.7</priority>
39        </url>
40      `).join('')}
41    </urlset>`;
42  
43  // Return response with appropriate Content-Type
44  return new Response(sitemap, {
45    headers: {
46      'Content-Type': 'application/xml',
47      'Cache-Control': 'public, max-age=3600, s-maxage=3600'
48    }
49  });
50}

This approach is flexible and allows full control over sitemap content, but requires manual management and updates.

Automatic sitemap generation using the sitemap() function

Next.js 13.4+ introduced a special function

sitemap()

that simplifies sitemap generation. In this approach, you define a file

sitemap.ts

or

sitemap.js

in the main directory of the application:

1// app/sitemap.ts
2import { MetadataRoute } from 'next';
3import { getAllProducts } from '@/lib/products';
4import { getAllPosts } from '@/lib/blog';
5
6export default async function sitemap(): Promise<MetadataRoute.Sitemap> {
7  // Get all products
8  const products = await getAllProducts();
9  const productEntries = products.map(product => ({
10    url: `https://example.com/products/${product.slug}`,
11    lastModified: new Date(product.updatedAt),
12    changeFrequency: 'weekly' as const,
13    priority: 0.8,
14  }));
15  
16  // Get all blog posts
17  const posts = await getAllPosts();
18  const blogEntries = posts.map(post => ({
19    url: `https://example.com/blog/${post.slug}`,
20    lastModified: new Date(post.publishedAt),
21    changeFrequency: 'monthly' as const,
22    priority: 0.7,
23  }));
24  
25  // Static pages
26  const staticPages = [
27    {
28      url: 'https://example.com',
29      lastModified: new Date(),
30      changeFrequency: 'daily' as const,
31      priority: 1.0,
32    },
33    {
34      url: 'https://example.com/about',
35      lastModified: new Date(),
36      changeFrequency: 'monthly' as const,
37      priority: 0.5,
38    },
39    {
40      url: 'https://example.com/contact',
41      lastModified: new Date(),
42      changeFrequency: 'yearly' as const,
43      priority: 0.5,
44    },
45  ];
46  
47  // Combine all entries
48  return [...staticPages, ...productEntries, ...blogEntries];
49}

When you build the application, Next.js will automatically generate the file

/sitemap.xml

based on the returned data. Additional benefits of this approach:

• Built-in TypeScript typing - thanks to

  MetadataRoute.Sitemap

• Automatic validation - Next.js validates data structure correctness
• Integration with the build system - the sitemap is generated during the build

Generating a sitemap for large sites: multi-file approach

For larger sites with thousands of URLs, it is recommended to create multiple sitemaps and a sitemap index. Here is an example implementation:

1// app/sitemap-index.xml/route.ts
2export async function GET() {
3  const baseUrl = 'https://example.com';
4  
5  const sitemapIndex = `<?xml version="1.0" encoding="UTF-8"?>
6    <sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
7      <sitemap>
8        <loc>${baseUrl}/sitemaps/static-sitemap.xml</loc>
9        <lastmod>${new Date().toISOString()}</lastmod>
10      </sitemap>
11      <sitemap>
12        <loc>${baseUrl}/sitemaps/products-sitemap.xml</loc>
13        <lastmod>${new Date().toISOString()}</lastmod>
14      </sitemap>
15      <sitemap>
16        <loc>${baseUrl}/sitemaps/blog-sitemap.xml</loc>
17        <lastmod>${new Date().toISOString()}</lastmod>
18      </sitemap>
19    </sitemapindex>`;
20  
21  return new Response(sitemapIndex, {
22    headers: {
23      'Content-Type': 'application/xml',
24      'Cache-Control': 'public, max-age=3600, s-maxage=3600'
25    }
26  });
27}
28
29// app/sitemaps/static-sitemap.xml/route.ts
30export async function GET() {
31  // Logic for generating the static pages sitemap...
32}
33
34// app/sitemaps/products-sitemap.xml/route.ts
35export async function GET() {
36  // Logic for generating product sitemap...
37}
38
39// app/sitemaps/blog-sitemap.xml/route.ts
40export async function GET() {
41  // Logic for generating blog sitemap...
42}

This approach is scalable and allows each sitemap to stay below the 50,000 URL limit (the recommended maximum size of a single sitemap).

Creating a robots.txt file

Manual creation of robots.txt

The simplest way is to create a Route Handler file for robots.txt:

1// app/robots.txt/route.ts
2export async function GET() {
3  const robotsTxt = `# https://www.robotstxt.org/robotstxt.html
4User-agent: *
5Allow: /
6
7Disallow: /admin/
8Disallow: /api/
9Disallow: /internal/
10
11Sitemap: https://example.com/sitemap.xml`;
12  
13  return new Response(robotsTxt, {
14    headers: {
15      'Content-Type': 'text/plain',
16      'Cache-Control': 'public, max-age=3600, s-maxage=3600'
17    }
18  });
19}

Using the robots() function in Next.js

Next.js 13.4+ also introduced a dedicated function

robots()

to generate the robots.txt file:

1// app/robots.ts
2import { MetadataRoute } from 'next';
3
4export default function robots(): MetadataRoute.Robots {
5  return {
6    rules: {
7      userAgent: '*',
8      allow: '/',
9      disallow: ['/admin/', '/api/', '/internal/'],
10    },
11    sitemap: 'https://example.com/sitemap.xml',
12  };
13}

Similarly to the function

sitemap()

, Next.js will automatically generate the file

/robots.txt

based on the returned data, with built-in validation and TypeScript support.

Dynamic generation of sitemaps and robots.txt

For more advanced use cases, you can create environment-dependent or dynamic sitemaps and robots.txt files:

1// app/robots.ts
2import { MetadataRoute } from 'next';
3
4export default function robots(): MetadataRoute.Robots {
5  // Different rules for different environments
6  const isProduction = process.env.NODE_ENV === 'production';
7  
8  if (!isProduction) {
9    return {
10      rules: {
11        userAgent: '*',
12        disallow: '/',
13      },
14    };
15  }
16  
17  return {
18    rules: {
19      userAgent: '*',
20      allow: '/',
21      disallow: ['/admin/', '/api/'],
22    },
23    sitemap: 'https://example.com/sitemap.xml',
24  };
25}

In the example above, in the development environment, robots.txt blocks indexing of the entire site, while in production it allows indexing of most pages.

Verification and testing

After deploying sitemaps and the robots.txt file, it's important to test them to make sure they work correctly:

Testing tools

1. Google Search Console - add your page and submit the sitemap for verification
2. Bing Webmaster Tools - similarly to Google, offers verification tools
3. Sitemap and Robots.txt checking tools:
   • Screaming Frog SEO Spider
   • Google Search Console Robots Testing Tool

Manual verification

You can also manually check your files to make sure they contain the expected data:

1. Visit

   https://yourdomain.com/sitemap.xml

   in the browser - a properly formatted XML file should appear
2. Visit

   https://yourdomain.com/robots.txt

   - a proper text file should appear
3. Check sample URLs from the sitemap to make sure they are accessible and contain the expected content

Best practices

For sitemaps

1. Update regularly - generate sitemaps on every deployment or content change
2. Include only canonical URLs - avoid duplicates and alternative versions of the same content
3. Use accurate lastmod dates - do not generate the date dynamically if the content has not changed
4. Include only important pages - skip administrative, login, cart pages, etc.
5. Include alternative language versions - use the element

   <xhtml:link rel="alternate">

   for multilingual pages

For the robots.txt file

1. Be selective with Disallow - block only those paths that really should not be indexed
2. Use appropriate User-agent - consider different rules for different robots
3. Add a link to the sitemap - always include a reference to sitemap.xml
4. Comment your rules - add comments explaining why you block certain paths
5. Test changes - verify that changes in robots.txt do not block important pages

Automating sitemap updates

For pages that change frequently, consider automating sitemap generation:

1// scripts/generate-sitemap.js
2const fs = require('fs');
3const path = require('path');
4const { getAllDynamicPaths } = require('../lib/api');
5
6async function generateSitemap() {
7  console.log('Generating sitemap...');
8  
9  // Get all dynamic paths
10  const dynamicPaths = await getAllDynamicPaths();
11  
12  // Generate XML
13  const date = new Date().toISOString();
14  const sitemap = `<?xml version="1.0" encoding="UTF-8"?>
15<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
16  <url>
17    <loc>https://example.com</loc>
18    <lastmod>${date}</lastmod>
19    <priority>1.0</priority>
20  </url>
21  ${dynamicPaths.map(path => `
22  <url>
23    <loc>https://example.com${path}</loc>
24    <lastmod>${date}</lastmod>
25    <priority>0.8</priority>
26  </url>
27  `).join('')}
28</urlset>`;
29  
30  // Save to file
31  const publicPath = path.join(process.cwd(), 'public');
32  fs.writeFileSync(path.join(publicPath, 'sitemap.xml'), sitemap);
33  
34  console.log('Sitemap generated successfully!');
35}
36
37generateSitemap().catch(console.error);

This script can be run as a step in the deployment process or through a scheduled task.

Integration with a CMS

If you use a CMS (Content Management System), you can automatically generate sitemaps based on content managed by the CMS:

1// app/sitemap.ts
2import { MetadataRoute } from 'next';
3import { getCmsClient } from '@/lib/cms';
4
5export default async function sitemap(): Promise<MetadataRoute.Sitemap> {
6  const cms = getCmsClient();
7  
8  // Get all pages from the CMS
9  const pages = await cms.getPages();
10  const pageEntries = pages.map(page => ({
11    url: `https://example.com/${page.slug}`,
12    lastModified: new Date(page.updatedAt),
13    changeFrequency: 'weekly' as const,
14    priority: 0.8,
15  }));
16  
17  // Get all posts from the CMS
18  const posts = await cms.getPosts();
19  const postEntries = posts.map(post => ({
20    url: `https://example.com/blog/${post.slug}`,
21    lastModified: new Date(post.updatedAt),
22    changeFrequency: 'weekly' as const,
23    priority: 0.7,
24  }));
25  
26  // Combine all entries
27  return [
28    {
29      url: 'https://example.com',
30      lastModified: new Date(),
31      changeFrequency: 'daily' as const,
32      priority: 1.0,
33    },
34    ...pageEntries,
35    ...postEntries,
36  ];
37}

Handling image and video sitemaps

For sites with a lot of images or video, it's worth considering specialized sitemaps:

Image sitemap

1// app/image-sitemap.xml/route.ts
2import { getAllProductImages } from '@/lib/products';
3
4export async function GET() {
5  const images = await getAllProductImages();
6  
7  const sitemap = `<?xml version="1.0" encoding="UTF-8"?>
8    <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
9            xmlns:image="http://www.google.com/schemas/sitemap-image/1.1">
10      ${images.map(image => `
11        <url>
12          <loc>https://example.com/products/${image.productSlug}</loc>
13          <image:image>
14            <image:loc>https://example.com${image.url}</image:loc>
15            <image:title>${image.title}</image:title>
16            <image:caption>${image.caption}</image:caption>
17          </image:image>
18        </url>
19      `).join('')}
20    </urlset>`;
21  
22  return new Response(sitemap, {
23    headers: {
24      'Content-Type': 'application/xml',
25      'Cache-Control': 'public, max-age=3600, s-maxage=3600'
26    }
27  });
28}

Summary

Sitemap and robots.txt are key elements of any Next.js application that wants to be visible in search results. Next.js 13.4+ introduced dedicated functions

sitemap()

and

robots()

that significantly simplify the process of generating these files, while providing TypeScript support and automatic validation.

Remember that effective SEO is not just technical aspects like sitemaps, but a broader set of practices, including content quality, loading speed, and accessibility. Generating sitemaps and robots.txt is, however, an important step towards increasing the visibility of your Next.js application in search engines.