Next.js Sitemap and robots.txt - A Practical Guide for SA Sites

A Next.js sitemap is a sitemap.ts file in your app folder that returns every public, indexable URL on your site, and robots.ts sits next to it to tell crawlers what they may fetch and where the sitemap lives. Both are generated by Next.js itself, so they stay in step with your content without a plugin. The hard part is not the code: it is deciding what belongs in the sitemap, keeping lastmod dates honest and making sure preview and staging copies of your site never end up in Google.

This guide walks through all of that for the App Router in Next.js 15 and 16.

What a sitemap is (and is not)

A sitemap is a hint, not an instruction. It tells search engines "these are the URLs I want indexed, and this is when each one last changed". Google still decides what to crawl and index. A good sitemap helps new pages get discovered quickly and helps search engines understand which version of a URL you consider canonical. A bad one, full of redirects, duplicates and pages that say noindex, teaches crawlers to trust it less.

What belongs in it

• Pages that return HTTP 200 and that you want in search results.
• The canonical version of each URL only: one host (www or not), https, and the same trailing-slash style your canonical tags use.
• Blog posts, product pages, category pages and paginated listing pages that are indexable.

What does not

• Redirects, 404s and anything marked noindex.
• Sign-in, account, checkout, admin and API routes.
• URLs with tracking parameters (?utm_source=...) or filters that produce duplicate content.
• Preview, staging or development hosts.

If you are unsure which host is canonical, settle that first. Our guide to www or non-www and HTTPS redirects explains how to pick one and redirect the rest.

A sitemap.ts that grows with your content

Put this in app/sitemap.ts. Next.js serves the result at /sitemap.xml.

import type { MetadataRoute } from "next";
import { getAllPosts } from "@/lib/posts"; // your CMS, database or Markdown files

const SITE = "https://www.example.co.za";

// The last real change to the static pages. Bump it when their content changes.
const STATIC_UPDATED = "2026-09-30";

export default async function sitemap(): Promise<MetadataRoute.Sitemap> {
  const posts = await getAllPosts();
  const staticPages = ["/", "/pricing", "/about", "/contact"].map((path) => ({
    url: `${SITE}${path}`,
    lastModified: STATIC_UPDATED,
  }));
  return [
    ...staticPages,
    { url: `${SITE}/blog`, lastModified: posts[0]?.updated ?? posts[0]?.date ?? STATIC_UPDATED },
    ...posts.map((post) => ({
      url: `${SITE}/blog/${post.slug}`,
      lastModified: post.updated ?? post.date,
    })),
  ];
}

A few details matter here:

• Absolute URLs. Sitemap URLs must be full URLs on your canonical host, not paths.
• One source of truth. The posts come from the same function that renders the blog, so a new post appears in the sitemap on the next build without anyone remembering to add it.
• Static by default. Unless you use dynamic functions, Next.js generates the sitemap at build time. That is what you want for most sites: it is fast and cached.

Be honest with lastmod

It is tempting to set lastModified: new Date() for every page. Don't. A date that changes on every deploy says "everything changed", which is rarely true, and search engines learn to ignore lastmod values that are always "now". Use the real publish or update date for content, and a constant you bump by hand for static pages, as above. The changeFrequency and priority fields are optional and Google says it ignores them, so leave them out unless another search engine you care about uses them.

Very large sites

One sitemap file can hold up to 50,000 URLs. If you have more, Next.js can split them with generateSitemaps, which produces several numbered sitemap files. The Next.js sitemap documentation shows the exact file names it generates. Most small business and SaaS sites never get near the limit.

A robots.ts that does not block what you need

Put this in app/robots.ts. Next.js serves it at /robots.txt.

import type { MetadataRoute } from "next";

export default function robots(): MetadataRoute.Robots {
  return {
    rules: { userAgent: "*", allow: "/", disallow: ["/api/", "/account/"] },
    sitemap: "https://www.example.co.za/sitemap.xml",
  };
}

Three rules of thumb:

1. Never block CSS, JavaScript or /_next/. Google renders pages like a browser. If it can't load your scripts and styles, it may misjudge the page.
2. robots.txt is not security. It is public and only well-behaved crawlers follow it. Protect private pages with authentication, not a Disallow line.
3. Disallow and noindex are different. Disallow stops crawling, but a blocked URL can still appear in results if other sites link to it, and Google can't see a noindex tag on a page it isn't allowed to fetch. To keep a page out of results, let it be crawled and mark it noindex.

To mark a single page noindex in the App Router, set it in the page's metadata:

export const metadata = { robots: { index: false, follow: true } };

Keep previews and staging out of Google

Preview deployments are one of the best things about Git-based hosting, and one of the easiest ways to create duplicate content. If a preview URL gets linked from a ticket, a chat or a public pull request, it can be crawled. The safest fix is a response header that tells search engines not to index anything that isn't served from your production host:

// next.config.ts
import type { NextConfig } from "next";

const nextConfig: NextConfig = {
  async headers() {
    return [
      {
        source: "/:path*",
        missing: [{ type: "host", value: "www.example.co.za" }],
        headers: [{ key: "X-Robots-Tag", value: "noindex" }],
      },
    ];
  },
};

export default nextConfig;

Because the rule keys off the request's host rather than an environment variable, it works the same on every preview, branch and staging copy, with no extra configuration per environment. If your previews sit behind a password, even better. Read more about how previews work in preview deployments explained.

Check it before you submit it

After deploying, open /sitemap.xml and /robots.txt in a browser, or from a terminal:

curl -s https://www.example.co.za/robots.txt
curl -s https://www.example.co.za/sitemap.xml | head -n 20

Check that:

• every <loc> starts with your canonical https:// host;
• a few URLs picked at random open without redirecting;
• robots.txt has a Sitemap: line with the full URL;
• nothing you need is disallowed.

Submit it to search engines

Add the sitemap URL in Google Search Console under Sitemaps. If you haven't set up Search Console yet, our guide on verifying your domain in Google Search Console walks through the DNS record step. Bing Webmaster Tools accepts the same sitemap. After that, watch the Pages report for URLs that were "Discovered" or "Crawled" but not indexed: it is often the quickest way to find duplicate or thin pages you didn't know about.

A sitemap is only one item on the launch list. For the rest, see our Next.js production checklist.

Frequently asked questions

Does a sitemap improve my rankings?

Not directly. It helps search engines discover and recrawl your pages, which matters most for new sites, large sites and pages with few internal links. Rankings still depend on the content and how useful it is.

Should I include paginated blog pages in the sitemap?

If they are indexable and have their own canonical URL, yes. If your listing pages are marked noindex or canonicalise to page one, leave them out so the sitemap matches what the pages themselves say.

How often should the sitemap update?

Whenever the content changes. With sitemap.ts reading the same data as your pages, that happens automatically on each build. You don't need to resubmit it in Search Console after every change; search engines refetch it on their own.

Can I have more than one sitemap?

Yes. Large sites split sitemaps by section or by count and list them in a sitemap index. You can also list several Sitemap: lines in robots.txt.

Why does Search Console say a sitemap URL is blocked by robots.txt?

A Disallow rule matches a URL you listed. Either the rule is too broad or the URL shouldn't be in the sitemap. Fix whichever is wrong so the two files agree.

Deploying a Next.js site for South African visitors? NewHost Next.js hosting builds from your Git repository on South African servers, with free SSL on your own domain and preview deploys per branch on the Developer plan and up.