Generating and Caching Dynamic Sitemaps and robots.txt

Introduction

A sitemap.xml is like a map that hands search engines a list saying "this site has these pages."
With thousands to tens of thousands of SKUs, managing that by hand just isn't realistic, right?

In this article, I'll explain how, on a headless EC site for motorcycle gear, I generated a sitemap.xml and robots.txt that automatically track additions and removals of products and categories, and how I used caching to control generation cost. I'll also touch on URL normalization via Edge middleware to prevent duplicate URLs.

Generating Sitemaps and robots Dynamically

Why Dynamic Generation Is Needed

Products are added daily and disappear when they sell out. If you build a static sitemap once and leave it, new products may never get indexed and URLs for removed products may linger. In headless, this auto-updating that the theme handled disappears, so you need to provide it yourself.

So I made it dynamic: the sitemap.xml is assembled from the current product and category list on each request. robots.txt is likewise returned dynamically, controlling which URLs you do and don't want crawled.

Controlling What You Don't Want Seen with noindex

You don't want every page indexed. For example, products outside a sales channel or pages you don't want in search get excluded with noindex. Limit the sitemap to "URLs you want in search," and suppress what you don't want shown via robots.txt or a meta noindex. This two-tier setup reduces wasted crawling.

Controlling Generation Cost with Caching

Full Generation Every Time Is Heavy

Fetching a list of tens of thousands of SKUs and assembling XML on every request is a fairly heavy operation. Full generation each time crawlers visit frequently loads both the server and the data source.

So I cache the generated sitemap result and return the same content for a set period. Product additions and removals are reflected when the cache expires, balancing freshness against load.

Monitoring with a Stats API

Leave auto-generation entirely on its own and you won't notice incidents like "the sitemap's URL count quietly cratered." So I built a stats API that returns things like the URL count within the sitemap, letting me monitor the generated results. A sudden swing in URL count is a sign to suspect a data-fetch failure or a misconfiguration.

Preventing Diluted Evaluation with URL Normalization

Why Duplicate URLs Are a Problem

Even for the same product, if the URL has mixed-case letters or a tracking query parameter, search engines may treat them as separate URLs. Then the SEO evaluation that should concentrate on one page is diluted across several URLs, leaving all of them half-evaluated.

Unifying with 301s via Edge Middleware

To prevent this, I normalize with Edge middleware. Mixed-case URLs are lowercased, unnecessary query parameters are stripped, and then it 301-redirects to the canonical URL. A 301 means "permanently moved," so evaluation carries over to the canonical URL.

Processing at the Edge normalizes lightly before reaching the app itself, applying consistently across all pages.

Conclusion

The essence of dynamic sitemaps and robots is to automatically track product changes while controlling generation cost with caching. Monitor the URL count with a stats API to prevent quietly-progressing incidents. Further normalize URLs with Edge middleware to prevent diluted evaluation from duplicates, keeping the site consistent from a search engine's view.

Once "SSR visibility," "structured data," and "dynamic sitemaps + normalization" are all in place, you can reclaim the SEO that headless tends to lose. The big picture is summarized in "SEO Design for Headless EC," so please take a look there too.