XML sitemaps 101: how they help engines find you

Short answer: an XML sitemap is a machine-readable list of the URLs on your site that you want search engines to know about, each with an optional "last changed" date. It doesn't make anything rank and it doesn't guarantee indexing — but it is the cheapest, most reliable way to get a new or fast-changing site discovered and recrawled. What actually matters in 2026: an honest lastmod, only canonical indexable URLs, submission to Google and Bing, and a line in robots.txt. What doesn't matter: priority and changefreq, which Google ignores.

What a sitemap is — and isn't

At its simplest, a sitemap is a text file at /sitemap.xml that lists URLs:

<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/guides/some-page/</loc>
    <lastmod>2026-09-14</lastmod>
  </url>
</urlset>

Crawlers read it to discover pages and to decide which known pages are worth revisiting. That's the whole job. A sitemap is not a ranking signal, it won't force a page into the index, and search engines still decide for themselves whether each URL is worth crawling and keeping. Think of it as a well-organised invitation list, not a command.

The one field that matters: lastmod

lastmod is the only optional field with a confirmed job. Google uses it to decide whether a URL it already knows is worth recrawling: a page that says it changed last week gets revisited sooner than one untouched since 2023. The catch is that Google only trusts it when it's consistently and verifiably accurate. Two rules:

• Set it from the page's real "updated" timestamp — not from the moment the sitemap file was generated. Stamping every URL with today's date each time the file is rebuilt is the single most common sitemap bug, and it teaches Google to ignore the field entirely.
• Only change it when the content meaningfully changes. Bumping dates to trigger recrawls backfires for the same reason.

The fields Google ignores: priority and changefreq

Google has said repeatedly that it doesn't use priority or changefreq for crawling or ranking decisions — so many sites set everything to priority=1.0 and changefreq=daily that the signals became meaningless. Including them is harmless, but don't spend a minute tuning them.

What to include — and what never to

A sitemap should contain only the URLs you actually want indexed, in their final, canonical form:

• ✅ Canonical URLs that return 200, are indexable, and use your preferred host and scheme (https://, with or without www — but consistently).
• ❌ Pages marked noindex.
• ❌ URLs that redirect somewhere else.
• ❌ URLs that return 404 or any error.
• ❌ Non-canonical variants — parameter versions, tracking URLs, paginated duplicates, alternate formats like AMP copies (the canonical page's rel links handle those).
• ❌ Thin utility pages you'd rather engines didn't bother with.

Every bad URL in a sitemap wastes crawl budget and quietly erodes the engine's trust in the file.

Limits and structure

One sitemap file may hold up to 50,000 URLs and 50 MB (uncompressed; gzip is fine). Past that, use a sitemap index — a sitemap of sitemaps — and split by section or content type, which also makes Search Console's per-sitemap reporting far more useful. There are dedicated extensions for images, videos and news; they only help if that media is a real part of what you publish.

Where to put it and how to submit it

• Serve it at the root — https://yoursite.com/sitemap.xml — with a 200 status, not blocked by robots.txt.
• Add a line to robots.txt: Sitemap: https://yoursite.com/sitemap.xml. Every crawler that reads robots.txt then finds it automatically.
• Submit it in Google Search Console → Sitemaps and in Bing Webmaster Tools (importing your site from Search Console carries the sitemap across — see how to submit your site to Bing).
• After any migration or URL change, resubmit and check the report for errors.

When a sitemap matters most

Sitemaps are most valuable exactly when discovery is hardest: a brand-new site with few inbound links, a large site with thousands of pages, a site with weak internal linking or pages that are hard to reach by clicking, and any site that publishes or updates often. For a young site, pair it with the other discovery tools — Search Console's URL inspection, Bing's IndexNow, and a few real inbound links — as covered in how to get indexed by Google faster.

The mistakes that block indexing

A five-minute audit

• Open /sitemap.xml in a browser — it should load as XML with a 200 status.
• Count the URLs and spot-check ten: each should return 200 and be its own canonical.
• Check that lastmod values differ and match when pages actually changed.
• Confirm robots.txt references it and doesn't block it.
• Read the Sitemaps report in Search Console: "Success" with a sensible discovered-URL count, no errors.

Where AIO fits

AIO doesn't touch your sitemap — that's yours to maintain for your own pages. What the network adds is other sites' sitemaps: each directory in the network publishes its own, and when you broadcast a listing, that listing page is written into the directory's sitemap. That's a documented discovery path for crawlers — a page on another domain, listed in that domain's sitemap, linking to yours — which is part of why directory listings help a new site get found (see do directory backlinks still help SEO?). Your sitemap for your pages; the network's sitemaps for your listings.