XML sitemaps and robots.txt

Two small files at the root of most websites shape how search engines approach them. An XML sitemap is a list of URLs you would like search engines to know about. robots.txt is a set of instructions about which parts of the site crawlers may request. They are often mentioned together, and they are often confused, so this lesson takes them one at a time and then puts them side by side.

XML sitemaps

A sitemap helps Google discover URLs, especially on large sites, new sites with few links pointing to them, and sites whose pages are hard to reach through navigation. It does not force Google to crawl or index anything; it is a strong hint about what exists and what changed.

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/guides/sourdough-starter</loc>
    <lastmod>2026-09-14</lastmod>
  </url>
  <url>
    <loc>https://example.com/guides/proofing-bread</loc>
    <lastmod>2026-08-02</lastmod>
  </url>
</urlset>

The only required field is <loc>, the full URL. Google uses <lastmod> when it is consistently accurate, meaning it reflects the date the content meaningfully changed. Setting every page's lastmod to today on every build teaches Google to ignore the field. Google ignores <priority> and <changefreq>, so there is no point spending time on them.

What belongs in a sitemap

List the canonical, indexable URLs you want to appear in search results. Leave out URLs that redirect, return errors, carry noindex, or are duplicates pointing their canonical elsewhere. A sitemap full of URLs you do not want indexed sends mixed signals.

Limits and sitemap indexes

A single sitemap file can contain up to 50,000 URLs and must be no larger than 50 MB uncompressed. Larger sites split their URLs across several sitemaps and list those in a sitemap index file.

Telling Google about your sitemap

Submit the sitemap in Search Console's Sitemaps report, and add a line to robots.txt so any crawler can find it:

Sitemap: https://example.com/sitemap.xml

The Sitemaps report then shows whether Google could read the file and how many URLs it discovered.

robots.txt

robots.txt is a plain text file that must sit at the root of the host: https://example.com/robots.txt. It applies only to that exact host and protocol, so shop.example.com needs its own file. Files placed in subfolders are ignored.

The file is made of groups. Each group names one or more user agents and lists rules for them.

User-agent: *
Disallow: /admin/
Disallow: /cart/

User-agent: Googlebot
Disallow: /admin/

Sitemap: https://example.com/sitemap.xml

Here, crawlers in general are kept out of /admin/ and /cart/. Googlebot has its own group, and a crawler follows only the most specific group that matches it, so Googlebot obeys the second group and may crawl /cart/. This catches many people out: rules in the * group do not add on top of a crawler's own group.

When rules conflict within a group, Google applies the most specific matching rule, the one with the longest path. Allow: /admin/help/ beats Disallow: /admin/ for URLs inside /admin/help/.

Blocking crawling is not blocking indexing

This is the most important idea in the lesson. robots.txt controls crawling, meaning whether Googlebot may request a URL. It does not reliably keep a URL out of the index. If other pages link to a disallowed URL, Google can index the URL itself, often shown with no description, without ever reading its content.

To keep a page out of search results, let Google crawl it and add a noindex directive:

<meta name="robots" content="noindex">

For files such as PDFs, which have no HTML head, send the directive as an HTTP header instead: X-Robots-Tag: noindex.

A combination that never works is Disallow plus noindex on the same URL. The Disallow stops Google fetching the page, so it never sees the noindex.