Screenshot Every Page of a Website from Its sitemap.xml

A client hands you a site and asks for a redesign. Or a migration went live at 2 a.m. and someone needs to confirm that all 400 URLs still render. Or the quarterly report needs a visual inventory of what the site currently looks like, page by page, before anyone touches it. In all three cases the job is the same: capture every page the site says it has, once, reproducibly, into a folder you can hand to a designer or diff against next month.

The site already tells you what it has. That’s what sitemap.xml is for. What follows is a working Node script that reads it — including the nested-index case that quietly defeats most tutorials — filters it, captures it with bounded concurrency, and survives dying halfway through.

Why the naive sitemap parser captures nothing

Here is the thing people get wrong, and it fails silently rather than loudly.

A small site’s sitemap.xml is a <urlset> containing <url><loc> entries. That’s the shape every tutorial handles. But past a few thousand URLs — and on essentially every WordPress, Shopify, or enterprise CMS install regardless of size — sitemap.xml is instead a sitemap index: a <sitemapindex> containing <sitemap><loc> entries that point at further sitemaps. /post-sitemap.xml, /page-sitemap.xml, /product-sitemap-1.xml, and so on. Frequently those children are served gzipped, as .xml.gz.

A parser that only looks for <url> elements finds zero of them in a sitemap index. It doesn’t error. It reports “0 URLs found”, you assume the client’s sitemap is broken, and you go and write a crawler instead. The fix is about fifteen lines: detect which document type you got, and recurse.

Gzip has its own trap. fetch transparently decompresses a response carrying Content-Encoding: gzip, but a .xml.gz file is usually served as gzip content with Content-Type: application/gzip — the bytes arrive still compressed and you have to unpack them yourself. The reliable test is the gzip magic number at the start of the buffer, which works whichever way the server chose to do it.

The script

Node 20 or newer, no dependencies. The blocks below are one file, sitemap-capture.mjs, in order.

// sitemap-capture.mjs
import { appendFile, mkdir, readFile, writeFile } from "node:fs/promises";
import { createHash } from "node:crypto";
import { gunzipSync } from "node:zlib";
import path from "node:path";

const API_KEY = process.env.SNAPSHOT_SITE_API_KEY;
const ENDPOINT = "https://api.prod.ss.snapshot-site.com/api/v1/screenshot";

const OUT_DIR = "captures";
const MANIFEST_FILE = "captures/manifest.ndjson";
const CONCURRENCY = 8;
const MAX_SITEMAP_DEPTH = 2;

const CAPTURE_OPTIONS = {
  format: "png",
  width: 1440,
  fullSize: true,
  hideCookie: true,
  delay: 2,
};

delay is in seconds, with an accepted range of 0 to 10 — a much narrower window than the millisecond field most browser-automation APIs expose, which is why habit tends to win here and produce a validation error on the very first URL. Two is a reasonable settle time for a site you don’t control.

Fetching and parsing, including the index case

async function fetchSitemap(url) {
  const response = await fetch(url, {
    headers: { "user-agent": "sitemap-capture/1.0" },
  });
  if (!response.ok) throw new Error(`${url} -> HTTP ${response.status}`);

  const buffer = Buffer.from(await response.arrayBuffer());
  // A .xml.gz file arrives as gzip *content*, which fetch does not unwrap.
  // Test the magic number rather than trusting the content type.
  const isGzip = buffer[0] === 0x1f && buffer[1] === 0x8b;
  return (isGzip ? gunzipSync(buffer) : buffer).toString("utf8");
}

function decodeXml(value) {
  return value
    .replace(/<![CDATA[([sS]*?)]]>/g, "$1")
    .replace(/&lt;/g, "<")
    .replace(/&gt;/g, ">")
    .replace(/&quot;/g, '"')
    .replace(/&apos;/g, "'")
    .replace(/&amp;/g, "&");
}

function parseEntries(xml, tag) {
  const blocks = new RegExp(
    `<(?:\w+:)?${tag}\b[^>]*>([\s\S]*?)</(?:\w+:)?${tag}>`,
    "gi",
  );
  const locPattern = /<(?:w+:)?locb[^>]*>([sS]*?)</(?:w+:)?loc>/i;
  const lastmodPattern =
    /<(?:w+:)?lastmodb[^>]*>([sS]*?)</(?:w+:)?lastmod>/i;

  const entries = [];
  for (const block of xml.matchAll(blocks)) {
    const body = block[1];
    const loc = body.match(locPattern);
    if (!loc) continue;
    const lastmod = body.match(lastmodPattern);
    entries.push({
      loc: decodeXml(loc[1].trim()),
      lastmod: lastmod ? lastmod[1].trim() : null,
    });
  }
  return entries;
}

async function collectUrls(sitemapUrl, depth = 0, seen = new Set()) {
  if (depth > MAX_SITEMAP_DEPTH || seen.has(sitemapUrl)) return [];
  seen.add(sitemapUrl);

  const xml = await fetchSitemap(sitemapUrl);

  // A <sitemapindex> yields <sitemap> children; a <urlset> yields <url>.
  const children = parseEntries(xml, "sitemap");
  if (children.length > 0) {
    const collected = [];
    for (const child of children) {
      try {
        collected.push(...(await collectUrls(child.loc, depth + 1, seen)));
      } catch (error) {
        console.warn(`skipping ${child.loc}: ${error.message}`);
      }
    }
    return collected;
  }

  return parseEntries(xml, "url");
}

Two deliberate choices. The regex allows an optional namespace prefix, because plenty of real sitemaps emit <sm:url> or similar, and b stops <urlset> from being mistaken for a <url> block. And a failing child sitemap warns and continues rather than aborting the run — one dead /product-sitemap-7.xml should not cost you the other six.

Regex parsing of XML is a compromise, and worth naming as one. Sitemaps are machine-generated, extremely regular, and constrained by a published schema, so it holds up in practice. If you’re parsing something that isn’t a sitemap, use a real parser.

Filtering the list

The sitemap is the site’s claim about itself, not your capture list. Archives, tag pages and feeds inflate the count without adding anything a designer or a stakeholder will look at.

const EXCLUDE = [
  //page/d+/i,
  //tag//i,
  //author//i,
  //feed/?$/i,
  /[?&]replytocom=/i,
  /.(?:xml|json|pdf|jpe?g|png|webp|gif|svg)$/i,
];

function keepUrl(entry, since) {
  if (EXCLUDE.some((pattern) => pattern.test(entry.loc))) return false;
  if (since && entry.lastmod && new Date(entry.lastmod) < since) return false;
  return true;
}

The --since filter uses <lastmod> to skip pages that claim not to have changed since your last pass, which on a large site is the difference between 400 captures and forty. Treat it as an optimisation rather than as truth: lastmod is self-reported, plenty of static-site generators stamp every URL with the build timestamp, and some CMSes never update it at all. Note that entries with no lastmod at all are kept, which is the safe default.

Be clear about what this filter is for: it’s a duplicate-cost guard, not change detection. Its sibling on the client side is the caching pattern for avoiding duplicate capture costs, which hashes the capture parameters to avoid paying twice for a request you already made — and which is explicitly the wrong tool for asking whether a page changed, since that question needs a fresh capture by definition. If “what changed since last quarter?” is the actual deliverable, that’s the compare endpoint, and it gets its own section at the end of this article.

Deterministic output filenames

This is the small detail that decides whether two runs are comparable. Derive the filename from the URL path, not from a counter or a timestamp, and the same page lands at the same path every single time — which means a run in August and a run in November can be diffed directory against directory.

function outputPath(pageUrl) {
  const { hostname, pathname, search } = new URL(pageUrl);
  const slug =
    `${pathname}${search}`
      .replace(//index.html?$/i, "/")
      .replace(/[^a-zA-Z0-9]+/g, "-")
      .replace(/^-+|-+$/g, "")
      .toLowerCase() || "home";

  // The slug is lossy — it lowercases, and collapses every run of
  // non-alphanumerics to one dash — so always suffix a hash of the full URL.
  // Deterministic, and no two URLs can land on the same file.
  const digest = createHash("sha1").update(pageUrl).digest("hex").slice(0, 8);
  const stem = slug.slice(0, 110).replace(/-+$/, ""); // truncation can land on a dash
  const safe = `${stem}-${digest}`;

  return path.join(OUT_DIR, hostname, `${safe}.${CAPTURE_OPTIONS.format}`);
}

https://example.com/blog/how-we-migrated/ becomes captures/example.com/blog-how-we-migrated-<hash>.png, and the homepage becomes captures/example.com/home-<hash>.png, where <hash> is the first eight hex characters of SHA-1 over the full URL. Both filenames are identical next month, which is the whole point — the slug keeps the name readable, the suffix keeps it unique.

The hash suffix is not decoration. Slugifying is lossy in three ways that bite on real sites: .toLowerCase() merges /About and /about, collapsing non-alphanumeric runs merges /services/seo with /services-seo, and the query string merges /search?q=shoes with /search-q-shoes. Each collision means the second capture silently overwrites the first, two manifest lines point at one file, and your directory-against-directory diff quietly compares the wrong pages. Eight hex characters of SHA-1 over the full URL costs nothing and removes the entire class of problem.

The manifest, written as you go

A 400-page audit that dies at page 300 because a laptop slept, a token expired, or the office wifi dropped should cost you one page of work, not three hundred. Write each result the moment it lands.

let writeChain = Promise.resolve();

function record(entry) {
  // Serialise manifest writes: eight workers appending concurrently to one
  // file is a race, and a chained promise is cheaper than a lock.
  // The .catch() keeps one failed append from poisoning the chain for
  // every later write — losing one manifest line beats losing the run.
  writeChain = writeChain
    .then(() => appendFile(MANIFEST_FILE, `${JSON.stringify(entry)}n`))
    .catch((error) => console.warn(`manifest write failed: ${error.message}`));
  return writeChain;
}

async function loadManifest(file) {
  try {
    const text = await readFile(file, "utf8");
    const done = new Map();
    for (const line of text.split("n")) {
      if (!line.trim()) continue;
      const entry = JSON.parse(line);
      done.set(entry.url, entry); // last write wins
    }
    return done;
  } catch (error) {
    if (error.code === "ENOENT") return new Map();
    throw error;
  }
}

Newline-delimited JSON rather than one big JSON object, because an append-only log has no read-modify-write step to lose. Last-write-wins on load means a URL that failed on Monday and succeeded on Tuesday reads as succeeded, so re-running the script is also how you retry failures.

Capturing, with retries

async function capture(pageUrl) {
  const response = await fetch(ENDPOINT, {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      Accept: "application/json",
      "x-snapshotsiteapi-key": API_KEY,
    },
    body: JSON.stringify({ url: pageUrl, ...CAPTURE_OPTIONS }),
  });

  if (!response.ok) {
    const error = new Error(`capture HTTP ${response.status}`);
    error.status = response.status; // so withRetry can tell 429 from 401
    throw error;
  }

  const payload = await response.json();
  if (!payload.link) throw new Error(payload.message || "no link in response");
  return payload.link;
}

async function download(link, destination) {
  const response = await fetch(link);
  if (!response.ok) throw new Error(`download HTTP ${response.status}`);
  await mkdir(path.dirname(destination), { recursive: true });
  await writeFile(destination, Buffer.from(await response.arrayBuffer()));
}

async function withRetry(task, attempts = 3) {
  let lastError;
  for (let attempt = 1; attempt <= attempts; attempt += 1) {
    try {
      return await task();
    } catch (error) {
      lastError = error;
      // Don't burn attempts on errors a retry can't fix: a bad key,
      // a rejected parameter, a 404. Retry 429, 408, and 5xx only.
      const status = error.status;
      if (status && status < 500 && status !== 429 && status !== 408) throw error;
      if (attempt < attempts) {
        await new Promise((resolve) => setTimeout(resolve, 2 ** attempt * 1000));
      }
    }
  }
  throw lastError;
}

The v1 response carries a hosted link to the rendered image; the download step pulls it into your folder so the audit is a set of local files rather than a set of URLs.

The status check in withRetry matters more than the backoff does. Without it, a mistyped API key means 400 URLs each retried three times with six seconds of waiting in between, and a run that takes half an hour to tell you something it knew on the first request. A 401, a 403, a 404 or a rejected parameter will return exactly the same answer on attempt three as on attempt one, so those fail immediately; only 429, 408 and 5xx are worth a second look. Beyond that, the backoff here is deliberately minimal — enough to ride out a transient blip, not a full-strength policy. If your runs are large enough to see sustained 429 responses, the proper treatment of backoff, jitter and which errors are worth retrying at all is in rate limits and retries for high-volume usage. Do bear in mind that every attempt is a billed request, not every success.

Bounded concurrency, and the run itself

async function runPool(items, limit, worker) {
  let index = 0;
  async function next() {
    while (index < items.length) {
      const current = index;
      index += 1;
      await worker(items[current]);
    }
  }
  await Promise.all(
    Array.from({ length: Math.min(limit, items.length) }, next),
  );
}

async function main() {
  const sitemapUrl = process.argv[2];
  if (!sitemapUrl || !API_KEY) {
    console.error(
      "usage: SNAPSHOT_SITE_API_KEY=... node sitemap-capture.mjs https://example.com/sitemap.xml [--since=2026-07-01]",
    );
    process.exit(1);
  }

  const sinceArg = process.argv.find((arg) => arg.startsWith("--since="));
  const since = sinceArg ? new Date(sinceArg.slice("--since=".length)) : null;
  if (since && Number.isNaN(since.getTime())) {
    console.error(`--since is not a valid date: ${sinceArg}`);
    process.exit(1);
  }

  const entries = await collectUrls(sitemapUrl);

  // Child sitemaps overlap more often than you'd think — the same URL listed
  // in both /page-sitemap.xml and /product-sitemap.xml would otherwise be
  // captured twice, billed twice, and raced by two workers onto one file.
  const targets = [
    ...new Map(
      entries.filter((entry) => keepUrl(entry, since)).map((e) => [e.loc, e]),
    ).values(),
  ];
  console.log(`${entries.length} URLs found, ${targets.length} after filtering`);

  const done = await loadManifest(MANIFEST_FILE);
  const pending = targets.filter(
    (entry) => done.get(entry.loc)?.status !== "ok",
  );
  console.log(
    `${targets.length - pending.length} already captured, ${pending.length} to go`,
  );

  await mkdir(OUT_DIR, { recursive: true });

  let completed = 0;
  await runPool(pending, CONCURRENCY, async (entry) => {
    const at = new Date().toISOString();
    let destination = null;
    try {
      // Inside the try: outputPath() calls new URL(), which throws on the
      // relative and unescaped <loc> values that broken sitemaps emit.
      destination = outputPath(entry.loc);
      const link = await withRetry(() => capture(entry.loc));
      await download(link, destination);
      await record({ url: entry.loc, file: destination, status: "ok", link, at });
    } catch (error) {
      await record({
        url: entry.loc,
        file: destination,
        status: "failed",
        error: error.message,
        at,
      });
    }
    completed += 1;
    if (completed % 25 === 0) console.log(`${completed}/${pending.length}`);
  });

  console.log("done — check manifest.ndjson; re-run to retry failures");
}

await main();

Read the full article on Snapshot Site →