XML Sitemap Generator

This XML sitemap generator crawls one site and builds a sitemap.xml you can download. It stays on your host, stops at 200 URLs, and skips anything robots.txt disallows.

Crawls up to 200 URLs on that host, skips addresses robots.txt disallows, and does not request private or local addresses.

How to generate an XML sitemap

  1. Enter your homepage. Use the public https address of the site. The crawl stays on that host, including a www redirect from the address you typed.
  2. Wait for the crawl. We request up to 200 URLs, skip anything robots.txt disallows for SEOchatBot, and refuse private, local, and non-public addresses.
  3. Download sitemap.xml. Upload the file somewhere public on your site, then submit that URL in Google Search Console and Bing Webmaster Tools.

What the file includes

Search engines use an XML sitemap as a list of URLs, not as a ranking signal. This sitemap generator adds a URL when the response is HTML, the status is 200, and the page does not say noindex. A redirect is listed as the URL it lands on, not the old address. Query parameters used for click tracking, such as utm tags, are dropped so the same page is not listed twice.

The crawl does not run JavaScript, sign in, or submit the file to Google for you. It does not invent last-modified dates. If the server does not send Last-Modified, that line is left out.

After you download it

Upload sitemap.xml and confirm it opens in a browser. In Google Search Console, use Sitemaps and submit the full URL. Bing Webmaster Tools has the same kind of submission. Then add the URL to robots.txt so other crawlers can find it.

Add that line with the robots.txt generator. Check pages that returned errors with the broken link checker. See the HTML a crawler reads on one URL with the spider simulator.

Questions

What does this XML sitemap generator make?

A sitemap.xml file that lists HTML pages we fetched with status 200 and that were not marked noindex. Each URL is one we actually requested. Dates come from the Last-Modified header when the server sent one.

Is a sitemap generator the same thing as getting indexed?

No. A sitemap helps search engines find URLs. Google and Bing still decide what to index. Uploading the file does not add rankings, and it does not remove pages that are already indexed.

How many URLs can one crawl include?

200 requests per run, including robots.txt and redirects. The crawl stops earlier if it runs out of time. Pages linked only through JavaScript are not seen, because we read the HTML the server sends.

Do you respect robots.txt?

Yes. We fetch robots.txt first. Disallowed URLs are not requested. If robots.txt returns a server error, the crawl stops instead of guessing. A missing robots.txt means there are no disallow rules to apply. Crawl-delay is not applied. Google does not use it either, and this crawl is already capped.

Will this request private or internal addresses?

No. Localhost, .local, .internal, and private, loopback, and link-local IP addresses are refused before any request. We only use http on port 80 and https on port 443, and we do not follow a start URL that redirects to a different site.

Where should the file go?

Anywhere the public can fetch it, often https://yourdomain.com/sitemap.xml. Add a Sitemap line in robots.txt if you want crawlers to see that address. The robots.txt generator on this site can add that line.

More free tools

See all tools