Denomica.Sitemap 1.0.4

Denomica.Sitemap

A library that facilitates discovering and enumerating XML sitemaps on a site.

Main Features

The main features that this library provides are described in the following sections.

Discover URLs in Sitemaps

The SitemapCrawler services attempts to discover all sitemaps defined on a site, and then enumerate all URLs defined in those sitemaps. It supports both standard sitemap files and sitemap index files.

If the URL provided is a sitemap XML file, only that sitemap file is processed. If that sitemap XML file contains references to other sitemap files, those are also processed recursively.

Getting Started

In addition to the Denomica.Sitemap package, you will need to install the Microsoft.Extensions.DependencyInjection Nuget package.

The following code sample demonstrates how to use the SitemapCrawler service to crawl a sitemap and retrieve all URLs defined in it.

using Denomica.Sitemap.Services;
using Microsoft.Extensions.DependencyInjection;

var provider = new ServiceCollection()
	.AddDenomicaSitemap()
	.Services
	.BuildServiceProvider();

var crawler = provider.GetRequiredService<SitemapCrawler>();
await foreach(var pageUrl in crawler.CrawlAsync(new Uri("https://yoursite.com")))
{
	Console.WriteLine(pageUrl.Location);
}

Version Highlights

v1.0.4

• Added configuration support for supplying a custom HttpClient instance from the consuming application for sitemap crawling.
• Added IHttpRequestFactory to centralize HTTP request creation and shared request configuration.
• Updated both SitemapCrawler and RobotsTxtParser to use the same configured request creation flow.
• Applied sample-aligned request defaults for sitemap and robots requests (HTTP/1.1, browser user-agent, and browser-like Accept and Connection headers).
• Replaced reflection-based request profile behavior with a straightforward reflection-free implementation.

v1.0.3

• Added explicit 301 and 302 redirect following as a fallback for sitemap requests in cases where automatic redirect following does not always work reliably.
• Preserved the final resolved target URL after redirect fallback so sitemap discovery continues from the redirected sitemap location.

v1.0.2

• Unified sitemap discovery in SitemapCrawler so CanCrawlAsync and CrawlAsync use the same logic to resolve sitemap documents.
• Improved default sitemap discovery by probing standard sitemap locations with the same GET-based logic used by the crawler, which fixes sites where GET succeeds or redirects but HEAD does not.
• Updated discovery fallback behavior so default sitemap locations are still checked when robots.txt lists sitemap URLs that do not resolve to a valid sitemap.

v1.0.1

• Added SitemapCrawler.CanCrawlAsync method that checks if a given URL can be crawled by the SitemapCrawler service. This method can be used to check if a URL is a sitemap XML file before attempting to crawl it.

v1.0.0

• The first stable version.
• Added support for crawling sitemap XML files from the local file system. The CrawlAsync method now accepts a Uri parameter that can be a local file path or a URL.

v1.0.0-beta.4

• Moved the UrlSetUrl class to the Denomica.Sitemap.Models namespace.
• Added support for images defined in sitemap files. The UrlsetUrl class now includes an Image property that contains a list of images defined in the sitemap. The Image property is null if no images are defined in the sitemap.

v1.0.0-beta.3

• Changed the user agent string that the crawler uses.

v1.0.0-beta.2

• Added support for crawling sitemaps that use an XML namespace other than the default http://www.sitemaps.org/schemas/sitemap/0.9. Many sites seem to incorrectly use a namespace with https prefix.

v1.0.0-beta.1

• Added overloaded CrawlAsync method that accepts a string parameter to specify the base URL of the site to crawl.

v1.0.0-alpha.2

• Modified the return type of the CrawlAsync method to return an async enumerable of UrlsetUrl objects.

v1.0.0-alpha.1

• Initial release of the Denomica.Sitemap package.
• Provides a SitemapCrawler service that crawls sitemap XML files and returns all URLs defined in them.