Soenneker.Playwrights.Crawler 4.0.17

Soenneker.Playwrights.Crawler

A configurable Playwright crawler for mirroring sites to disk with support for:

• HTML-only or full resource capture
• crawl limits by depth, page count, duration, and storage
• same-host restrictions with optional cross-origin asset capture
• throttling, retries, slow mode, and cooldown behavior
• optional stealth launch/context settings

Installation

dotnet add package Soenneker.Playwrights.Crawler

Register With DI

using Microsoft.Extensions.DependencyInjection;
using Soenneker.Playwrights.Crawler.Registrars;

var services = new ServiceCollection();

services.AddLogging();
services.AddPlaywrightCrawlerAsSingleton();

Use AddPlaywrightCrawlerAsScoped() if you prefer a scoped lifetime.

Basic Usage

using Soenneker.Playwrights.Crawler.Abstract;
using Soenneker.Playwrights.Crawler.Dtos;
using Soenneker.Playwrights.Crawler.Enums;

IPlaywrightCrawler crawler = serviceProvider.GetRequiredService<IPlaywrightCrawler>();

PlaywrightCrawlResult result = await crawler.Crawl(new PlaywrightCrawlOptions
{
    Url = "https://example.com",
    SaveDirectory = @"C:\temp\example",
    Mode = PlaywrightCrawlMode.Full,
    MaxDepth = 2,
    ClearSaveDirectory = true,
    SameHostOnly = true
});

Advanced Example

using Soenneker.Playwrights.Crawler.Abstract;
using Soenneker.Playwrights.Crawler.Dtos;
using Soenneker.Playwrights.Crawler.Enums;
using Soenneker.Playwrights.Extensions.Stealth.Options;

PlaywrightCrawlResult result = await crawler.Crawl(new PlaywrightCrawlOptions
{
    Url = "https://example.com",
    SaveDirectory = @"C:\temp\example",
    Mode = PlaywrightCrawlMode.Full,
    MaxDepth = 2,
    MaxPages = 50,
    MaxStorageBytes = 250_000_000,
    MaxDuration = TimeSpan.FromMinutes(10),
    SameHostOnly = true,
    IgnoreQueryStringsInDuplicateDetection = true,
    FormatHtml = true,
    IncludeCrossOriginAssets = true,
    RewriteCrossOriginAssetUrls = true,
    ClearSaveDirectory = true,
    OverwriteExistingFiles = true,
    Headless = true,
    UseStealth = true,
    NavigationTimeoutMs = 45_000,
    PostNavigationDelayMs = 0,
    ContinueOnPageError = true,
    StealthLaunchOptions = new StealthLaunchOptions
    {
        IgnoreDetectableDefaultArguments = true
    },
    StealthContextOptions = new StealthContextOptions
    {
        NormalizeDocumentHeaders = true,
        EnableCdpDomainHardening = false
    },
    Policy = new PlaywrightCrawlPolicy
    {
        GlobalMaxConcurrency = 20,
        PerDomainMaxConcurrency = 2,
        PerIpMaxConcurrency = 2,
        MinimumDelayBetweenRequestsMs = 750,
        DelayJitterMaxMs = 500,
        RequestTimeoutMs = 30_000,
        MaxRetries = 4
    }
});

Modes

HtmlOnly

Saves only rendered HTML documents discovered during the crawl.

Full

Saves:

• rendered HTML documents
• same-origin network resources observed while pages load
• optional cross-origin assets under _external when IncludeCrossOriginAssets = true
• optional rewriting of cross-origin asset URLs in saved HTML when RewriteCrossOriginAssetUrls = true

Key Options

Result

Crawl() returns PlaywrightCrawlResult, which includes:

• crawl timing (StartedAtUtc, CompletedAtUtc, Duration)
• page counts (PagesDiscovered, PagesVisited)
• file counts (HtmlFilesSaved, AssetFilesSaved)
• total bytes written (BytesWritten)
• stop reasons (StorageLimitReached, DurationLimitReached, PageLimitReached)
• per-file details in Files
• page-level failures in Errors

Output Layout

Saved files preserve URL structure so the output can be served by a simple static web server.

Examples:

• https://example.com/ → index.html
• https://example.com/docs/getting-started → docs/getting-started/index.html
• https://cdn.example.com/app.css → _external/cdn.example.com/app.css when cross-origin asset capture is enabled
• a saved page can reference that asset as ../../_external/cdn.example.com/app.css when URL rewriting is enabled

Behavior Notes

• Playwright browser installation is ensured automatically before the crawl starts.
• Duplicate detection ignores query strings by default.
• HTML formatting is opt-in and uses Soenneker.Html.Formatter when FormatHtml = true.
• Challenge and captcha-like pages contribute to the crawler's blocking and slow-mode signals.
• Cross-origin URL rewriting only applies to captured cross-origin assets that are actually available on disk.
• Full mode captures resources observed during page loads, but the rewrite pass is limited to captured cross-origin asset URLs rather than a full offline-mirroring transform.
• Some response types are intentionally skipped, such as empty bodies and certain framework/internal fetch endpoints.