Soenneker.Playwrights.Crawler 4.0.152

Soenneker.Playwrights.Crawler

A configurable Playwright crawler for mirroring sites to disk with support for:

• HTML-only or full resource capture
• crawl limits by depth, page count, duration, and storage
• same-host restrictions with optional cross-origin asset capture
• DOM attribute resource discovery for lazy widgets and deferred assets
• throttling, retries, slow mode, and cooldown behavior
• optional stealth launch/context settings

Related Repos

You might also be interested in:

• soenneker.playwrights.installation for ensuring Playwright browsers are installed before runtime.
• soenneker.playwrights.extensions.stealth for stealth-oriented Chromium launch and browser-context extensions.

Installation

dotnet add package Soenneker.Playwrights.Crawler

Register With DI

using Microsoft.Extensions.DependencyInjection;
using Soenneker.Playwrights.Crawler.Registrars;

var services = new ServiceCollection();

services.AddLogging();
services.AddPlaywrightCrawlerAsSingleton();

Use AddPlaywrightCrawlerAsScoped() if you prefer a scoped lifetime.

Basic Usage

using Soenneker.Playwrights.Crawler.Abstract;
using Soenneker.Playwrights.Crawler.Dtos;
using Soenneker.Playwrights.Crawler.Enums;

IPlaywrightCrawler crawler = serviceProvider.GetRequiredService<IPlaywrightCrawler>();

PlaywrightCrawlResult result = await crawler.Crawl(new PlaywrightCrawlOptions
{
    Url = "https://example.com",
    SaveDirectory = @"C:\temp\example",
    Mode = PlaywrightCrawlMode.Full,
    MaxDepth = 2,
    ClearSaveDirectory = true,
    SameHostOnly = true
});

Advanced Example

using Soenneker.Playwrights.Crawler.Abstract;
using Soenneker.Playwrights.Crawler.Dtos;
using Soenneker.Playwrights.Crawler.Enums;
using Soenneker.Playwrights.Extensions.Stealth.Options;

PlaywrightCrawlResult result = await crawler.Crawl(new PlaywrightCrawlOptions
{
    Url = "https://example.com",
    SaveDirectory = @"C:\temp\example",
    Mode = PlaywrightCrawlMode.Full,
    MaxDepth = 2,
    MaxPages = 50,
    MaxStorageBytes = 250_000_000,
    MaxDuration = TimeSpan.FromMinutes(10),
    SameHostOnly = true,
    IgnoreQueryStringsInDuplicateDetection = true,
    FormatHtml = true,
    IncludeCrossOriginAssets = true,
    RewriteCrossOriginAssetUrls = true,
    ClearSaveDirectory = true,
    OverwriteExistingFiles = true,
    Headless = true,
    UseStealth = true,
    ThrottleMode = PlaywrightCrawlThrottleMode.Automatic,
    NavigationTimeoutMs = 45_000,
    WaitUntil = WaitUntilState.DOMContentLoaded,
    PostNavigationDelayMs = 0,
    ContinueOnPageError = true,
    StealthLaunchOptions = new StealthLaunchOptions
    {
        IgnoreDetectableDefaultArguments = true
    },
    StealthContextOptions = new StealthContextOptions
    {
        NormalizeDocumentHeaders = true,
        EnableCdpDomainHardening = false
    },
    Policy = new PlaywrightCrawlPolicy
    {
        GlobalMaxConcurrency = 20,
        PerDomainMaxConcurrency = 2,
        PerIpMaxConcurrency = 2,
        MinimumDelayBetweenRequestsMs = 750,
        DelayJitterMaxMs = 500,
        RequestTimeoutMs = 30_000,
        MaxRetries = 4
    }
});

In-Memory Multi-Page Capture

Capture an explicit set of rendered pages through one shared browser and browser context without writing to disk or discovering links:

PlaywrightCrawlResult result = await crawler.Crawl(new PlaywrightCrawlOptions
{
    StartingUrls =
    [
        "https://example.com/",
        "https://example.com/about",
        "https://example.com/pricing"
    ],
    SaveToDisk = false,
    CaptureRenderedHtml = true,
    DiscoverLinks = false,
    Mode = PlaywrightCrawlMode.HtmlOnly,
    WaitUntil = WaitUntilState.DOMContentLoaded,
    ThrottleMode = PlaywrightCrawlThrottleMode.Disabled,
    ExtraHttpHeaders = new Dictionary<string, string>
    {
        ["X-Crawler-Capture"] = "1"
    },
    ReadinessExpression =
        "() => document.readyState === 'complete'" +
        " && !!document.querySelector('main')",
    PageCompletedHandler = (page, pageResult, cancellationToken) =>
    {
        Console.WriteLine($"Completed {pageResult.FinalUrl} with status {pageResult.StatusCode}");
        return ValueTask.CompletedTask;
    }
});

foreach (PlaywrightCrawlPageResult page in result.Pages)
{
    Console.WriteLine($"{page.FinalUrl}: {page.Title} ({page.Html?.Length ?? 0} characters)");
}

PageReadinessHandler can be used instead of, or after, ReadinessExpression when readiness requires application-specific .NET logic. PageCompletedHandler runs after each successfully processed page and before its live Playwright page is closed.

Modes

HtmlOnly

Saves only rendered HTML documents discovered during the crawl.

Full

Saves:

• rendered HTML documents
• same-origin network resources observed while pages load
• resource URLs discovered in DOM attributes such as data-src and data-css-url
• optional cross-origin assets under _external when IncludeCrossOriginAssets = true
• optional rewriting of cross-origin asset URLs in saved HTML when RewriteCrossOriginAssetUrls = true
• optional lazy-load scrolling to capture below-the-fold media
• optional rewriting of same-origin absolute URLs in saved HTML and CSS to root-relative paths when RewriteSameOriginAbsoluteUrls = true

Key Options

Result

Crawl() returns PlaywrightCrawlResult, which includes:

• crawl timing (StartedAtUtc, CompletedAtUtc, Duration)
• page counts (PagesDiscovered, PagesVisited)
• file counts (HtmlFilesSaved, AssetFilesSaved)
• total bytes written (BytesWritten)
• stop reasons (StorageLimitReached, DurationLimitReached, PageLimitReached)
• per-file details in Files
• rendered page details and optional in-memory HTML in Pages
• page-level failures in Errors

Output Layout

Saved files preserve URL structure so the output can be served by a simple static web server.

Examples:

• https://example.com/ → index.html
• https://example.com/docs/getting-started → docs/getting-started/index.html
• https://example.com/script.js → /script.js inside saved HTML when same-origin URL rewriting is enabled
• https://cdn.example.com/app.css → _external/cdn.example.com/app.css when cross-origin asset capture is enabled
• a saved page can reference that asset as ../../_external/cdn.example.com/app.css when URL rewriting is enabled

Behavior Notes

• Playwright browser installation is ensured automatically before the crawl starts.
• Multiple starting URLs use one Playwright instance, browser, and browser context.
• PageCompletedHandler callbacks can run concurrently when the crawl uses multiple workers.
• Adaptive throttling uses main-document HTTP timing rather than total browser readiness, requires multiple samples before latency can trigger slow mode, and fails a page instead of waiting longer than Policy.MaximumThrottleWaitMs.
• Setting DiscoverLinks = false captures only explicitly supplied starting URLs.
• Setting SaveToDisk = false avoids output-directory creation and returns rendered HTML through Pages.
• Duplicate detection ignores query strings by default.
• HTML formatting is opt-in and uses Soenneker.Html.Formatter when FormatHtml = true.
• Challenge and captcha-like pages contribute to the crawler's blocking and slow-mode signals.
• Setting ThrottleMode = PlaywrightCrawlThrottleMode.Disabled keeps configured concurrency limits and retries, but skips the crawler's automatic pacing and adaptive slowdown behavior.
• Cross-origin URL rewriting only applies to captured cross-origin assets that are actually available on disk.
• Full mode captures resources observed during page loads, but the rewrite pass is limited to captured cross-origin asset URLs rather than a full offline-mirroring transform.
• Some response types are intentionally skipped, such as empty bodies and certain framework/internal fetch endpoints.