WebTools.NET 1.3.0

WebTools.NET

Web tools for .NET applications: web search, browser-based content fetching, and navigation.

WebTools.NET gives applications and automation scripts reliable access to the real web — searching, fetching rendered page content, checking URL reachability, and caller-controlled link navigation — through a small set of async, interface-based APIs.

Why WebTools.NET?

HttpClient alone is not enough for web-facing agents:

• JavaScript-rendered pages come back empty — the content you need is built by scripts after load.
• Bot protection blocks plain HTTP clients and even default headless browsers.
• Search engines have no stable free API, so scraping them by hand is brittle.
• Redirect chains, timeouts, and error handling get reimplemented in every project.

WebTools.NET solves this by driving real browsers (Playwright Chromium, or the stealth-patched CloakBrowser) behind small abstractions, while keeping plain-HTTP fast paths for the cases where a browser is overkill. Every operation returns a result object (SearchResult, WebContent, UrlCheckResult) instead of throwing, so caller code can reason about failures and retry or fall back on its own terms.

Feature Overview

Installation

dotnet add package WebTools.NET

Requires .NET 10.0 or later. Browser-based features drive Chromium through Microsoft Playwright; install the browser binaries once per machine:

pwsh bin/Debug/net10.0/playwright.ps1 install chromium

DuckDuckGoSearchProvider and IWebAccessService use plain HTTP and need no browser.

Quick Start

Register the services you need — one call per concern:

using Microsoft.Extensions.DependencyInjection;
using WebTools.NET.Abstractions;

var services = new ServiceCollection();

services.AddWebToolsCore();   // IWebAccessService
services.AddBrowserServices(); // IWebContentFetcher, IWebSearchProvider,
                               // legacy IBrowserInteraction, and
                               // IBrowserSessionFactory
                               // (Playwright by default)

await using var provider = services.BuildServiceProvider();

Check URL reachability (no browser)

var webAccess = provider.GetRequiredService<IWebAccessService>();

var check = await webAccess.CheckReachabilityAsync("https://test.example.com");
Console.WriteLine(check.Reachable
    ? $"reachable - HTTP {check.HttpStatus}, {check.RedirectCount} redirect(s)"
    : $"unreachable - {check.ErrorMessage}");

Fetch page content through a real browser

var fetcher = provider.GetRequiredService<IWebContentFetcher>();

var content = await fetcher.FetchAsync("https://test.example.com");
if (content.Success)
{
    Console.WriteLine(content.FinalUrl);  // URL after redirects
    Console.WriteLine(content.Content);   // plain-text rendered page content
}

Search the web

WebSearchService wraps any IWebSearchProvider and automatically retries with fallback queries when the first attempt returns nothing:

using WebTools.NET;
using WebTools.NET.Search;

using var ddg = new DuckDuckGoSearchProvider();   // plain HTTP, no browser
var search = new WebSearchService(ddg);

var result = await search.SearchAsync("dotnet web scraping", maxResults: 5);
if (result.Success)
{
    foreach (var item in result.Results)
    {
        Console.WriteLine($"{item.Title} -> {item.Url}");
    }
}

For sites behind bot protection, resolve IWebSearchProvider from DI instead — the browser-based providers type the query into a real search page and scrape the rendered results.

Drive a page interactively

var browser = provider.GetRequiredService<IBrowserInteraction>();

await browser.NavigateAsync("https://test.example.com/search");
await browser.FillAsync("input[name=q]", "WebTools.NET");
await browser.ClickAsync("button[type=submit]");

var url  = await browser.GetCurrentUrlAsync();
var text = await browser.GetContentAsync();   // readable text of the result page

Verify links on a page

WebNavigationService extracts same-host links from a page and verifies each one in the browser:

var browser = provider.GetRequiredService<IBrowserInteraction>();
var navService = new WebNavigationService(browser);

var workingLinks = await navService.NavigateAsync("https://test.example.com", maxLinks: 20);
foreach (var link in workingLinks)
{
    Console.WriteLine(link);
}

Stateful browser-session interaction

BrowserSession receives an explicitly created IBrowserSession session and never creates or disposes it. Create one session per independent workflow; the session returns a BrowserSnapshot after each operation, reports HTTP failures in Error, supports cancellation, and can load/save Playwright storage state when StorageStatePath is configured.

using WebTools.NET;
using WebTools.NET.Browsing;
using WebTools.NET.Models;

var options = new BrowserSessionOptions
{
    IncludeScreenshot = false,
    StorageStatePath = "./cookies.json"
};
var sessionFactory = new BrowserSessionFactory(
    storageStatePath: options.StorageStatePath);
await using var session = sessionFactory.Create();
await using var browserSession = new BrowserSession(session, options);

var snapshot = await browserSession.StartAsync("https://test.example.com/login");

// Fill login form in one action
snapshot = await browserSession.ExecuteAsync(new BrowserOperation(
    EBrowserOperationType.FillForm,
    Fields: [
        new FormFieldValue(2, "user@test.example.com"),
        new FormFieldValue(3, "test-password")
    ]));

// Click submit
snapshot = await browserSession.ExecuteAsync(new BrowserOperation(
    EBrowserOperationType.Click, ElementIndex: 4));

// snapshot.Url, snapshot.Content, snapshot.Elements are now the dashboard

Operations: Navigate, Click, Fill, FillForm, Select, Submit, ScrollDown, ScrollUp, WaitFor, Back, Snapshot. See the browser-session docs for the full vocabulary.

Detect the caller's region

using WebTools.NET.Geo;

using var geo = new GeoRegionService();
var region = await geo.DetectRegionAsync();   // e.g. "DE" - Geo-IP with locale fallback, cached

Backward Compatibility and Migration

The session-oriented names are the preferred API, but the previous public names remain available during the compatibility period and are marked [Obsolete]. Existing applications can continue compiling while migrating incrementally:

The compatibility types forward to the current session/service implementations; new code should use the preferred names and follow the compiler migration messages. BrowserAgent converts legacy actions and snapshots to the current operation/session models. The old AddBrowserServices overloads accepting BrowserAgentOptions, and the old DI contracts, coexist with the BrowserSessionOptions and session contracts. They resolve the same current browser implementations and do not register a shared BrowserSession.

The parameterless WebSearchAgent and WebNavigationAgent retain their historical behavior by creating and owning their default browser dependencies. This ownership behavior is limited to those obsolete wrappers; the preferred WebSearchService and WebNavigationService require caller-supplied dependencies and do not own them. Injected legacy wrappers also leave supplied dependencies caller-owned.

Choosing a Browser Engine

Browser services are engine-agnostic — the same interfaces work with either backend, selected with one enum:

services.AddBrowserServices(EBrowserEngine.Playwright);    // default
services.AddBrowserServices(EBrowserEngine.CloakBrowser);  // stealth-patched Chromium,
                                                           // resists bot detection
services.AddBrowserServices(EBrowserEngine.Playwright, headless: false);

Rule of thumb: use Playwright for normal automation, CloakBrowser when target sites detect and block headless browsers.

Design Highlights

• Interface-based — all capabilities sit behind abstractions in WebTools.NET.Abstractions, easy to mock in tests.
• Engine-agnostic — swap Playwright for CloakBrowser (or your own implementation) without touching calling code.
• Result objects, not exceptions — every operation reports success, error message, and payload in one record.
• DI-first, but constructor-friendly — create a browser session from the factory (or directly), pass it to BrowserSession, and manage both lifetimes explicitly.
• Fully async with CancellationToken support throughout.

Demo Project

The repository contains WebTools.NET.Demo, a console app that exercises every feature end to end: reachability checks, geo detection, HTTP and browser search, content fetching, the stealth engine, interactive browsing, and link navigation — with a per-section summary table at the end.

Documentation

The developer manual is published at alexnek.github.io/WebTools.NET.

Changelog

See CHANGELOG.md for release history.

License

MIT – see LICENSE.txt for details.