Meziantou.Framework.WebPlatform 1.0.1

Meziantou.Framework.WebPlatform

A spec-compliant, high-performance implementation of the web platform for .NET. It parses HTML, XML, SVG, MathML and CSS exactly as a browser does, and exposes the browser DOM API (the WebIDL surface of the WHATWG and W3C specifications) with .NET naming: queries (QuerySelector, XPath, live collections), mutations, events, the CSSOM and the computed styles, forms, custom elements, accessibility, and more. A managed layout engine computes the geometry exposed by the web APIs (CSSOM View, the observers...), and the documents are painted into a display list that a rendering backend (a managed rasterizer, in a separate package) rasterizes for the screenshots and the canvases. Documents can be loaded in a browsing context that fetches their resources over the network and runs their scripts with a pluggable JavaScript engine (Meziantou.Framework.JavaScriptEngine is provided).

Every public member mapping a WebIDL member is annotated with [IdlMember(name, specUrl)]. The WebIDL names follow the .NET naming guidelines: acronyms of 3 letters or more are PascalCased and acronyms of 2 letters stay upper case (HTMLElement.innerHTML → HtmlElement.InnerHtml, toJSON → ToJson, CDATASection → CDataSection, UIEvent and HTMLBRElement → HtmlBRElement). MathML and XPath keep their casing.

Examples

Parse HTML, query the nodes and the styles

HtmlDocumentParser is in the Meziantou.Framework.WebPlatform namespace, and the DOM types (BrowsingContext, Document, HtmlAnchorElement...) are in Meziantou.Framework.WebPlatform.Dom.

var document = HtmlDocumentParser.ParseDocument("""
    <style>.important { color: red; font-weight: bold }</style>
    <ul>
      <li><a href="https://example.com/a" class="important">A</a>
      <li><a href="/b" style="color: blue">B</a>
    </ul>
    """);

// CSS selectors, with typed overloads and LINQ
var links = document.QuerySelectorAll<HtmlAnchorElement>("li > a");
var external = links.Where(a => a.Href.StartsWith("https:", StringComparison.Ordinal)).Select(a => a.TextContent);
var first = document.GetElementsByTagName("li")[0];
Assert.Equal(["A"], external);
Assert.Same(first, links[0].ParentElement);

// XPath
var result = document.Evaluate("//a[@class='important']", document, type: XPathResult.FirstOrderedNodeType);
var important = (Element?)result.SingleNodeValue;
Assert.Equal("A", important?.TextContent);

// Inline style (style attribute)
var color = links[1].Style.GetPropertyValue("color");
Assert.Equal("blue", color);

The computed styles (cascade of the style sheets, inheritance, media queries) need a browsing context, which provides the window and the viewport:

var html = """
    <style>.important { color: red; font-weight: bold }</style>
    <p class="important">Hello</p>
    """;

using var context = new BrowsingContext(new BrowsingContextOptions
{
    Viewport = new ViewportOptions { Width = 1280, Height = 720 }, // optional, 800x600 by default
});

var document = context.Navigate("https://example.com/", _ => html);
var style = context.Window.GetComputedStyle(document.QuerySelector(".important")!);
Assert.Equal("rgb(255, 0, 0)", style.Color);
Assert.Equal("700", style.GetPropertyValue("font-weight"));

See docs/dom-helpers.md, docs/selectors.md, docs/computed-style.md and docs/css-typed-om.md.

Update the DOM

The formatters are in the Meziantou.Framework.WebPlatform.Formatting namespace.

var document = HtmlDocumentParser.ParseDocument("""
    <ul>
      <li><a href="/a">A</a>
      <li><a href="/b">B</a>
    </ul>
    """);
var list = document.QuerySelector("ul")!;

// Create elements
var item = document.CreateElement<HtmlLIElement>();
var link = document.CreateElement<HtmlAnchorElement>();
link.Href = "/c";
link.TextContent = "C";
link.ClassList.Add("important");
link.SetAttribute("data-id", "3");
item.Append(link);
list.Append(item);

// Parse markup into an element, edit the styles, remove nodes
list.InsertAdjacentHtml("beforeend", "<li><a href=/d>D</a>");
link.Style.SetProperty("text-decoration", "underline");
document.QuerySelector("a[href='/b']")?.Remove();

// Serialize the document
var html = document.GetHtml();
var indented = document.ToHtml(new HtmlPrettyFormatter());
var minified = document.ToHtml(new HtmlMinifyFormatter());
Assert.StartsWith("<html><head></head><body><ul>", html, StringComparison.Ordinal);
Assert.Equal("""
    <html>
      <head></head>
      <body>
        <ul>
          <li>
            <a href="/a">A</a>
          </li>
          <li></li>
          <li>
            <a href="/c" class="important" data-id="3" style="text-decoration: underline;">C</a>
          </li>
          <li>
            <a href="/d">D</a>
          </li>
        </ul>
      </body>
    </html>
    """, indented);
Assert.Equal("""<ul><li><a href=/a>A</a><li><li><a href=/c class=important data-id=3 style="text-decoration: underline;">C</a><li><a href=/d>D</a></ul>""", minified);

Element.SetHtml() and Document.ParseHtml() insert untrusted markup with the HTML Sanitizer API. See docs/html-formatting.md and docs/css-formatting.md for the serialization options.

Allow network requests

Nothing is fetched unless a DocumentLoader is configured. With a loader, the browsing context loads the documents of the navigations and the iframe elements, the style sheets (<link rel=stylesheet>, @import), the images and, when a script engine is configured, the scripts:

using var context = new BrowsingContext(new BrowsingContextOptions
{
    DocumentLoader = DocumentLoader.Create(),  // HTTP GET requests with a shared HttpClient (or pass your own)
    CookieJar = new CookieJar(),               // optional: the cookies (RFC 6265bis, document.cookie, cookieStore)
});

context.Navigate("https://example.com/");
await context.EventLoop.RunUntilIdleAsync(); // runs the tasks until the document and its resources are loaded

var document = context.Document;

DocumentLoader.Create() requests any URL. A custom loader can filter the requests (to prevent server-side request forgery with untrusted documents), serve them from a cache or from the disk:

internal sealed class SameHostLoader(string host) : DocumentLoader
{
    private static readonly DocumentLoader Http = DocumentLoader.Create();

    public override ValueTask<DocumentResource?> LoadAsync(DocumentRequest request, CancellationToken cancellationToken)
    {
        if (new Uri(request.Url).Host != host)
            return ValueTask.FromResult<DocumentResource?>(null); // network error

        return Http.LoadAsync(request, cancellationToken);
    }
}

See docs/browsing-context.md.

Execute scripts

Scripts only run when a JavaScript engine is plugged into the browsing context. The Meziantou.Framework.WebPlatform.JavaScript package provides one, based on Meziantou.Framework.JavaScriptEngine, in the Meziantou.Framework.WebPlatform.JavaScript namespace:

using var engine = new JavaScriptScriptEngine(new JavaScriptScriptEngineOptions
{
    Timeout = TimeSpan.FromSeconds(5), // per script or callback
});

using var context = new BrowsingContext(new BrowsingContextOptions
{
    ScriptEngine = engine,
    DocumentLoader = DocumentLoader.Create(), // optional: external scripts, modules, fetch, XMLHttpRequest, WebSocket, EventSource...
    UnhandledScriptError = (window, error) => Console.Error.WriteLine(error),
    ConsoleMessage = (window, message) => Console.WriteLine(message.Text), // console.log(), console.count()...
});

var document = await context.NavigateAsync("https://example.com/", _ => """
    <title>Example</title>
    <ul id=list></ul>
    <script>
      setTimeout(() => document.getElementById('list').innerHTML = '<li>a<li>b', 100);
    </script>
    """);

// Run the tasks and the timers until there is nothing left
do { await context.EventLoop.RunUntilIdleAsync(); } while (await context.EventLoop.WaitForNextTimerAsync());

Assert.Equal("<li>a</li><li>b</li>", document.GetElementById("list")!.InnerHtml);

// Evaluate code in the realm of a window
var title = engine.Evaluate(context.Window, "document.title");
Assert.Equal("Example", title.AsString());

The scripts run in the process: read the threat model in docs/scripting.md before running untrusted documents.

Take screenshots

The documents are painted by the library into a backend-agnostic display list; the Meziantou.Framework.WebPlatform.Imaging package draws them, and decodes and encodes the images, with Meziantou.Framework.Imaging, a fully managed library (no native dependency: the same pixels on every platform). ImagingRenderingBackend and the screenshot methods are in the Meziantou.Framework.WebPlatform.Imaging namespace:

using var context = new BrowsingContext(new BrowsingContextOptions
{
    Viewport = new ViewportOptions { Width = 1280, Height = 720, DevicePixelRatio = 2 },
    DocumentLoader = DocumentLoader.Create(),
    RenderingBackend = new ImagingRenderingBackend(),
});

context.Navigate("https://example.com/");
await context.EventLoop.RunUntilIdleAsync();

await File.WriteAllBytesAsync("page.png", context.TakeScreenshotAsPng());            // the viewport: 2560x1440 pixels
using var title = context.Document.QuerySelector("h1")!.TakeScreenshot();           // an element, as an Image<Rgba32>

// Print the document: the print media type, the @page rules, one image per page
var pages = context.RenderPagesAsPng(new PrintOptions { PageWidth = 794, PageHeight = 1123 }); // A4 at 96 dpi

See docs/rendering.md for what is painted, and Printing for the pages.

Limitations

• Layout. The layout engine is internal (docs/layout.md): it runs on demand, when a web API reads the geometry, and it is exposed only through the web APIs: CSSOM View (getBoundingClientRect(), getClientRects(), the offset*, scroll* and client* members, elementFromPoint() and the hit testing, caretPositionFromPoint(), getBoxQuads(), the MouseEvent coordinates, the scrolling and its events), the used values of getComputedStyle() (see docs/computed-style.md), the SVG geometry, ResizeObserver, IntersectionObserver and the lazy loading, the layout shifts, the size and scroll-state container queries, the viewport units and window.visualViewport, Selection.modify() and the caret movement keys. It implements the block and inline layouts (line breaking, bidirectional text, the CSS Text features), the floats and shape-outside, the replaced elements and the form controls, the intrinsic sizes, the containment and content-visibility, the positioned layout and the anchor positioning, the overflow (and text-overflow, line-clamp), the scrolling, the scroll snapping and the scroll anchoring, the scroll markers and the scroll buttons (::scroll-marker-group, ::scroll-marker, ::scroll-button(), scroll-target-group), the flex (and -webkit-box), grid (with the subgrids), grid lanes (masonry) and table layouts, the ruby layout, the box alignment, the gap decorations (the column and row rules of the grid, flex and multi-column containers), the writing modes, the multi-column layout and the block fragmentation in its columns (with the floats pushed to the next column and the repeated table headers and footers), the paged media (the pagination of the printed documents: @page, the page breaks, the named pages, the page-margin boxes), the MathML Core layout and the SVG layout. The wheel, a panning touch, the keys and the scrollbars scroll (BrowsingContext.Input). Not implemented: the selection with the pointer, the layout of the pages whose inline size differs from the first page, the repetition of a hyphen at the start of the next line, the spelling changes at a hyphenation break of the languages other than Dutch, Hungarian and Pinyin and the hyphenation dictionaries under copyleft licenses (cs, hu, hy, lv), the scrolling of a ::scroll-marker-group that is a scroll container, pinch zoom, and, in the shaping of the complex scripts (the shaper of Meziantou.Framework.Fonts), the Myanmar Zawgyi encoding and the Arabic fallback shaping of the Windows-1256 fonts.
• No rasterizer in the core library. The documents are painted into a backend-agnostic display list (the painting order of CSS, the stacking contexts, the clips, the transforms and effects, the backgrounds, the borders, the outlines, the texts with the glyphs of the managed text shaping, the images including the SVG images, the SVG content): the pixels are produced by a rendering backend plugged with BrowsingContextOptions.RenderingBackend: ImagingRenderingBackend of the Meziantou.Framework.WebPlatform.Imaging package (the managed drawing API of Meziantou.Framework.Imaging), which also takes the screenshots (see docs/rendering.md) and draws the 2D canvases (CanvasRenderingContext2D, OffscreenCanvas, also in the workers) and the paint() images of the paint worklets (CSS.paintWorklet, see docs/rendering.md). Without a backend, nothing is rasterized and canvas.getContext('2d') returns null. Not painted yet: the color glyphs other than the layers of COLR version 0 with the palettes of font-palette (the paint graphs of COLR version 1, the bitmap and SVG glyphs), the wrapping of the SVG texts in the shapes of shape-inside. The canvas layers (beginLayer()) and the float16 canvases are not implemented. As in Blink, the painted boxes and the baselines are snapped to the device pixels and the metrics of the fonts are rounded to whole pixels (the layout keeps its fractional geometry).
• Injected user input. There is no window: the host injects the pointer and wheel input of the user with BrowsingContext.Input (mouse, pen and touch pointers, wheel; see docs/browsing-context.md), which fires the trusted pointer, mouse, touch and click events and runs the default actions (focus, activation, light dismiss, drag and drop, scrolling, the picker of the select elements: the customizable select, appearance: base-select, is rendered) and the classic scrollbars (a press on the track, a drag of the thumb). The keyboard events are dispatched by the host. There is no pinch zoom.
• Opt-in network and scripting. Without a DocumentLoader, nothing is fetched; without a script engine, the scripts are never fetched nor run, and the event handler attributes have no effect. A document parsed with HtmlDocumentParser has no browsing context: no window, no computed style, no event loop (the mutation observer callbacks are not invoked).
• Images are decoded by the rendering backend, not by the core library: without a backend, only the natural dimensions of the images are read from their header (decode() resolves once the image is available, createImageBitmap() rejects the images and the blobs). The Imaging backend decodes PNG (including the animated PNG), JPEG (including CMYK), GIF, WebP, AVIF, BMP and ICO images, with their EXIF orientation, their EXIF resolution (the density-corrected size) and their color profile.
• Media (audio, video) are not decoded by the library: the processing model of the media elements (resource selection, events, play(), seeking...) runs on top of a media backend provided by the application (BrowsingContextOptions.MediaBackend, see docs/browsing-context.md); without it, the media elements have no media resource. The Media Source Extensions are implemented on top of the segment parsers of the backend (the coded frames are not decoded); Encrypted Media and the controls of the media elements are not implemented.
• No downloads, and the third-party cookies are not blocked.
• Content Security Policy covers the fetch directives, the inline checks (scripts, styles, event handlers, style attributes, javascript: URLs), 'unsafe-eval'/'wasm-unsafe-eval' (eval() and the Function constructors), the navigation and document directives (frame-src, form-action, frame-ancestors, base-uri, object-src, worker-src), upgrade-insecure-requests, the inheritance of the policies and the Trusted Types directives, with their violation reports (securitypolicyviolation, report-uri, and report-to with the Reporting API): no CSP Embedded Enforcement (the csp attribute of the iframe element). The Reporting API only generates the CSP, COEP and test reports (no deprecation, intervention, COOP or Document Policy reports).
• Streams cannot be transferred, and the tentative URLPattern methods compareComponent() and generate() are not implemented.
• Experimental APIs. The public APIs of the tentative features (proposals that no specification defines yet, e.g. Element.StreamAppendHtml(), Element.MatchContainer()) have the [Experimental] attribute: suppress their diagnostic (WEBPLATEXP0002...) to use them, see docs/experimental-apis.md.

Specifications

Building

./eng/webplatform/fetch-tests.sh
dotnet test --project tests/Meziantou.Framework.WebPlatform.Tests
dotnet test --project tests/Meziantou.Framework.WebPlatform.Imaging.Tests
dotnet test --project tests/Meziantou.Framework.WebPlatform.Wpt

The web-platform-tests (testharness.js tests and reftests, the reftests being rendered by the Imaging backend) run with the script engine of Meziantou.Framework.WebPlatform.JavaScript: see docs/web-platform-tests.md.

See docs/updating-specs.md to update the specification data, docs/fuzzing.md to fuzz the parsers, and docs/performance.md for the benchmarks.