Nefarius.Utilities.ETW 2.9.0

Nefarius.Utilities.ETW

ETW Log Files (.ETL) to JSON parser/converter library.

This is a fork of the fantastic ETW2JSON project by Microsoft and contributors.

Changes of this fork

• Converted console tool into a reusable class library
• Replaced P/Invoke code with source generators
• Changed namespace to Nefarius.Utilities.ETW to avoid conflicts with the origin library
• Added support for WPP Software Tracing decoding
  • Supports .PDB files as a decoding source — WPP provider GUIDs (ETW trace control GUIDs) are automatically extracted from TMC: annotations in the PDB symbol stream so no manual GUID lookup is required
  • Supports .TMF files as a decoding source
  • Full support for all WPP extended format specification strings (%!FUNC!, %!LEVEL!, %!FLAGS!, %!IPADDR!, %!TIMESTAMP!, %!delta!, %!due!, %!GUID!, %!CLSID!/%!LIBID!/%!IID!, %!PORT!, %!STATUS!, %!WINERROR!, %!HRESULT!, %!NDIS_STATUS!, %!NDIS_OID!, %!sid!, bitset/list enumerations, and more)
  • USEPREFIX/USESUFFIX trace message prefixes are automatically expanded — the %0 standard-prefix sentinel and %!FUNC!/%!LEVEL! context markers are resolved from the TMF metadata at decode time
• Added EtwUtil.EnumeratePdbReferences for lightweight pre-scanning of ETL files to collect all PDB metadata (symbol GUIDs, ages and file names) referenced in the trace before performing a full decode — enabling proper multi-PDB symbol resolution via symbol servers or local paths
• Added EtwUtil.EnumerateEventsAsync — a streaming IAsyncEnumerable<ReadOnlyMemory<byte>> API that yields each decoded ETW event as a self-contained UTF-8 JSON buffer as it is produced, rather than waiting for the full trace to finish; a dedicated background thread runs the blocking ProcessTrace call and feeds a bounded channel so the consumer is naturally backpressured and can process events concurrently with parsing; works well as a data source for real-time delivery scenarios such as FastEndpoints Server Sent Events:

public override async Task HandleAsync(CancellationToken ct)
{
    await Send.EventStreamAsync("etw-event", GetEtwStream(ct), ct);
}

private async IAsyncEnumerable<object> GetEtwStream([EnumeratorCancellation] CancellationToken ct)
{
    await foreach (ReadOnlyMemory<byte> eventJson in EtwUtil.EnumerateEventsAsync(
        [@"C:\traces\capture.etl"],
        opts => opts.WppDecodingContext = myDecodingContext,
        ct))
    {
        yield return JsonSerializer.Deserialize<object>(eventJson.Span)!;
    }
}

• Added realtime ETW session support via EtwRealtimeSession and EtwUtil.EnumerateRealtimeEventsAsync:
  • EtwRealtimeSession.Create(name) starts a user-mode ETW session (StartTraceW)
  • EnableProvider / DisableProvider toggle providers at runtime via EnableTraceEx2
  • EtwUtil.EnumerateRealtimeEventsAsync attaches to any externally-managed session and streams decoded events as UTF-8 JSON using the same bounded-channel / background-thread architecture as the file-based EnumerateEventsAsync
  • EtwUtil.ConvertRealtimeToJson blocks and writes events to a Utf8JsonWriter until cancelled
  • EtwUtil.StopOrphanSession cleanly stops a session left behind by a previous crash

Realtime decoding

The library ships two independent layers for realtime ETW consumption.

Layer 1 — attach to any existing session (consumer-only)

If the session is already running (started by logman, xperf, another process, or your own code), attach directly with a session name:

using CancellationTokenSource cts = new();

// Cancel after Ctrl+C or when your app shuts down.
Console.CancelKeyPress += (_, e) => { e.Cancel = true; cts.Cancel(); };

await foreach (ReadOnlyMemory<byte> eventJson in EtwUtil.EnumerateRealtimeEventsAsync(
    "MySession",
    cancellationToken: cts.Token))
{
    Console.WriteLine(System.Text.Encoding.UTF8.GetString(eventJson.Span));
}

Layer 2 — full session management (start + enable + stop)

Use EtwRealtimeSession when your application owns the session lifetime. Requires administrator / SeSystemProfilePrivilege.

// Clean up any orphan from a previous crash.
EtwUtil.StopOrphanSession("MyApp-Live");

using EtwRealtimeSession session = EtwRealtimeSession.Create("MyApp-Live",
    opts =>
    {
        opts.BufferSizeKb = 64;
        opts.FlushTimerSeconds = 1;
        opts.ClockResolution = EtwClockResolution.QueryPerformanceCounter;
    });

// Enable one or more providers at the desired verbosity level.
Guid microsoftWindowsKernelProcess = new("22FB2CD6-0E7B-422B-A0C7-2FAD1FD0E716");
session.EnableProvider(microsoftWindowsKernelProcess, TraceEventLevel.Information);

using CancellationTokenSource cts = new();
Console.CancelKeyPress += (_, e) => { e.Cancel = true; cts.Cancel(); };

await foreach (ReadOnlyMemory<byte> eventJson in EtwUtil.EnumerateRealtimeEventsAsync(
    session.SessionName,
    cancellationToken: cts.Token))
{
    Console.WriteLine(System.Text.Encoding.UTF8.GetString(eventJson.Span));
}
// Disposing session calls ControlTraceW(STOP) — no orphan left behind.

Realtime caveats

• Admin required to start a session. EtwRealtimeSession.Create throws EtwStartTraceException with ERROR_ACCESS_DENIED when the process is not elevated.
• Sessions persist after process exit. Always dispose EtwRealtimeSession, and call EtwUtil.StopOrphanSession("name") at startup as a safety net.
• WPP decoding in realtime mode requires a pre-built DecodingContext. The file-based EnumeratePdbReferences pre-scan only works on .etl files, not on live sessions. Build the DecodingContext from known .pdb or .tmf paths before starting the session.
• Kernel-mode providers / NT Kernel Logger are not supported in this release.

CLI (etwutils)

The tools/Nefarius.Utilities.ETW.CLI project builds a .NET global tool (etwutils) that wraps the realtime API and writes decoded events as NDJSON or plain tab-separated text to stdout.

dotnet tool install -g Nefarius.Utilities.ETW.CLI

See tools/Nefarius.Utilities.ETW.CLI/README.md for full installation instructions, command reference, and examples.

Known limitations

• Currently relies on Windows-only APIs so no support for other platforms
• %!ItemEnum!/%!ItemFlagsEnum! types display raw numeric values; PDB-based enum name resolution is not yet implemented
• Kernel-mode ETW providers and the NT Kernel Logger session are not yet supported for realtime mode

Documentation

Link to API docs.

Sources & 3rd party credits

• Microsoft/ETW2JSON
• Microsoft/ETW
• WPP Software Tracing
• microsoftarchive/bcl/Tools/ETW/traceEvent/SymbolEventParser.cs
• enum _TDH_CONTEXT_TYPE
• Learn / Windows / Windows Drivers / How do I add a prefix and suffix to a trace message?
• Learn / Windows / Windows Drivers / What are the WPP extended format specification strings
• kaitai-pdb
• MinVer
• Fast access to .net fields/properties
• Nefarius.Shared.PdbUtils