ActDim.Observability 1.0.7

ActDim.Observability

A lightweight, OpenTelemetry-centric observability library for .NET applications built on top of Microsoft.Extensions.Logging and System.Diagnostics.Activity.

Features

• Zero-Ceremony Developer API: Developers write standard ILogger calls and logger.BeginScope() without needing custom logger interfaces.
• DI Decorator (EventObservabilityLoggerFactory): Transparently decorates ILoggerFactory via DI container to inject EventObservabilityBridge for enriching logs and traces.
• Activity & OpenTelemetry Enrichment: Automatically transforms scope objects, DTOs, and structured log parameters into flattened, dotted OpenTelemetry attributes (user.id, order.price).
• Auto Activity Creation on Scope: Automatically starts an Activity span on logger.BeginScope() when no ambient span exists (Activity.Current == null), resolved via observability.PushActivitySourceName(...) or EventObservabilityOptions.DefaultActivitySourceName.
• Ambient Context Separation: IAmbientContext serves as a neutral ambient variable store. Only properties explicitly pushed via IObservabilityContext are exported to Activity tags.
• Status & Progress Tracking: First-class support for setting operation status text, icons, and progress percentage (observability.SetStatus("Downloading", icon: "🚀"), observability.SetProgress(45.5)).
• Selective Provider & Scope Suppression: Dynamically suppress console loggers, specific logger providers, or external scope export per async flow (observability.SuppressConsole(), observability.SuppressProviders("File"), observability.SuppressExternalScopes()).
• Provider Alias Resolution: Automatically resolves provider aliases via official .NET [ProviderAlias] attributes or custom provider mappings.

Architectural Rationale: Dedicated Observability Engines vs Relational Databases

Storing high-throughput logs and distributed traces in traditional relational databases (like PostgreSQL or MySQL) creates significant operational and performance bottlenecks. Dedicated observability engines solve these problems through specialized architectures.

Key Bottlenecks of Relational Databases for Telemetry

• I/O & WAL Overhead: Transactional databases prioritize strict ACID guarantees. Every log entry generates Write-Ahead Log (WAL) traffic and buffer churn, causing massive disk I/O overhead that degrades core application performance.
• Storage Inefficiency & Bloat: Row-oriented architectures compress telemetry poorly. Rotating old data via deletes or TTL triggers heavy background cleanup processes (like VACUUM), causing table bloat and CPU spikes.
• JSON Indexing Trade-Off: Querying dynamic JSON attributes requires heavy indexing (such as GIN indexes), which cripples write speeds and increases index size beyond the data itself. Without indexes, queries result in slow full-table scans.
• Lack of Observability Tooling: Relational databases lack native primitives for live tailing, distributed trace waterfalls (spans/DAGs), and log-centric aggregate pipelines.

Core Advantages of Dedicated Observability Engines

• Columnar & Append-Only Storage: Uses efficient storage engines (e.g., Apache Parquet, LSM-trees) tailored for time-series and log data, achieving 10–15x higher compression ratios.
• Telemetry-First Query Languages: Purpose-built query languages (like LogsQL or telemetry-aware SQL dialects) parse, extract, and filter arbitrary JSON fields on the fly without heavy index maintenance.
• Built-in APM Visualizations: Native support for end-to-end trace waterfalls, span trees, and real-time log streaming right out of the box.

Recommended Lightweight Open-Source Solutions

• VictoriaLogs: A high-performance, resource-efficient log engine requiring minimal CPU and RAM (~50–100 MB). It eliminates high-cardinality bottlenecks, indexes all fields automatically, and features the expressive LogsQL language for structured JSON analysis.
• OpenObserve: A single Rust binary that covers logs, traces, and metrics out of the box. It uses Apache Parquet for storage, natively accepts OpenTelemetry (OTLP) data, and provides a full-featured web UI with trace waterfalls, log exploration, and dashboards without requiring Docker, Java, or external databases.
• ClickHouse: An industry-standard, ultra-high-performance columnar analytical database engine for high-volume logs, metrics, and trace telemetry. While we do not maintain a dedicated integration test suite for ClickHouse in this repository, modern distributions and telemetry stacks bundle or integrate with HyperDX (an open-source APM & log exploration Web UI), providing a comprehensive out-of-the-box user experience for analyzing traces and logs.

.NET Observability & Logging Architecture: Best Practices Guide

Understanding how modern telemetry works in .NET is essential for building resilient, high-performance applications. Modern observability rests on Three Pillars: Traces, Metrics, and Logs.

flowchart TD
    subgraph App[".NET Application (ActDim.Observability)"]
        Logs["Logs (ILogger / EventObservabilityBridge)"]
        Traces["Traces (System.Diagnostics.Activity)"]
        Metrics["Metrics (System.Diagnostics.Metrics)"]
    end

    subgraph Instrumentations["Built-in & BCL Instrumentations"]
        ASPNET["ASP.NET Core (HTTP Requests)"]
        Kestrel["Kestrel & Hosting (Server)"]
        HTTP["HttpClient & System.Net (DNS / HTTP)"]
        EF["EF Core & SqlClient (DB Queries)"]
        Runtime["System.Runtime (GC / ThreadPool / CPU)"]
    end

    subgraph Collection["Export Architecture & Pipeline"]
        Direct["Direct OTLP Exporter (App -> Sink)"]
        Collector["OpenTelemetry Collector (Tail Sampling & Multi-Sink Routing)"]
    end

    subgraph Sinks["Observability Backends & Visualizers"]
        VL["VictoriaLogs (High-Perf Log Engine / LogsQL)"]
        OO["OpenObserve (All-in-One: Logs + Traces + Metrics UI)"]
        Grafana["Grafana Dashboards (Loki/VL Logs, Tempo Traces, Prom Metrics)"]
    end

    Instrumentations --> Traces
    Instrumentations --> Metrics

    Logs --> Direct
    Traces --> Direct
    Metrics --> Direct

    Logs --> Collector
    Traces --> Collector
    Metrics --> Collector

    Direct --> VL
    Direct --> OO
    Collector --> VL
    Collector --> OO
    Collector --> Grafana

1. Complete .NET Setup Code (Logging, Tracing & Metrics)

Below is the standard, production-ready configuration using Microsoft.Extensions.DependencyInjection and OpenTelemetry:

using Microsoft.AspNetCore.Builder;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging;
using OpenTelemetry;
using OpenTelemetry.Exporter;
using OpenTelemetry.Metrics;
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;
using ActDim.Observability;

var builder = WebApplication.CreateBuilder(args);

// 1. Define Resource attributes (Service Name, Version, Environment)
var resourceBuilder = ResourceBuilder.CreateDefault()
    .AddService(
        serviceName: builder.Configuration["Telemetry:ServiceName"] ?? "PractixService",
        serviceVersion: "1.0.0",
        serviceInstanceId: Environment.MachineName);

// 2. Configure Dynamic Trace Sampling from appsettings.json
var samplingRatio = builder.Configuration.GetValue<double>("Telemetry:TraceSamplingRatio", 0.1); // Default 10%
var enableSampling = builder.Configuration.GetValue<bool>("Telemetry:EnableSampling", true);

// During active incidents, set Telemetry:EnableSampling = false in appsettings.json to capture 100% of traces!
Sampler sampler = enableSampling
    ? new ParentBasedSampler(new TraceIdRatioBasedSampler(samplingRatio))
    : new AlwaysOnSampler();

// 3. Register OpenTelemetry Tracing & Metrics
builder.Services.AddOpenTelemetry()
    .WithResource(resourceBuilder)
    .WithTracing(tracing =>
    {
        tracing
            .SetSampler(sampler)
            .AddSource(EventObservabilityOptions.DefaultActivitySourceName)
            .AddAspNetCoreInstrumentation(opts =>
            {
                opts.RecordException = true;
            })
            .AddHttpClientInstrumentation(opts =>
            {
                opts.RecordException = true;
            })
            .AddEntityFrameworkCoreInstrumentation(opts =>
            {
                opts.SetDbStatementForText = true;
            })
            .AddOtlpExporter(opts =>
            {
                opts.Endpoint = new Uri(builder.Configuration["Telemetry:OtlpEndpoint"] ?? "http://localhost:4317");
                opts.Protocol = OtlpExportProtocol.Grpc;
            });
    })
    .WithMetrics(metrics =>
    {
        metrics
            // Enable Exemplars (attaches active TraceId/SpanId to Metric Histograms)
            .SetExemplarFilter(ExemplarFilterType.TraceBased)
            // System & Framework Meters
            .AddMeter("Microsoft.AspNetCore.Hosting")
            .AddMeter("Microsoft.AspNetCore.Server.Kestrel")
            .AddMeter("System.Net.Http")
            .AddMeter("System.Net.NameResolution") // DNS resolution timing & metrics
            .AddMeter("System.Runtime")             // GC, ThreadPool, Locks, CPU
            .AddRuntimeInstrumentation()
            .AddOtlpExporter(opts =>
            {
                opts.Endpoint = new Uri(builder.Configuration["Telemetry:OtlpEndpoint"] ?? "http://localhost:4317");
                opts.Protocol = OtlpExportProtocol.Grpc;
            });
    });

// 4. Register ActDim.Observability & EventObservabilityBridge
builder.Services.AddEventObservability(logging =>
{
    logging.AddConsole();
    logging.AddOtlpExporter(opts =>
    {
        opts.Endpoint = new Uri(builder.Configuration["Telemetry:OtlpEndpoint"] ?? "http://localhost:4318/v1/logs");
        opts.Protocol = OtlpExportProtocol.HttpProtobuf;
    });
}, options =>
{
    options.IncludeExternalScopes = true;
});

var app = builder.Build();
app.Run();

2. Built-in Framework & System Instrumentations

OpenTelemetry .NET leverages BCL ActivitySource and Meter events built into the .NET runtime and ASP.NET Core:

3. Dynamic Ratio-Based Sampling via appsettings.json

High-throughput production services emitting 100% of traces create massive storage costs and network overhead. Head Sampling evaluates trace sampling at span creation.

appsettings.json Configuration

{
  "Telemetry": {
    "ServiceName": "OrderProcessingService",
    "OtlpEndpoint": "http://otel-collector:4317",
    "TraceSamplingRatio": 0.05,
    "EnableSampling": true
  }
}

Incident Override Mode (Zero-Loss Telemetry Toggle)

During production incidents or debugging sessions, operators can update Telemetry:EnableSampling to false via environment variables or configuration reload without deploying code:

# Override env var during incident to capture 100% of traces:
Telemetry__EnableSampling=false

// Code implementation dynamically evaluates the configuration value:
Sampler sampler = configuration.GetValue<bool>("Telemetry:EnableSampling", true)
    ? new ParentBasedSampler(new TraceIdRatioBasedSampler(configuration.GetValue<double>("Telemetry:TraceSamplingRatio", 0.1)))
    : new AlwaysOnSampler();

4. Metrics & Exemplars

An Exemplar links a metric measurement (such as a 99th percentile request duration histogram bucket) directly to the exact trace_id and span_id of the HTTP request that produced it.

Grafana Metric Chart: kestrel.request.duration [Histogram Bucket: > 500ms]
                     │
                     └── Exemplar Attached: [trace_id = 4bf92f3577b34da6a3ce929d0e0e4736]
                                 │
                                 └── (Click) -> Opens Trace Waterfall in OpenObserve / Tempo / Jaeger!

Enabling Exemplars in .NET

To enable Exemplars, use .SetExemplarFilter(ExemplarFilterType.TraceBased). This automatically attaches trace context from Activity.Current whenever a metric measurement is recorded while a trace is sampled.

5. OpenTelemetry Collector Architecture & Pipelines

The OpenTelemetry Collector is a high-performance proxy component deployed alongside your application stack.

flowchart LR
    App[".NET App (Logs, Traces, Metrics)"] -->|OTLP / gRPC or HTTP| Receiver["Receiver (otlp)"]
    subgraph OtelCollector["OpenTelemetry Collector Pipeline"]
        Receiver --> Processors["Processors (batch, memory_limiter, tail_sampling)"]
        Processors --> Exporters["Exporters (otlp, prometheus, victorialogs)"]
    end
    Exporters -->|Logs + Traces + Metrics| OO["OpenObserve (All-in-One APM UI)"]
    Exporters -->|Logs| VL["VictoriaLogs (LogsQL)"]
    Exporters -->|Traces| Tempo["Grafana Tempo / Jaeger"]
    Exporters -->|Metrics| Prom["Prometheus"]

Why Use an Otel Collector?

1. Process Offloading: Offloads heavy batching, compression (GZip/Zstd), retries, and network TLS overhead out of the .NET application process.
2. Security & Credential Isolation: API tokens, basic auth headers, and production credentials live in the Collector configuration rather than microservice environment variables.
3. Multi-Backend Routing: Simultaneously forwards logs to VictoriaLogs, traces to Tempo or OpenObserve, and metrics to Prometheus.
4. Tail Sampling: Evaluates sampling rules after the entire distributed trace finishes.

OpenTelemetry Collector Tail Sampling Configuration (otel-collector-config.yaml)

Unlike Head Sampling (which drops traces randomly at the start), Tail Sampling buffers completed traces in the Collector memory and applies intelligent rules:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 75
    spike_limit_percentage: 15

  batch:
    send_batch_size: 8192
    timeout: 1s

  tail_sampling:
    decision_wait: 10s
    num_traces: 10000
    expected_new_traces_per_sec: 2000
    policies:
      # Policy 1: Always drop health checks & metric scrapes
      - name: drop-health-checks
        type: string_attribute
        string_attribute:
          key: http.target
          values: [ "/healthz", "/metrics", "/ready" ]
          enabled_regex_matching: false
          invert_match: true

      # Policy 2: Always keep 100% of traces containing HTTP 5xx errors or exceptions
      - name: keep-all-errors
        type: status_code
        status_code:
          status_codes: [ ERROR ]

      # Policy 3: Always keep 100% of slow requests (> 500ms duration)
      - name: keep-slow-requests
        type: latency
        latency:
          threshold_ms: 500

      # Policy 4: Sample 5% of normal successful requests (HTTP 200 OK)
      - name: sample-normal-traffic
        type: probabilistic
        probabilistic:
          sampling_percentage: 5.0

exporters:
  otlp/openobserve:
    endpoint: "http://openobserve:5080/api/default"
    headers:
      Authorization: "Basic cm9vdEBleGFtcGxlLmNvbTpDb21wbGV4cGFzcyMxMjM="
  
  otlp/victorialogs:
    endpoint: "http://victoria-logs:9428/insert/opentelemetry/v1/logs"
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, tail_sampling, batch]
      exporters: [otlp/openobserve]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp/victorialogs, otlp/openobserve]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp/openobserve]

6. Structured Logging & Context Scopes (logger.BeginScope)

Structured Logging vs. String Interpolation

// ❌ INCORRECT: String Interpolation (Destroys structure, produces unindexed raw text)
logger.LogInformation($"Processed order {orderId} for user {userId}");

// ✅ CORRECT: Named Template Parameters (Extracts structured key-value pairs)
logger.LogInformation("Processed order {OrderId} for user {UserId}", orderId, userId);

• Why it matters: Columnar log engines (VictoriaLogs, OpenObserve, ClickHouse) automatically parse template parameters into typed JSON attributes (OrderId: "1234"). This enables instant filtering with LogsQL (OrderId:="1234") or SQL without slow regex searches.

Why Logging Scopes are Vital (logger.BeginScope)

Logging scopes push ambient contextual key-value pairs onto the current async execution flow:

using (logger.BeginScope(new Dictionary<string, object> { ["tenant.id"] = "acme", ["user.id"] = "42" }))
{
    // Every log statement executed in this block (or child async methods) automatically inherits tenant.id and user.id!
    logger.LogInformation("Processing payment");
    await ExecutePaymentStepAsync();
    logger.LogInformation("Payment completed");
}

How ActDim.Observability Enhances Scopes

• Scope Flattening: EventObservabilityBridge automatically flattens dictionary scopes, anonymous objects, and DTOs into dotted OTel tags (tenant.id, user.id, order.price).
• Auto Activity Creation: Calling logger.BeginScope() or logger.BeginMethodScope() starts a new OpenTelemetry Activity span if no active span exists (Activity.Current == null).
• Ambient Context Binding: Integrates directly with IAmbientContext and IObservabilityContext, capturing ambient state without parameter drilling.

Zero-Ceremony & Framework Independence (Serilog vs. Native .NET Logging)

• Zero Third-Party Logging Dependencies: ActDim.Observability eliminates the need for heavy external logging frameworks like Serilog or NLog. Standard Microsoft.Extensions.Logging combined with OpenTelemetry OTLP Exporters delivers zero-allocation, high-performance structured logging natively out of the box.
• Seamless Serilog Interop: If an existing application already relies on Serilog, ActDim.Observability integrates transparently. Developers can retain builder.Host.UseSerilog() or Serilog sinks—EventObservabilityBridge decorates ILoggerFactory via standard BCL interfaces, capturing scopes and ambient state without conflicts.

Installation

Install via the .NET CLI:

dotnet add package ActDim.Observability

Or via Package Manager Console:

Install-Package ActDim.Observability

Registration

Register observability in your IServiceCollection:

services.AddEventObservability(logging =>
{
    logging.AddConsole();
}, options =>
{
    options.IncludeExternalScopes = false; // Default: false
});

Usage

1. Status & Progress Reporting

var observability = serviceProvider.GetRequiredService<IObservabilityContext>();

using (observability.SetStatus("Downloading Dataset", icon: "🚀"))
using (observability.SetProgress(45.5))
using (observability.Push("priority", "high"))
{
    logger.LogInformation("Importing rows into database");
}

2. Method Scopes with OpenTelemetry Semantic Conventions

Use logger.BeginMethodScope() to automatically capture the executing method name, source file, and line number without manual string formatting. Scope properties strictly adhere to the OpenTelemetry Source Code Semantic Conventions:

public class OrderService
{
    private readonly ILogger<OrderService> _logger;

    public OrderService(ILogger<OrderService> logger)
    {
        _logger = logger;
    }

    public async Task ProcessOrderAsync(string orderId)
    {
        // Automatically captures code.function="ProcessOrderAsync", code.filename="OrderService.cs", code.lineno=...
        using (_logger.BeginMethodScope())
        {
            _logger.LogInformation("Processing order {OrderId}", orderId);
        }

        // Merge custom state with caller code context
        using (_logger.BeginMethodScope(new Dictionary<string, object?> { ["OrderId"] = orderId }))
        {
            _logger.LogInformation("Order completed");
        }
    }
}

Why OpenTelemetry Semantic Conventions? Standard attribute names (code.function, code.filepath, code.lineno) enable APM tools, log aggregators, and distributed trace visualizers (Jaeger, Grafana Tempo, Datadog, Dynatrace, and .NET Aspire) to natively index, filter, and navigate directly to source code locations.

3. Selective Provider Suppression

// Suppress Console logger output while preserving Activity traces and other logger sinks
using (observability.SuppressConsole())
{
    logger.LogInformation("Log without console output");
}

// Suppress specific providers by alias or name (e.g., "File", "Console")
using (observability.SuppressProviders("File", "Console"))
{
    logger.LogInformation("Log without File and Console outputs");
}

4. Integration Testing & Tooling (VictoriaLogs & OpenObserve)

ActDim.Observability.Tests includes integration test suites and developer scripts for validating telemetry ingestion and log search:

• VictoriaLogs Integration (VictoriaLogsIntegrationTests):

  • Validates NDJSON ingestion (/insert/jsonline), _msg field format, AmbientContext properties, BeginMethodScope() OTel caller metadata (code.function, code.filename, code.filepath, code.lineno), and LogsQL queries.
  • Launcher & Download Scripts: Tools/victoria-logs/run-victoria-logs.cmd (auto-opens VMUI Web GUI at http://localhost:9428/select/vmui) and download-victoria-logs.cmd.

• OpenObserve Integration (OpenObserveIntegrationTests):

  • Validates JSON log ingestion (/api/{org}/{stream}/_json), AmbientContext enrichment, and SQL Search API (POST /api/{org}/_search).
  • Launcher & Download Scripts: Tools/openobserve/run-openobserve.cmd (auto-opens Web GUI at http://localhost:5080 with default admin credentials root@example.com / Complexpass#123) and download-openobserve.cmd.

• Process Auto-Launch: Both integration tests automatically detect running local instances or auto-launch local binaries from Tools/ into isolated temporary storage paths.

Testing & Quality

• Test Suite: ActDim.Observability.Tests
• Total Tests: 30 passed (100% success rate, 0 failed, 0 skipped)
• Target Framework: .NET 10.0

dotnet test Tests/Observability.Tests/ActDim.Observability.Tests.csproj

License

This project is licensed under the MIT License.