D4S.AgentMetrics 1.1.0

D4S.AgentMetrics

Internal NuGet package for Dev4Side AI agents built on .NET 10.

It handles, transparently:

• Handshake — verifies at startup that the agent is registered and active in the D4S back-end.
• Metric definition registration — sends all metric descriptors (default + custom) to the back-end.
• Business metric tracking — exposes IMetricWriter.TrackAsync with a bounded in-memory queue.
• HTTP request tracking — middleware measuring interactions, response time and error rate, scoped to the endpoints you choose.
• AI token tracking — an OpenTelemetry listener harvests gen_ai.usage.* from any Microsoft.Extensions.AI chat pipeline (D4S.Agent works out of the box, zero code).
• Periodic flush — drains the queue and POSTs the event batch to the back-end, with retry on transient failures.
• Heartbeat — sends periodic liveness pings to the back-end.

Internal use only. This package is not intended for public distribution.

Prerequisites

The agent must already be registered in the D4S back-end (manually or via the marketplace onboarding process) — this package only verifies that the registration exists.

Everything is fail-soft: if the AgentMetrics configuration is missing, or the back-end is unreachable, or the handshake is rejected, metric tracking is silently disabled and the agent starts and runs normally. This makes it safe to keep the wiring always on (see Quick start), including in local dev where no back-end is configured.

Installation

dotnet add package D4S.AgentMetrics

Quick start

This is the wiring used by the MAF365Agent template — two lines of code, everything else is configuration:

// Program.cs
builder.Services.AddAgentMetrics(options =>
{
    builder.Configuration.GetSection("AgentMetrics").Bind(options);
    // Project-specific business metrics go here (see "Tracking custom metrics"):
});

// After UseRouting, before the endpoint mappings:
app.UseAgentMetricsTracking();

// appsettings.json
{
  "AgentMetrics": {
    "AgentId": "my-agent",
    "DisplayName": "My Agent",
    "TenantId": "customer-tenant",
    "BaseUrl": "https://my-agent.azurewebsites.net",
    "D4SApiUrl": "https://d4s-api.azurewebsites.net",
    "D4SApiKey": "", // secret: set per environment (user secrets / app settings), never commit
    // Only measure the business endpoints — keeps health probes, platform warmup pings and
    // scanner noise out of total_interactions / avg_response_time_ms:
    "TrackedPathPrefixes": [ "/api/messages", "/mcp" ]
  }
}

Leave the section empty (or omit D4SApiUrl/D4SApiKey) to run with metrics disabled — the wiring is a clean no-op in that case.

Configuration

When a required option is missing, tracking is disabled for the session (logged at startup) and every metric operation becomes a no-op.

Default profile metrics

These seven metrics are always included automatically — no extra code required. UseProfile(AgentProfile.Default) is optional but recommended for clarity.

avg_confidence_score was removed from the default profile in 1.1.0: it had no automatic producer and rendered as a permanently empty tile. Register it as a custom scalar in agents that actually compute a confidence score.

AI token tracking

Token metrics are collected automatically by listening to the OpenTelemetry GenAI spans emitted by Microsoft.Extensions.AI pipelines wrapped with UseOpenTelemetry(...):

• D4S.Agent / MAF365Agent hosts: zero configuration — the D4S.Agent chat and guardrail clients are already instrumented with the source names the listener observes by default.
• Other MEAI hosts: set TokenTelemetrySources to your instrumented source names (e.g. Experimental.Microsoft.Extensions.AI for the MEAI default). Do not include agent-level sources that re-emit aggregated usage, or tokens are double-counted.
• The listener reads both the current semantic conventions (gen_ai.usage.input_tokens / gen_ai.usage.output_tokens) and the legacy names (prompt_tokens / completion_tokens).
• avg_tokens_per_interaction is averaged per LLM call (a single agent turn with tool calls spans several LLM calls).

For hosts that use the Azure AI Foundry ChatCompletionsClient directly (no MEAI pipeline), the legacy explicit wrapper is still available:

builder.Services.AddTrackedChatClient(
    new ChatCompletionsClient(endpoint, new AzureKeyCredential(apiKey)),
    model: "gpt-4o");

Tracking custom metrics

Register the project's business metrics at startup, then record events wherever they happen:

// Registration (inside AddAgentMetrics):
options.AddRanking("top_exams", "Esami più richiesti", dimensionName: "exam_name", topN: 20);
options.AddScalar("teams_messages", "Domande da Teams");                     // Count
options.AddScalar("avg_confidence_score", "Confidence medio", AggregationType.Average);

// Then inject IMetricWriter wherever you need it:
public class MyService(IMetricWriter metrics)
{
    public async Task HandleAsync(string examName, double confidence)
    {
        await metrics.TrackAsync("teams_messages");                          // pure count
        await metrics.TrackAsync("top_exams", dimension: examName);          // ranking increment
        await metrics.TrackAsync("avg_confidence_score", value: (decimal)confidence);
    }
}

Rules:

• Metric definitions are sent to the back-end at handshake: keys must be registered at startup and stay stable across releases. Events for unregistered keys are dropped with a one-time warning.
• Ranking metrics are count-based; the dimension string is the ranked value. Never use free user text or user identifiers as a dimension — only catalog-controlled values.

Delivery guarantees

Events are queued in memory and flushed every FlushInterval:

• Transient back-end failures (network errors, HTTP 5xx) → the batch is re-enqueued and retried on the next cycle.
• Permanent rejections (HTTP 4xx) → the batch is dropped with a warning.
• The queue is bounded by MaxQueuedEvents; when full, the newest events are dropped with a warning. On process shutdown, unflushed events are lost — metrics are KPI-grade, not an audit log.

Internal use notice

This package is developed and maintained by Dev4Side for internal use only. It is not published on NuGet.org and is not intended for external consumption.