D4S.AgentMetrics 1.2.0

D4S.AgentMetrics

Internal NuGet package for Dev4Side AI agents built on .NET 10.

It handles, transparently:

• Handshake — verifies at startup that the agent is registered and active in the D4S back-end.
• Metric definition registration — sends all metric descriptors (default + custom) to the back-end.
• Business metric tracking — exposes IMetricWriter.TrackAsync with a bounded in-memory queue.
• Interaction tracking — measures total_interactions, avg_response_time_ms and error_rate automatically. Pick the tracker with the Transport setting: an HTTP middleware for REST/MCP agents (scoped to the endpoints you choose) or a Microsoft 365 Agents SDK middleware for Teams / custom engine agents (counts only real user messages).
• AI token tracking — an OpenTelemetry listener harvests gen_ai.usage.* from any Microsoft.Extensions.AI chat pipeline (D4S.Agent works out of the box, zero code).
• Periodic flush — drains the queue and POSTs the event batch to the back-end, with retry on transient failures.
• Heartbeat — sends periodic liveness pings to the back-end.

Internal use only. This package is not intended for public distribution.

Prerequisites

The agent must already be registered in the D4S back-end (manually or via the marketplace onboarding process) — this package only verifies that the registration exists.

Everything is fail-soft: if the AgentMetrics configuration is missing, or the back-end is unreachable, or the handshake is rejected, metric tracking is silently disabled and the agent starts and runs normally. This makes it safe to keep the wiring always on (see Quick start), including in local dev where no back-end is configured.

Installation

dotnet add package D4S.AgentMetrics

Quick start

This is the wiring used by the MAF365Agent template — two lines of code, everything else is configuration:

// Program.cs
builder.Services.AddAgentMetrics(options =>
{
    builder.Configuration.GetSection("AgentMetrics").Bind(options);
    // Project-specific business metrics go here (see "Tracking custom metrics"):
});

// After UseRouting, before the endpoint mappings:
app.UseAgentMetricsTracking();

// appsettings.json
{
  "AgentMetrics": {
    "AgentId": "my-agent",
    "DisplayName": "My Agent",
    "TenantId": "customer-tenant",
    "BaseUrl": "https://my-agent.azurewebsites.net",
    "D4SApiUrl": "https://d4s-api.azurewebsites.net",
    "D4SApiKey": "", // secret: set per environment (user secrets / app settings), never commit
    // Only measure the business endpoints — keeps health probes, platform warmup pings and
    // scanner noise out of total_interactions / avg_response_time_ms:
    "TrackedPathPrefixes": [ "/api/messages", "/mcp" ],
    // Only the verb that submits a question — excludes status-polling GETs or CORS preflight
    // OPTIONS that might share the same path, so total_interactions counts exactly one event
    // per question:
    "TrackedHttpMethods": [ "POST" ]
  }
}

Leave the section empty (or omit D4SApiUrl/D4SApiKey) to run with metrics disabled — the wiring is a clean no-op in that case.

Configuration

When a required option is missing, tracking is disabled for the session (logged at startup) and every metric operation becomes a no-op.

Interaction tracking: HTTP vs Bot

total_interactions and avg_response_time_ms must count one real user question and measure from question received to complete answer produced. How to capture that depends on how the agent receives questions, so the Transport setting selects the tracker. Only the selected one is active; the other is a no-op even if wired, so a shared startup template can register both.

Transport = Http (default) — REST / MCP agents

The agent receives questions as HTTP requests. app.UseAgentMetricsTracking() (see Quick start) measures each tracked request; scope it with TrackedPathPrefixes + TrackedHttpMethods so only the question-submitting call is counted.

Transport = Bot — Teams / custom engine agents (Microsoft 365 Agents SDK)

Microsoft 365 Agents SDK agents receive every interaction as an activity on a single /api/messages endpoint, so an HTTP filter cannot tell a real message from a conversationUpdate, typing or invoke. The bot middleware (AgentMetricsBotMiddleware, a Microsoft.Agents.Builder.IMiddleware) inspects Activity.Type and counts only message activities; it times the whole turn (message in → reply sent). Just set Transport to Bot:

// appsettings.json
"AgentMetrics": { /* ...required options... */ "Transport": "Bot" }

AddAgentMetrics(...) registers the middleware in DI as IMiddleware, and the default CloudAdapter created by AddAgent<TAgent>() consumes DI-registered middleware automatically — so no code change is needed in Program.cs or the agent. Only a custom adapter that bypasses DI needs an explicit attach in its constructor:

// Only for a hand-built adapter; the default CloudAdapter picks the middleware up on its own.
adapter.UseAgentMetricsBotTracking(services); // active only when Transport = Bot

With Transport = Bot the package also registers and emits conversations — a distinct count of Activity.Conversation.Id (one event per turn), so the dashboard can show distinct sessions and derive turns-per-session as total_interactions ÷ conversations. It is not registered for Transport = Http (no conversation-id producer, so it would be a permanently empty tile).

Copilot Studio (low-code, Microsoft-hosted) agents run no code of yours, so this package cannot instrument them — interaction metrics there must come from Dataverse analytics / Power Automate.

Default profile metrics

These six metrics are always included automatically — no extra code required. UseProfile(AgentProfile.Default) is optional but recommended for clarity.

error_rate is a Percentage: the back-end computes it as count(error_rate) ÷ count(total_interactions). The descriptor carries DenominatorMetricKey = "total_interactions" so the back-end knows what to divide by — without it the metric renders with no value.

A seventh metric, conversations (DistinctCount), is added automatically only when Transport = Bot (see above).

avg_confidence_score was removed from the default profile in 1.1.0: it had no automatic producer and rendered as a permanently empty tile. Register it as a custom scalar in agents that actually compute a confidence score.

avg_tokens_per_interaction was removed in 1.2.0: it was averaged per LLM call, not per interaction (a single turn can span several LLM calls), so the label was misleading. Raw token counts remain for volume telemetry; cost/token analysis is handled by Azure AI Foundry.

AI token tracking

Token metrics are collected automatically by listening to the OpenTelemetry GenAI spans emitted by Microsoft.Extensions.AI pipelines wrapped with UseOpenTelemetry(...):

• D4S.Agent / MAF365Agent hosts: zero configuration — the D4S.Agent chat and guardrail clients are already instrumented with the source names the listener observes by default.
• Other MEAI hosts: set TokenTelemetrySources to your instrumented source names (e.g. Experimental.Microsoft.Extensions.AI for the MEAI default). Do not include agent-level sources that re-emit aggregated usage, or tokens are double-counted.
• The listener reads both the current semantic conventions (gen_ai.usage.input_tokens / gen_ai.usage.output_tokens) and the legacy names (prompt_tokens / completion_tokens).
• Token counts are raw volume telemetry; monetary cost is derived in Azure AI Foundry, not here.

For hosts that use the Azure AI Foundry ChatCompletionsClient directly (no MEAI pipeline), the legacy explicit wrapper is still available:

builder.Services.AddTrackedChatClient(
    new ChatCompletionsClient(endpoint, new AzureKeyCredential(apiKey)),
    model: "gpt-4o");

Tracking custom metrics

Register the project's business metrics at startup, then record events wherever they happen:

// Registration (inside AddAgentMetrics):
options.AddRanking("top_exams", "Esami più richiesti", dimensionName: "exam_name", topN: 20);
options.AddScalar("teams_messages", "Domande da Teams");                     // Count
options.AddScalar("avg_confidence_score", "Confidence medio", AggregationType.Average);

// Then inject IMetricWriter wherever you need it:
public class MyService(IMetricWriter metrics)
{
    public async Task HandleAsync(string examName, double confidence)
    {
        await metrics.TrackAsync("teams_messages");                          // pure count
        await metrics.TrackAsync("top_exams", dimension: examName);          // ranking increment
        await metrics.TrackAsync("avg_confidence_score", value: (decimal)confidence);
    }
}

Rules:

• Metric definitions are sent to the back-end at handshake: keys must be registered at startup and stay stable across releases. Events for unregistered keys are dropped with a one-time warning.
• Ranking metrics are count-based; the dimension string is the ranked value. Never use free user text or user identifiers as a dimension — only catalog-controlled values.

Delivery guarantees

Events are queued in memory and flushed every FlushInterval:

• Transient back-end failures (network errors, HTTP 5xx) → the batch is re-enqueued and retried on the next cycle.
• Permanent rejections (HTTP 4xx) → the batch is dropped with a warning.
• The queue is bounded by MaxQueuedEvents; when full, the newest events are dropped with a warning. On process shutdown, unflushed events are lost — metrics are KPI-grade, not an audit log.

Internal use notice

This package is developed and maintained by Dev4Side for internal use only. It is not published on NuGet.org and is not intended for external consumption.