D4S.AgentMetrics 1.3.0

D4S.AgentMetrics

Internal NuGet package for Dev4Side AI agents built on .NET 10.

It handles, transparently:

• Handshake — verifies at startup that the agent is registered and active in the D4S back-end.
• Metric definition registration — sends all metric descriptors (default + custom) to the back-end.
• Business metric tracking — exposes IMetricWriter.TrackAsync with a bounded in-memory queue.
• Interaction tracking — measures total_interactions, avg_response_time_ms and error_rate automatically. Pick the tracker with the Transport setting: an HTTP middleware for REST/MCP agents (scoped to the endpoints you choose) or a Microsoft 365 Agents SDK middleware for Teams / custom engine agents (counts only real user messages).
• AI token tracking — an OpenTelemetry listener harvests gen_ai.usage.* from any Microsoft.Extensions.AI chat pipeline (D4S.Agent works out of the box, zero code), dimensioned by model so the back-end can price it. An opt-in chat-pipeline decorator adds the cached-input share, which the spans do not expose.
• Periodic flush — drains the queue and POSTs the event batch to the back-end, with retry on transient failures.
• Heartbeat — sends periodic liveness pings to the back-end.

Internal use only. This package is not intended for public distribution.

Prerequisites

The agent must already be registered in the D4S back-end (manually or via the marketplace onboarding process) — this package only verifies that the registration exists.

Everything is fail-soft: if the AgentMetrics configuration is missing, or the back-end is unreachable, or the handshake is rejected, metric tracking is silently disabled and the agent starts and runs normally. This makes it safe to keep the wiring always on (see Quick start), including in local dev where no back-end is configured.

Installation

dotnet add package D4S.AgentMetrics

Quick start

This is the wiring used by the MAF365Agent template — two lines of code, everything else is configuration:

// Program.cs
builder.Services.AddAgentMetrics(options =>
{
    builder.Configuration.GetSection("AgentMetrics").Bind(options);
    // Project-specific business metrics go here (see "Tracking custom metrics"):
});

// After UseRouting, before the endpoint mappings:
app.UseAgentMetricsTracking();

// appsettings.json
{
  "AgentMetrics": {
    "AgentId": "my-agent",
    "DisplayName": "My Agent",
    "TenantId": "customer-tenant",
    "BaseUrl": "https://my-agent.azurewebsites.net",
    "D4SApiUrl": "https://d4s-api.azurewebsites.net",
    "D4SApiKey": "", // secret: set per environment (user secrets / app settings), never commit
    // Only measure the business endpoints — keeps health probes, platform warmup pings and
    // scanner noise out of total_interactions / avg_response_time_ms:
    "TrackedPathPrefixes": [ "/api/messages", "/mcp" ],
    // Only the verb that submits a question — excludes status-polling GETs or CORS preflight
    // OPTIONS that might share the same path, so total_interactions counts exactly one event
    // per question:
    "TrackedHttpMethods": [ "POST" ]
  }
}

Leave the section empty (or omit D4SApiUrl/D4SApiKey) to run with metrics disabled — the wiring is a clean no-op in that case.

Configuration

When a required option is missing, tracking is disabled for the session (logged at startup) and every metric operation becomes a no-op.

Interaction tracking: HTTP vs Bot

total_interactions and avg_response_time_ms must count one real user question and measure from question received to complete answer produced. How to capture that depends on how the agent receives questions, so the Transport setting selects the tracker. Only the selected one is active; the other is a no-op even if wired, so a shared startup template can register both.

Transport = Http (default) — REST / MCP agents

The agent receives questions as HTTP requests. app.UseAgentMetricsTracking() (see Quick start) measures each tracked request; scope it with TrackedPathPrefixes + TrackedHttpMethods so only the question-submitting call is counted.

Transport = Bot — Teams / custom engine agents (Microsoft 365 Agents SDK)

Microsoft 365 Agents SDK agents receive every interaction as an activity on a single /api/messages endpoint, so an HTTP filter cannot tell a real message from a conversationUpdate, typing or invoke. The bot middleware (AgentMetricsBotMiddleware, a Microsoft.Agents.Builder.IMiddleware) inspects Activity.Type and counts only message activities; it times the whole turn (message in → reply sent). Just set Transport to Bot:

// appsettings.json
"AgentMetrics": { /* ...required options... */ "Transport": "Bot" }

AddAgentMetrics(...) registers the middleware in DI as IMiddleware, and the default CloudAdapter created by AddAgent<TAgent>() consumes DI-registered middleware automatically — so no code change is needed in Program.cs or the agent. Only a custom adapter that bypasses DI needs an explicit attach in its constructor:

// Only for a hand-built adapter; the default CloudAdapter picks the middleware up on its own.
adapter.UseAgentMetricsBotTracking(services); // active only when Transport = Bot

With Transport = Bot the package also registers and emits conversations — a distinct count of Activity.Conversation.Id (one event per turn), so the dashboard can show distinct sessions and derive turns-per-session as total_interactions ÷ conversations. It is not registered for Transport = Http (no conversation-id producer, so it would be a permanently empty tile).

Copilot Studio (low-code, Microsoft-hosted) agents run no code of yours, so this package cannot instrument them — interaction metrics there must come from Dataverse analytics / Power Automate.

Default profile metrics

These six metrics are always included automatically — no extra code required. UseProfile(AgentProfile.Default) is optional but recommended for clarity.

error_rate is a Percentage: the back-end computes it as count(error_rate) ÷ count(total_interactions). The descriptor carries DenominatorMetricKey = "total_interactions" so the back-end knows what to divide by — without it the metric renders with no value.

Every token event also carries the model name as its dimension (TrackTokensByModel, on by default), because unit prices differ per model. The scalar totals are unchanged — summing across all dimensions gives the same number as before.

Two more metrics are added automatically only under a specific configuration:

• conversations (DistinctCount) when Transport = Bot (see above).
• cached_prompt_tokens_total (Sum) when CachedTokenTrackingEnabled is set — see below.

avg_confidence_score was removed from the default profile in 1.1.0: it had no automatic producer and rendered as a permanently empty tile. Register it as a custom scalar in agents that actually compute a confidence score.

avg_tokens_per_interaction was removed in 1.2.0: it was averaged per LLM call, not per interaction (a single turn can span several LLM calls), so the label was misleading. Raw token counts remain, and since 1.3.0 they are priceable in the D4S back-end — see Token cost.

AI token tracking

Token metrics are collected automatically by listening to the OpenTelemetry GenAI spans emitted by Microsoft.Extensions.AI pipelines wrapped with UseOpenTelemetry(...):

• D4S.Agent / MAF365Agent hosts: zero configuration — the D4S.Agent chat and guardrail clients are already instrumented with the source names the listener observes by default.
• Other MEAI hosts: set TokenTelemetrySources to your instrumented source names (e.g. Experimental.Microsoft.Extensions.AI for the MEAI default). Do not include agent-level sources that re-emit aggregated usage, or tokens are double-counted.
• The listener reads both the current semantic conventions (gen_ai.usage.input_tokens / gen_ai.usage.output_tokens) and the legacy names (prompt_tokens / completion_tokens).
• Each event is dimensioned by model, taken from gen_ai.response.model and falling back to gen_ai.request.model.

For hosts that use the Azure AI Foundry ChatCompletionsClient directly (no MEAI pipeline), the legacy explicit wrapper is still available:

builder.Services.AddTrackedChatClient(
    new ChatCompletionsClient(endpoint, new AzureKeyCredential(apiKey)),
    model: "gpt-4o");

Token cost: per-model and cached tokens

Token counts only become a cost once the back-end knows which model produced them and how many input tokens were served from cache (providers bill cached input at a fraction of the normal rate). The package reports both, from two producers with disjoint metric keys — so they never double-count, in any pipeline order, and either one working alone still yields consistent data:

Enabling cached-token tracking

The cached count is not on the OpenTelemetry spans — Microsoft.Extensions.AI only emits gen_ai.usage.input_tokens and gen_ai.usage.output_tokens. It lives on ChatResponse.Usage.AdditionalCounts, so reaching it means sitting in the chat pipeline:

// 1. Register the metric (inside AddAgentMetrics, or via appsettings):
options.CachedTokenTrackingEnabled = true;

// 2. Add the decorator to the chat pipeline, next to UseOpenTelemetry:
chatClientBuilder
    .UseOpenTelemetry(sourceName: "Dev4Side.MAF365Agent.ChatClient")
    .UseAgentMetricsTokenDetails();

Order relative to UseOpenTelemetry does not matter. Both steps are needed:

• Neither done (the default) — everything else keeps working, cached_prompt_tokens_total is simply never registered nor reported. This is the fail-soft path: no agent breaks by not adopting it.
• Flag only — the metric is registered but has no producer, so it renders as an empty tile.
• Pipeline only — the decorator logs a one-time warning naming the missing flag and stays inert.

Streaming is supported: usage is read from the UsageContent update the provider emits at the end of the stream.

How the back-end turns this into a cost

cached_prompt_tokens_total is a subset of prompt_tokens_total, not an additional amount, so the input side has to be split before pricing:

cost = (prompt_tokens_total - cached_prompt_tokens_total) x input_rate(model)
     +  cached_prompt_tokens_total                       x cached_input_rate(model)
     +  completion_tokens_total                          x output_rate(model)

• Never price total_tokens_total — it is prompt + completion, kept for volume dashboards only. Multiplying it by any blended rate gives a wrong number.
• With cached tracking off, cached_prompt_tokens_total is absent, the first term collapses to prompt_tokens_total x input_rate and the cost is overstated by exactly the cached discount.
• The dimension value is the model as reported by the provider (e.g. gpt-4o-2024-11-20), or the Azure OpenAI deployment name when the response carries no model. A model with no entry in the price table must be surfaced as unpriced, not treated as free — otherwise cost silently drops when an agent switches deployment.
• Reasoning tokens need no separate metric: providers already bill them inside the output tokens.

Tracking custom metrics

Register the project's business metrics at startup, then record events wherever they happen:

// Registration (inside AddAgentMetrics):
options.AddRanking("top_exams", "Esami più richiesti", dimensionName: "exam_name", topN: 20);
options.AddScalar("teams_messages", "Domande da Teams");                     // Count
options.AddScalar("avg_confidence_score", "Confidence medio", AggregationType.Average);

// Then inject IMetricWriter wherever you need it:
public class MyService(IMetricWriter metrics)
{
    public async Task HandleAsync(string examName, double confidence)
    {
        await metrics.TrackAsync("teams_messages");                          // pure count
        await metrics.TrackAsync("top_exams", dimension: examName);          // ranking increment
        await metrics.TrackAsync("avg_confidence_score", value: (decimal)confidence);
    }
}

Rules:

• Metric definitions are sent to the back-end at handshake: keys must be registered at startup and stay stable across releases. Events for unregistered keys are dropped with a one-time warning.
• Ranking metrics are count-based; the dimension string is the ranked value. Never use free user text or user identifiers as a dimension — only catalog-controlled values.

Delivery guarantees

Events are queued in memory and flushed every FlushInterval:

• Transient back-end failures (network errors, HTTP 5xx) → the batch is re-enqueued and retried on the next cycle.
• Permanent rejections (HTTP 4xx) → the batch is dropped with a warning.
• The queue is bounded by MaxQueuedEvents; when full, the newest events are dropped with a warning. On process shutdown, unflushed events are lost — metrics are KPI-grade, not an audit log.

Internal use notice

This package is developed and maintained by Dev4Side for internal use only. It is not published on NuGet.org and is not intended for external consumption.