D4S.AgentMetrics 1.3.2

D4S.AgentMetrics

Internal NuGet package for Dev4Side AI agents built on .NET 10.

It handles, transparently:

• Handshake — verifies at startup that the agent is registered and active in the D4S back-end, reports a rejection with the reason to check, and retries until it succeeds.
• Metric definition registration — sends all metric descriptors (default + custom) to the back-end.
• Business metric tracking — exposes IMetricWriter.TrackAsync with a bounded in-memory queue.
• Interaction tracking — measures total_interactions, avg_response_time_ms and error_rate automatically. Pick the tracker with the Transport setting: an HTTP middleware for REST/MCP agents (scoped to the endpoints you choose) or a Microsoft 365 Agents SDK middleware for Teams / custom engine agents (counts only real user messages).
• AI token tracking — an OpenTelemetry listener harvests gen_ai.usage.* from any Microsoft.Extensions.AI chat pipeline (D4S.Agent works out of the box, zero code), dimensioned by model so the back-end can price it — including the cached-input share, behind one flag.
• Periodic flush — drains the queue and POSTs the event batch to the back-end, with retry on transient failures.
• Heartbeat — sends periodic liveness pings to the back-end.

Internal use only. This package is not intended for public distribution.

Prerequisites

The agent must already be registered in the D4S back-end (manually or via the marketplace onboarding process) — this package only verifies that the registration exists.

Everything is fail-soft: if the AgentMetrics configuration is missing, or the back-end is unreachable, or the handshake is rejected, metric tracking is disabled and the agent starts and runs normally. This makes it safe to keep the wiring always on (see Quick start), including in local dev where no back-end is configured. Fail-soft does not mean silent, though — a configuration that looks complete but is rejected is reported at Error and retried; see Startup and recovery.

Installation

dotnet add package D4S.AgentMetrics

Quick start

This is the wiring used by the MAF365Agent template — two lines of code, everything else is configuration:

// Program.cs
builder.Services.AddAgentMetrics(options =>
{
    builder.Configuration.GetSection("AgentMetrics").Bind(options);
    // Project-specific business metrics go here (see "Tracking custom metrics"):
});

// After UseRouting, before the endpoint mappings:
app.UseAgentMetricsTracking();

// appsettings.json
{
  "AgentMetrics": {
    // Unique per environment: never let two deployments share one registration.
    "AgentId": "my-agent",
    "TenantId": "customer-tenant",
    "BaseUrl": "https://my-agent.azurewebsites.net",
    "D4SApiUrl": "https://d4s-api.azurewebsites.net",
    "D4SApiKey": "", // secret: set per environment (user secrets / app settings), never commit
    // Only measure the business endpoints — keeps health probes, platform warmup pings and
    // scanner noise out of total_interactions / avg_response_time_ms:
    "TrackedPathPrefixes": [ "/api/messages", "/mcp" ],
    // Only the verb that submits a question — excludes status-polling GETs or CORS preflight
    // OPTIONS that might share the same path, so total_interactions counts exactly one event
    // per question:
    "TrackedHttpMethods": [ "POST" ]
  }
}

Leave the section empty (or omit D4SApiUrl/D4SApiKey) to run with metrics disabled — the wiring is a clean no-op in that case.

Configuration

Startup and recovery

At startup, before the host serves traffic, the package performs the handshake and one of three things happens:

The middle row is worth spelling out because it is the one misconfiguration that looks configured. A reference that resolves is substituted by the platform, so the process receives the secret and this code cannot even tell a reference from a plain key. A reference that fails to resolve — the secret does not exist, or the app's managed identity cannot read it — is passed through as the literal @Microsoft.KeyVault(SecretUri=...) text. That is non-empty, so it would be sent as the API key and rejected with 401 on every attempt.

Note that the value shown in the portal (and by az webapp config appsettings list) is always the reference, resolved or not: it is the configured value, not the one injected into the process, so it tells you nothing about whether resolution succeeded. What does:

az rest --method get --url "https://management.azure.com/subscriptions/<sub>/resourceGroups/<rg>/providers/Microsoft.Web/sites/<app>/config/configreferences/appsettings?api-version=2022-03-01" --query "value[].{name:name, status:properties.status, details:properties.details}"

Rejections are diagnosed rather than just reported — 401 points at the key, 403 at an AgentId/TenantId that belongs to another registration, 404 at a registration that does not exist.

Nothing has to be restarted when a retry finally succeeds: the flush and heartbeat loops re-check on every tick, and the token listener is attached from the start (it drops what it harvests while tracking is off). Only the first rejection is logged at Error; the retries are Debug, so a permanently misconfigured agent produces one loud line instead of one per minute forever.

Interaction tracking: HTTP vs Bot

total_interactions and avg_response_time_ms must count one real user question and measure from question received to complete answer produced. How to capture that depends on how the agent receives questions, so the Transport setting selects the tracker. Only the selected one is active; the other is a no-op even if wired, so a shared startup template can register both.

Transport = Http (default) — REST / MCP agents

The agent receives questions as HTTP requests. app.UseAgentMetricsTracking() (see Quick start) measures each tracked request; scope it with TrackedPathPrefixes + TrackedHttpMethods so only the question-submitting call is counted.

Transport = Bot — Teams / custom engine agents (Microsoft 365 Agents SDK)

Microsoft 365 Agents SDK agents receive every interaction as an activity on a single /api/messages endpoint, so an HTTP filter cannot tell a real message from a conversationUpdate, typing or invoke. The bot middleware (AgentMetricsBotMiddleware, a Microsoft.Agents.Builder.IMiddleware) inspects Activity.Type and counts only message activities; it times the whole turn (message in → reply sent). Just set Transport to Bot:

// appsettings.json
"AgentMetrics": { /* ...required options... */ "Transport": "Bot" }

No code change is needed in Program.cs or the agent: at startup AddAgentMetrics(...) attaches the middleware to the adapter the container resolves as IChannelAdapter — which is the same instance the CloudAdapter created by AddAgent<TAgent>() serves requests with — before the web server starts accepting requests.

Registering the middleware in DI is not what does it, despite being the SDK's documented contract. CloudAdapter receives its middleware through an IMiddleware[] constructor parameter, and Microsoft.Extensions.DependencyInjection resolves IEnumerable<T> but never T[], so that parameter always falls back to its null default and the adapter starts with an empty middleware set. Versions up to 1.2.1 relied on that contract and therefore reported no interaction metric at all for every Transport = Bot agent. BotMiddlewareAttachTests pins both halves of this down.

Only an adapter the container cannot hand out as IChannelAdapter — a hand-built one, never registered — needs an explicit attach:

// Only for an adapter that DI cannot resolve; anything resolvable is instrumented automatically.
// Safe to call even when it is not needed: an adapter that already carries the middleware is skipped,
// so interactions are never counted twice.
adapter.UseAgentMetricsBotTracking(services); // active only when Transport = Bot

With Transport = Bot the package also registers and emits conversations — a distinct count of Activity.Conversation.Id (one event per turn), so the dashboard can show distinct sessions and derive turns-per-session as total_interactions ÷ conversations. It is not registered for Transport = Http (no conversation-id producer, so it would be a permanently empty tile).

Copilot Studio (low-code, Microsoft-hosted) agents run no code of yours, so this package cannot instrument them — interaction metrics there must come from Dataverse analytics / Power Automate.

Default profile metrics

These six metrics are always included automatically — no extra code required. UseProfile(AgentProfile.Default) is optional but recommended for clarity.

error_rate is a Percentage: the back-end computes it as count(error_rate) ÷ count(total_interactions). The descriptor carries DenominatorMetricKey = "total_interactions" so the back-end knows what to divide by — without it the metric renders with no value.

Every token event also carries the model name as its dimension (TrackTokensByModel, on by default), because unit prices differ per model. The scalar totals are unchanged — summing across all dimensions gives the same number as before.

Two more metrics are added automatically only under a specific configuration:

• conversations (DistinctCount) when Transport = Bot (see above).
• cached_prompt_tokens_total (Sum) when CachedTokenTrackingEnabled is set — see below.

avg_confidence_score was removed from the default profile in 1.1.0: it had no automatic producer and rendered as a permanently empty tile. Register it as a custom scalar in agents that actually compute a confidence score.

avg_tokens_per_interaction was removed in 1.2.0: it was averaged per LLM call, not per interaction (a single turn can span several LLM calls), so the label was misleading. Raw token counts remain, and since 1.3.0 they are priceable in the D4S back-end — see Token cost.

AI token tracking

Token metrics are collected automatically by listening to the OpenTelemetry GenAI spans emitted by Microsoft.Extensions.AI pipelines wrapped with UseOpenTelemetry(...):

• D4S.Agent / MAF365Agent hosts: zero configuration — the D4S.Agent chat and guardrail clients are already instrumented with the source names the listener observes by default.
• Other MEAI hosts: set TokenTelemetrySources to your instrumented source names (e.g. Experimental.Microsoft.Extensions.AI for the MEAI default). Do not include agent-level sources that re-emit aggregated usage, or tokens are double-counted.
• The listener reads both the current semantic conventions (gen_ai.usage.input_tokens / gen_ai.usage.output_tokens) and the legacy names (prompt_tokens / completion_tokens).
• Each event is dimensioned by model, taken from gen_ai.response.model and falling back to gen_ai.request.model.

For hosts that use the Azure AI Foundry ChatCompletionsClient directly (no MEAI pipeline), the legacy explicit wrapper is still available:

builder.Services.AddTrackedChatClient(
    new ChatCompletionsClient(endpoint, new AzureKeyCredential(apiKey)),
    model: "gpt-4o");

Token cost: per-model and cached tokens

Token counts only become a cost once the back-end knows which model produced them and how many input tokens were served from cache (providers bill cached input at a fraction of the normal rate). All four metrics come from the same OpenTelemetry span listener, so there is no pipeline wiring to get right and nothing that can double-count:

Enabling cached-token tracking

options.CachedTokenTrackingEnabled = true;   // inside AddAgentMetrics, or via appsettings

That is all — no chat-pipeline change. Microsoft.Extensions.AI 9.8 puts the cached share on the same chat span as the other usage tags, so the listener already has it.

Why it is still opt-in rather than on by default. The back-end reads the presence of the cached_prompt_tokens_total definition as "this agent knows its cached share" and stops marking the cost as an estimate. Registering it for every agent would make that claim on behalf of providers whose instrumentation never reports the tag, and their cost would silently be computed as if nothing had ever been cached. Turning the flag on is therefore a statement about the provider, which only the host can make.

Two consequences worth knowing:

• A span without the tag reports nothing for that call, rather than a zero. Missing means unknown; a zero would tell the back-end the input was billed entirely at full rate.
• The cached count is a subset of the input tokens, so it is never folded into total_tokens_total.

How the back-end turns this into a cost

cached_prompt_tokens_total is a subset of prompt_tokens_total, not an additional amount, so the input side has to be split before pricing:

cost = (prompt_tokens_total - cached_prompt_tokens_total) x input_rate(model)
     +  cached_prompt_tokens_total                       x cached_input_rate(model)
     +  completion_tokens_total                          x output_rate(model)

• Never price total_tokens_total — it is prompt + completion, kept for volume dashboards only. Multiplying it by any blended rate gives a wrong number.
• With cached tracking off, cached_prompt_tokens_total is absent, the first term collapses to prompt_tokens_total x input_rate and the cost is overstated by exactly the cached discount.
• The dimension value is the model as reported by the provider (e.g. gpt-4o-2024-11-20), or the Azure OpenAI deployment name when the response carries no model. A model with no entry in the price table must be surfaced as unpriced, not treated as free — otherwise cost silently drops when an agent switches deployment.
• Reasoning tokens need no separate metric: providers already bill them inside the output tokens.

Tracking custom metrics

Register the project's business metrics at startup, then record events wherever they happen:

// Registration (inside AddAgentMetrics):
options.AddRanking("top_exams", "Esami più richiesti", dimensionName: "exam_name", topN: 20);
options.AddScalar("teams_messages", "Domande da Teams");                     // Count
options.AddScalar("avg_confidence_score", "Confidence medio", AggregationType.Average);

// Then inject IMetricWriter wherever you need it:
public class MyService(IMetricWriter metrics)
{
    public async Task HandleAsync(string examName, double confidence)
    {
        await metrics.TrackAsync("teams_messages");                          // pure count
        await metrics.TrackAsync("top_exams", dimension: examName);          // ranking increment
        await metrics.TrackAsync("avg_confidence_score", value: (decimal)confidence);
    }
}

Rules:

• Metric definitions are sent to the back-end at handshake: keys must be registered at startup and stay stable across releases. Events for unregistered keys are dropped with a one-time warning.
• Ranking metrics are count-based; the dimension string is the ranked value. Never use free user text or user identifiers as a dimension — only catalog-controlled values.

Delivery guarantees

Events are queued in memory and flushed every FlushInterval:

• Transient back-end failures (network errors, HTTP 5xx) → the batch is re-enqueued and retried on the next cycle.
• Permanent rejections (HTTP 4xx) → the batch is dropped with a warning.
• The queue is bounded by MaxQueuedEvents; when full, the newest events are dropped with a warning. On process shutdown, unflushed events are lost — metrics are KPI-grade, not an audit log.

Internal use notice

This package is developed and maintained by Dev4Side for its own agents: it is written against the D4S back-end contract and is not intended for external consumption.