Webority.Azure.Telemetry.Functions 0.31.3

Webority.Azure

Shared Azure infrastructure for the Webority product fleet — one canonical, optimized implementation of the Azure plumbing every product used to hand-copy (and drift on).

Package IDs stay Webority.Azure.* for a tidy nuget.org grouping, but the C# namespaces are deliberately Webority.AzureStorage / Webority.AzureTelemetry — not Webority.Azure.*. A Webority.Azure.* namespace would shadow the Azure SDK's own Azure root namespace for any consumer code living under a Webority.* namespace, since C# walks enclosing namespaces and finds Webority.Azure before it ever reaches global Azure.Core/Azure.Identity.

Content stored by hash (Storage)

Large content that never changes once written (a design revision, an AI finding, a transcript, an email body) goes to blob storage, and the SQL row keeps only its address. AddWeborityContentStore() registers IContentStore on the same Azure:Storage settings as the blob service.

• PutAsync(container, bytes, kind) names the blob by the SHA-256 of the uncompressed bytes and stores it Brotli-compressed: <container>/<sha256>.json.br or .txt.br. It returns a ContentAddress with the hash, both sizes and whether this call wrote it; the same bytes put twice write once.
• GetAsync returns the uncompressed bytes, or null when no such blob exists. A missing container throws, as every read path here does.
• Keep the hash and the size on the SQL row. Lifecycle tiering (Cool at 30 days, Cold at 180) is a rule on the storage account, not code.

Two-layer cache (Storage)

AddWeborityHybridCache() registers HybridCache with an in-process layer over the Azure Table cache, replacing any earlier IDistributedCache registration. Defaults: 30 minutes in the shared layer, 5 minutes in memory, 4 MiB largest value.

• A value over 60,000 bytes stays in memory only, because Azure Table caps a property at 64 KiB; writing it removes any smaller value stored under the same key.
• Give every entry tags, so a change clears what it affects with RemoveByTagAsync.
• Content stored by hash never needs a cache expiry: cache it in memory over IContentStore, not here.

Public files in R2

AddWeborityR2() binds Cloudflare:R2 (AccountId, AccessKeyId, SecretAccessKey, the keys from Key Vault in production) and validates them on start. IR2ObjectStore writes to and lists the product's buckets (<product>-media, <product>-media-staging for runtime uploads); links point at the bucket's custom domain (media.<zone>), never the storage endpoint. Team files go through cdn-guard, not this package.

Webority.Azure.NotificationHubs is retired and no longer built or published. Use the Webority.Push package instead (Android via FCM, iOS via APNs). Existing versions on NuGet stay installable.

Auth model (Storage)

One options contract, two modes — set exactly one:

• Azure:Storage:StorageAccountName → Entra / managed identity via a narrowed DefaultAzureCredential chain (ManagedIdentity in Azure, Azure CLI locally — nothing else, so no slow credential probing on constrained plans). One cached credential per process.
• Azure:Storage:ConnectionString → shared key (the local-development default: developers without Azure access use the staging account's connection string).

Blob containers at start-up (Storage)

The write paths (SaveAsync and block staging) create a missing container on their first write to it in each process, which costs a create round trip on that write, several seconds on a cold account. EnsureContainerAsync(containerName) runs the same create-if-missing once, ahead of time, and records the container in the same per-process cache, so the first write to it makes no create call. Leave containerName out to ensure Azure:Storage:Blob:DefaultContainerName.

• The container is created private (PublicAccessType.None); an existing one is left as it is.
• It fails loudly: an invalid container name, an authorization failure or an unreachable account throws, and nothing is cached, so the next write tries the create again.
• Call it on the registered singleton (inject IAzureStorageBlobService). The cache belongs to that instance, so a second instance built by hand does not share it.
• Read paths are unchanged: they still never create a container.

Call it from a hosted service's ExecuteAsync, not StartAsync, so the round trips do not hold up every start (dotnet.md → Startup). An exception thrown there stops the host under the default BackgroundServiceExceptionBehavior.StopHost, so a wrong name or a missing role assignment fails the deploy instead of the first upload. A write that arrives before it finishes creates the container itself, as it always has.

public sealed class BlobContainerStartup(IAzureStorageBlobService blobs) : BackgroundService
{
    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        await blobs.EnsureContainerAsync("reports", stoppingToken);
        await blobs.EnsureContainerAsync("logos", stoppingToken);
    }
}

// Startup.ConfigureServices
services.AddAzureStorageBlob();
services.AddHostedService<BlobContainerStartup>();

Key Vault (app secrets)

Webority.Azure.KeyVault loads an app's secrets from its own Key Vault when the host starts. Wire it with one line in Program.cs, before anything reads configuration:

builder.Configuration.AddWeborityKeyVault(builder.Environment);

The vault's name is the only Key Vault setting a committed file holds:

{
  "KeyVault": {
    "VaultName": "webority-lumex-prod"
  }
}

• On Azure the app signs in as its managed identity; on a laptop it uses the developer's Azure sign-in. Run az login once on a laptop. The identity needs the Key Vault Secrets User role on the vault.
• Startup stops when KeyVault:VaultName is blank, when the vault cannot be read (no role, no access, or no such vault), or when a secret's key is also set in a JSON settings file. A secret lives in Key Vault only, so delete any copy from the settings files.
• Secret names write -- for the : separator: the secret ConnectionStrings--Db loads as ConnectionStrings:Db.
• The secrets load once, at startup. A changed secret takes effect on the next start.

Conventions

• All services register as singletons (Azure SDK clients own their transport and are built for process-wide reuse — see dotnet.md, HttpClient/SDK-client rule).
• Options are validated on start (ValidateDataAnnotations().ValidateOnStart()); a misconfigured host crashes at boot, never at first request. AddWeborityTelemetry is the one that validates earlier still, at registration, because it consumes its own values while wiring the exporter and the span processor, so a startup check would run after the wiring it protects.
• Public type names match the pre-extraction fleet implementations (IAzureStorageBlobService, …), so migrating a product is a using swap plus deleting its local copy.
• A blob download returns an open stream; dispose it (await using) as soon as it is read, or the connection stays held.
• The synchronous IDistributedCache members block a thread while the storage call runs, so they must not be used on a request path.
• A non-seekable upload through SaveAsync buffers up to 32 MiB, so a host should cap its concurrent uploads.

Telemetry collection policy

AddWeborityTelemetry's fleet defaults implement one rule: collect only what is consumed. Application Insights ingestion is billed, and near nobody looks at the Metrics blade or a complete log of every successful outbound call — so none of that ships to Azure Monitor unless a head opts in.

• No metrics export by default (EnableMetrics = false). This suppresses the distro's standard-metrics/performance-counter pipeline (AzureMonitorOptions.EnableStandardMetrics / EnablePerformanceCounters, both true by default upstream) and the ASP.NET Core/HttpClient runtime meters UseAzureMonitor wires in on .NET 8+, via a catch-all dropping view. Set EnableMetrics = true to restore metrics fleet-wide; DroppedMetrics then trims just the known-noisy subset instead of everything.

• Failed-only spans, sampled successes. Failed spans (Status == Error, or any span carrying a recorded exception event) are always kept, and so is any span answering an HTTP status of 400 or above. ASP.NET Core leaves a 4xx at Unset by semantic convention, so without that rule a refusal a caller saw was sampled like a success and three in four never arrived, and a 5xx recorded by instrumentation that sets no span status went the same way. Successful dependency/client spans are always dropped. Successful server/request spans are kept with probability SuccessfulRequestSamplingRatio (default 0.25), decided deterministically per trace so every span belonging to one trace agrees. Outside Production the default is 1.0, so a staging head keeps every request: the environment is read from configuration (environment, then AZURE_FUNCTIONS_ENVIRONMENT, ASPNETCORE_ENVIRONMENT, DOTNET_ENVIRONMENT), none set counts as Production, and a configured ratio still wins.

• Query values are never exported. Every span's url.query, url.full, http.url, http.target, microsoft.request.url and microsoft.dependency.data, and a display name carrying a query, keep their keys with each value replaced by Redacted (/reset?email=Redacted&token=Redacted). These are every attribute the exporter builds a request row's url or a dependency row's data from. It is always on: the Azure Monitor distribution turns the OpenTelemetry instrumentation's own query redaction off, so without it emails, reset tokens and webhook keys sit in the request table for the whole retention. A segment with no = is kept as a key. Put a secret in a header or the body, never in the query.

• Email addresses in exported logs are redacted. The formatted message and every string attribute have each address, %40-encoded ones included, replaced with Redacted-email. An exception's message cannot be rewritten and is exported as it is.

• A log written during a request carries the request's name. Its traces or exceptions row gets the same operation_Name as the requests row (the method and route, else the method and path), so exceptions can be found by endpoint:

  exceptions | summarize count() by operation_Name, type | order by count_ desc
  

• A call the caller abandoned is not a failure. A dependency span cancelled because the request was aborted (its error type and every exception event name OperationCanceledException or TaskCanceledException) is judged as a success. A log of such a cancellation, Azure Identity's LogMsalError log among them, is not exported as an error while the request is aborted or the host is stopping: a log written inside the request's trace is dropped by the trace-based log sampler, and one written outside any trace is exported at Information with no exception. The abandoned request itself is not a failure either: whatever its status code (typically 499) or exception events, it is recorded with success = true and its real resultCode, carries customDimensions["webority.aborted"], and is always kept with webority.sample_rate 1, so a spike of abandoned requests stays visible. An exception event on it still reaches the exceptions table. A cancellation while the request is still open (a timeout) is still a failure. To see the abandoned requests:

  requests | where tobool(customDimensions["webority.aborted"]) | summarize count() by name, resultCode
  

• Nothing is exported without a ServiceName. It defaults to WEBSITE_SITE_NAME. A process outside App Service that sets neither it nor WeborityTelemetry:ServiceName starts normally, exports nothing rather than writing rows as unknown_service:dotnet, and logs one warning at startup naming the setting. WeborityTelemetryOptions.Enabled then reads false.

• A 404 that matched no endpoint is sampled like a success (SampleUnmatchedRoute404, default true). Scanners probing paths that do not exist are most of them. Its span has no http.route, so it is recorded with success = true, its real resultCode and customDimensions["webority.expected_failure"], exactly like a marked expected failure. A 404 answered by a matched endpoint keeps its route and stays a failure. A head that calls UseStatusCodePagesWithReExecute ends every request on the error page's route, so the package adds a middleware (an IStartupFilter, outside everything the head registers) that reads the original request's endpoint when the error page starts its response. Its unmatched 404s are then sampled the same way, with no change in the head; an error page that writes no body leaves them failures. Nothing is added when the option is off, and a Functions worker has no pipeline for it.

• A request carries the signed-in account's id as enduser.id, which Application Insights shows as user_AuthenticatedId. It is read from the ClaimTypes.NameIdentifier claim of an authenticated user and written only when it is a positive 32-bit integer, so an email, a phone number, a GUID or any other text is never exported. A head whose authentication keeps the id in sub without mapping it to NameIdentifier gets no id.

• The exporter's customer SDK stats are off. They write item counts into the component every 15 minutes through a meter provider of their own, which EnableMetrics does not reach, and the exporter reads only the process environment variable APPLICATIONINSIGHTS_SDKSTATS_DISABLED. The package sets it to true when it wires the exporter and the variable is unset; a head that sets it itself keeps its value.

• "Always kept" is literal, because head sampling is switched off. A head sampler decides at span start, before a status code, a response or an exception exists, so it cannot tell a failure from a success and nothing downstream can bring back a trace it dropped. The package therefore pins the exporter to always-on (SamplingRatio to 1.0 and TracesPerSecond to null, both needed because the distro defaults the second to 5.0 and lets it win) and makes every trim after the span ends, where the status is known. SuccessfulRequestSamplingRatio is the one ratio the package has.

  • The cost, stated plainly. Every span is created and fully recorded before it can be judged, which is CPU and allocation a head sampler would have saved. Application Insights also stops extrapolating counts: the exporter stamps one sample rate on every item, taken from its own head ratio, and reads no per-span rate, so itemCount is 1 on every row and count() reads successful requests as the kept fraction while failure counts are exact. Count with the sample-rate dimension below instead.

• Every exported span carries the rate it was kept at, as customDimensions["webority.sample_rate"]: 1 for a span kept whole (a failure, a refusal, a kept dependency), 1/ratio for a successful request kept by sampling (4 at the default 0.25). The true number of requests a row stands for is that value, so:

  requests
  | where timestamp > ago(1d)
  | summarize requests = round(sum(todouble(customDimensions["webority.sample_rate"]))),
              failed = countif(success == false)
      by bin(timestamp, 1h)
  

  Rows exported before this release have no such dimension and count as nothing in sum(), so start a trend at the release. The exporter's own itemCount cannot carry this: it takes one sample rate from the exporter options for every item.

• A request can mark its 4xx as an expected outcome. A business-rule refusal answered with 400 is not a failure, but Application Insights counts every request at 400 or above as failed. A head calls MarkExpectedFailure where it produces such an answer:

  using Webority.AzureTelemetry;
  
  httpContext.MarkExpectedFailure();
  

  The request is then recorded with success = true and its real resultCode, carries customDimensions["webority.expected_failure"], and is sampled at SuccessfulRequestSamplingRatio like any success. The mark only ever softens a 4xx: a request that goes on to a 5xx, records an exception or carries an Error status is still a failure, and the dimension is removed. With telemetry off the call does nothing. Setting WeborityTelemetry:TreatConflictAndUnprocessableAsExpected to true treats every 409 and 422 the same way without a call; 400 is left out on purpose, because the framework also answers 400 for malformed input and failed antiforgery checks. To see the refusals themselves:

  requests
  | where tobool(customDimensions["webority.expected_failure"])
  | summarize refusals = round(sum(todouble(customDimensions["webority.sample_rate"]))) by name, resultCode
  

  • The one hole the package cannot close: do not set OTEL_TRACES_SAMPLER or OTEL_TRACES_SAMPLER_ARG on a Webority head. The exporter reads both from the environment and either one installs a head sampler underneath the package, which puts back exactly the trace loss the pin above removes, silently and with no signal in the app's own config. Nothing in the fleet sets them today.
  • The second hole, and it is not closable here either: the trim is not scoped to Azure Monitor. A span is dropped by clearing its Recorded flag, which every exporter in the pipeline reads, so a head that chains its own exporter onto the returned builder loses exactly the same spans, silently. OpenTelemetry has no per-exporter filter and the Azure Monitor distribution exposes no hook for a processor inside its own pipeline. A head that needs the complete span stream somewhere else should give that exporter its own provider rather than chaining it here.

• Opt-in keeps for successful spans (both off by default, so a head that sets neither sees no change). AlwaysKeepSources lists ActivitySource names whose spans are always kept, and SlowDependencyThresholdMs keeps a successful client or internal span lasting at least that many milliseconds (1 to 600000). A configured AlwaysKeepSources array replaces the default empty list. Each kept span is billed ingestion, so list only sources someone reads and set the threshold above normal latency.

  "WeborityTelemetry": {
    "AlwaysKeepSources": [ "Experimental.Microsoft.Agents.AI", "Experimental.Microsoft.Extensions.AI" ],
    "SlowDependencyThresholdMs": 500,
    "SlowRequestThresholdMs": 1000
  }
  

• Slow requests are kept (SlowRequestThresholdMs, off by default). A successful server span at least that many milliseconds long is kept whole, and so is the server span of any trace in which a slow dependency was kept, for up to two minutes after that dependency ended. Both are recorded at rate 1 (webority.sample_rate), not at the sampled rate, so counts of kept requests stay exact and the sampled remainder estimates only the requests under the threshold. A SignalR connection (http.long_running) is excluded.

• SQL statement text (CaptureSqlText, on by default). A kept SQL dependency carries its statement as the dependency's data. The instrumentation has already replaced every literal with ?, and the text is cut at 2,000 characters. Parameter values are never exported. The dependency's name is the statement summary (for example SELECT Projects) rather than SQL: <database>. A stored procedure keeps its procedure name as data. CaptureSqlText: false keeps the summary and drops the text.

• Kept metrics (KeptMetrics, off by default). An allow list of metric names exported although EnableMetrics is false; every other instrument is dropped. A name starting with dotnet. subscribes the in-box System.Runtime meter. The exporter sends metrics every 60 seconds: a counter as the change in that minute, an up-down counter as its level. Combining it with EnableMetrics: true fails at registration.

• Runtime resource row off. The exporter adds one resource item to every trace batch, which Application Insights shows as _APPRESOURCEPREVIEW_ in AppMetrics. Telemetry sets OTEL_DOTNET_AZURE_MONITOR_ENABLE_RESOURCE_METRICS to false unless the head sets it itself. The item repeats the service.* attributes already on every row, so nothing is lost.

• Which logs reach Azure Monitor is decided only by the head's Logging:LogLevel settings.

These are all options on WeborityTelemetryOptions, overridable per head via the configure delegate or the WeborityTelemetry configuration section — this is the fleet default, not a hard rule.

The distro's own AzureMonitor configuration section is not a supported surface here. The package calls the UseAzureMonitor(Action<AzureMonitorOptions>) overload, and only the parameterless overload registers the internal options type that reads that section, so a head putting an AzureMonitor block in its appsettings finds it silently inert. That is deliberate rather than an oversight: the section carries SamplingRatio and TracesPerSecond, so honouring it would hand a head back the head sampler this package pins off. Configure the exporter through the configure delegate.

Tenant on a request

A head calls SetTelemetryTenant where it resolves the tenant, and the request row carries customDimensions["webority.tenant"]. Only a positive id is written.

using Webority.AzureTelemetry;

httpContext.SetTelemetryTenant(accountId);

Client dimensions on a request

A request span carries customDimensions["webority.client_kind"] from the Client-Kind header and customDimensions["webority.app_build"] from the App-Build header. Each is trimmed and cut to 64 characters, and a blank header writes no dimension.

Background job runs

A job wraps each run in WeborityJobRun.Start. The run is one always-kept request row named after the job, tagged customDimensions["webority.job"]. Call WeborityJobRun.Fail(run, exception) in the catch block to mark it failed. A head referencing Webority.Azure.Telemetry has it already; a project below the web layer references Webority.Azure.Telemetry.Jobs alone, and the head's telemetry exports its runs.

using var run = WeborityJobRun.Start("OutboxDrain");

Destructive jobs

A job that deletes data or removes access registers its limits once and checks its whole plan before it removes anything. A refused verdict removes nothing; ThrowIfRefused ends the run failed with the refusal as its message. The limits are code defaults; <section>:Mode=DryRun in configuration stops removals without a deploy, and <section>:MaxPerRun or <section>:MaxShareOfPopulation raises a limit after a review. Invalid limits fail the host on start.

services.AddDestructiveRun("GoogleWorkspaceSync", "GoogleWorkspace:DestructiveRun", o => { o.Mode = DestructiveRunMode.Live; o.MaxPerRun = 10; o.MaxShareOfPopulation = 0.1; });
var verdict = guard.Check(new DestructiveRunPlan("GoogleWorkspaceSync", accounts.Count, toDelete));

Telemetry for Functions hosts

An Azure Functions isolated-worker host calls AddWeborityFunctionsTelemetry from Webority.Azure.Telemetry.Functions instead of AddWeborityTelemetry, with the same arguments and the same environment gate:

using Webority.AzureTelemetry.Functions;

if (!environment.IsDevelopment())
{
    services.AddWeborityFunctionsTelemetry(configuration);
}

It does everything AddWeborityTelemetry does, then sets the worker capability WorkerApplicationInsightsLoggingEnabled. That tells the Functions host the worker exports its own logs, so the host stops exporting them a second time as Function.<Name>.User rows. ConfigureFunctionsApplicationInsights sets the same capability but needs the classic Application Insights SDK, which is why it cannot be used here.

• The capability follows WeborityTelemetry:Enabled. A host that switches telemetry off keeps the Functions host's own log export, rather than losing both.
• It returns the same builder, so a host adds its own processors by chaining: .WithTracing(tracing => tracing.AddProcessor<MyProcessor>()). The collection policy above, including the warning about chaining a second exporter, applies unchanged.
• Worker logs and exceptions carry the invocation's operation id. While telemetry is exported the package applies UseFunctionsWorkerDefaults from Microsoft.Azure.Functions.Worker.OpenTelemetry, which listens to the worker's invocation ActivitySource, so each function runs inside an activity parented on the host's trace. That span is internal, not a request, so the host's own request row is not duplicated, and a successful one is dropped like any successful dependency. The worker defaults also name the service after the process's WEBSITE_SITE_NAME or the entry assembly; the package pins ServiceName again after them, so the configured name still wins.
• Reference the package directly from the Functions project. The isolated worker drops packages that arrive only transitively from its deployed closure, so check that Webority.Azure.Telemetry.Functions.dll and Microsoft.Azure.Functions.Worker.OpenTelemetry.dll are listed in the published *.deps.json.

Releasing

Version lives once in Directory.Build.props (<VersionPrefix>), the fleet's standard version-of-record for a package repo. Push to main publishes all four packages to public nuget.org through the fleet's shared workflow (.github/workflows/publish.yml, a thin caller into Webority/gh-actions). No code checks run in CI; build and test locally before pushing (CLAUDE.md). Work happens on development; a release is a PR development → main merged with a merge commit, per the fleet git conventions.

version-check.yml gates that PR on two things, both failing the PR rather than warning:

1. Directory.Build.props <VersionPrefix> has increased against main. Without it the publish skips every package and exits green having shipped nothing.
2. The release paperwork names the new version: CHANGELOG.md carries a ## [<version>] - DD-MMM-YYYY heading, and its ## [Unreleased] section is empty.

So cutting a release is two edits in one commit: the version and the changelog heading.