AICentral.Logging.PIIStripping 0.20.6-pullrequest0150-0003

AICentral.Logging.PIIStripping

An Asynchronous logger for use with (AICentral)[www.github.com/microsoft/aicentral]. This logger leverages the Azure Text Analytics API to strip prompts and responses of PII data.

Configuration

dotnet add package AICentral.Logging.PIIStripping

builder.Services.AddAICentral(
    builder.Configuration,
    additionalComponentAssemblies:
    [
        typeof(PIIStrippingLogger).Assembly,
    ]);

{
  "AICentral": {
    "GenericSteps": [
      {
        "Type": "PIIStrippingLogger",
        "Name": "pii-stripping-logger",
        "Properties": {
          "UseManagedIdentities": "true|false",
          "UserAssignedManagedIdentityId": "<id-or-blank>",
          "StorageQueueConnectionString": "<storage-connection-string>",
          "QueueName": "queue-to-write-to",
          "TextAnalyticsEndpoint": "<text-analytics-uri>",
          "TextAnalyticsKey": "<text-analytics-key-if-if-not-using-managed-identities>",
          "CosmosDatabase": "cosmos-database-to-log-to",
          "CosmosContainer": "container-to-log-to",
          "CosmosConnectionString": "<cosmos-connection-string-if-not-using-managed-identities>",
          "StorageUri": "<uri-to-storage-if-using-managed-identities>",
          "PIIStrippingDisabled": false
        }
      }
    ]    
  }
}

How does it work?

This logger is a wrapper around the AICentral logger that uses the Text Analytics API to strip PII data from the prompts and responses before logging them to Cosmos. Log statements are initially written to an Azure Queue, and then read from the queue by a separate background service, that uses the Text Analytics API to strip PII data. The stripped log statements are then written to Cosmos.

The Container needs to have a partition key specified by '/id'.