AICentral 0.19.0-pullrequest0136-0014

This is a prerelease version of AICentral.
There is a newer version of this package available.
See the version list below for details.
dotnet add package AICentral --version 0.19.0-pullrequest0136-0014
                    
NuGet\Install-Package AICentral -Version 0.19.0-pullrequest0136-0014
                    
This command is intended to be used within the Package Manager Console in Visual Studio, as it uses the NuGet module's version of Install-Package.
<PackageReference Include="AICentral" Version="0.19.0-pullrequest0136-0014" />
                    
For projects that support PackageReference, copy this XML node into the project file to reference the package.
<PackageVersion Include="AICentral" Version="0.19.0-pullrequest0136-0014" />
                    
Directory.Packages.props
<PackageReference Include="AICentral" />
                    
Project file
For projects that support Central Package Management (CPM), copy this XML node into the solution Directory.Packages.props file to version the package.
paket add AICentral --version 0.19.0-pullrequest0136-0014
                    
#r "nuget: AICentral, 0.19.0-pullrequest0136-0014"
                    
#r directive can be used in F# Interactive and Polyglot Notebooks. Copy this into the interactive tool or source code of the script to reference the package.
#addin nuget:?package=AICentral&version=0.19.0-pullrequest0136-0014&prerelease
                    
Install AICentral as a Cake Addin
#tool nuget:?package=AICentral&version=0.19.0-pullrequest0136-0014&prerelease
                    
Install AICentral as a Cake Tool

AI Central

AI Central gives you control over your AI services.

  • Lightweight out-the-box token logging using Open Telemetry
  • Intelligent Routing
  • Custom consumer OAuth2 authorisation
  • Fallback AI service
  • Round Robin AI services
  • Lowest Latency AI service
  • Circuit breakers, and backoff-retry over downstream AI services
  • Request based and Token based rate limiting
  • Prompt and usage logging
    • Works for streaming endpoints as-well as non streaming
  • Additional Open Telemetry Metrics and Traces

Configuration

See docs on Github for more details.

Minimal

This sample produces a AI-Central proxy that

  • Listens on a hostname of your choosing
  • Proxies directly through to a back-end Open AI server
  • Can be accessed using standard SDKs
  • Outputs open-telemetry metrics to capture usage information
{
  "AICentral": {
    "Endpoints": [
      {
        "Type": "AzureOpenAIEndpoint",
        "Name": "openai-1",
        "Properties": {
          "LanguageEndpoint": "https://<my-ai>.openai.azure.com",
          "AuthenticationType": "ApiKey",
          "ApiKey": "<key>"
        }
      }
    ],
    "EndpointSelectors": [
      {
        "Type": "SingleEndpoint",
        "Name": "default",
        "Properties": {
          "Endpoint": "openai-1"
        }
      }
    ],
    "AuthProviders": [
      {
        "Type": "ApiKey",
        "Name": "apikey",
        "Properties": {
          "Clients": [
            {
              "ClientName": "Consumer-1",
              "Key1": "<random-key>",
              "Key2": "<random-key>"
            }
          ]
        }
      }
    ],
    "Pipelines": [
      {
        "Name": "OpenAIPipeline",
        "Host": "mypipeline.mydomain.com",
        "EndpointSelector": "default",
        "AuthProvider": "apikey",
        "OpenTelemetryConfig": {
          "AddClientNameTag": true,
          "Transmit": true
        }
      }
    ]
  }
}

Full example

This pipeline will:

  • Present an Azure Open AI, and an Open AI downstream as a single upstream endpoint
    • maps the incoming deployment Name "GPT35Turbo0613" to the downstream Azure Open AI deployment "MyGptModel"
    • maps incoming Azure Open AI deployments to Open AI models
  • Present it as an Azure Open AI style endpoint
  • Protect the front-end by requiring an AAD token issued for your own AAD application
  • Put a local Asp.Net core rate-limiting policy over the endpoint
  • Emit Open Telemetry to be picked up by your OTel collector
  • Add rich logging to Azure monitor
    • Logs quota, client caller information (IP and identity name), and in this case the Prompt but not the response.
  • Publish the client-name as a tag in Open Telemetry
{
  "AICentral": {
    "Endpoints": [
      {
        "Type": "AzureOpenAIEndpoint",
        "Name": "openai-priority",
        "Properties": {
          "LanguageEndpoint": "https://<my-ai>.openai.azure.com",
          "AuthenticationType": "Entra|EntraPassThrough|ApiKey",
          "MaxConcurrency": 10,
          "ModelMappings": {
            "Gpt35Turbo0613": "MyGptModel"
          }
        }
      },
      {
        "Type": "OpenAIEndpoint",
        "Name": "openai-fallback",
        "Properties": {
          "LanguageEndpoint": "https://api.openai.com",
          "ModelMappings": {
            "Gpt35Turbo0613": "gpt-3.5-turbo",
            "Ada002Embedding": "text-embedding-ada-002"
          },
          "ApiKey": "<my-api-key>",
          "Organization": "<optional-organisation>"
        }
      }
    ],
    "AuthProviders": [
      {
        "Type": "Entra",
        "Name": "simple-aad",
        "Properties": {
          "Entra": {
            "ClientId": "<my-client-id>",
            "TenantId": "<my-tenant-id>",
            "Instance": "https://login.microsoftonline.com/",
            "Audience": "<custom-audience>"
          }
        }
      }
    ],
    "EndpointSelectors": [
      {
        "Type": "Prioritised",
        "Name": "my-endpoint-selector",
        "Properties": {
          "PriorityEndpoints": ["openai-1"],
          "FallbackEndpoints": ["openai-fallback"]
        }
      }
    ],
    "GenericSteps": [
      {
        "Type": "AspNetCoreFixedWindowRateLimiting",
        "Name": "token-rate-limiter",
        "Properties": {
          "LimitType": "PerConsumer|PerAICentralEndpoint",
          "MetricType": "Tokens",
          "Options": {
            "Window": "00:01:00",
            "PermitLimit": 1000
          }
        }
      },
      {
        "Type": "AspNetCoreFixedWindowRateLimiting",
        "Name": "window-rate-limiter",
        "Properties": {
          "LimitType": "PerConsumer|PerAICentralEndpoint",
          "MetricType": "Requests",
          "Options": {
            "Window": "00:00:10",
            "PermitLimit": 100
          }
        }
      },
      {
        "Type": "AzureMonitorLogger",
        "Name": "azure-monitor-logger",
        "Properties": {
          "WorkspaceId": "<workspace-id>",
          "Key": "<key>",
          "LogPrompt": true,
          "LogResponse": false,
          "LogClient": true
        }
      },
      {
        "Type": "BulkHead",
        "Name": "bulk-head",
        "Properties": {
          "MaxConcurrency": 20
        }
      }
    ],
    "Pipelines": [
      {
        "Name": "MyPipeline",
        "Host": "prioritypipeline.mydomain.com",
        "EndpointSelector": "my-endpoint-selector",
        "AuthProvider": "simple-aad",
        "Steps": [
          "window-rate-limiter",
          "bulk-head",
          "azure-monitor-logger"
        ],
        "OpenTelemetryConfig": {
          "AddClientNameTag": true,
          "Transmit": true
        }
      }
    ]
  }
}

Product Compatible and additional computed target framework versions.
.NET net8.0 is compatible.  net8.0-android was computed.  net8.0-browser was computed.  net8.0-ios was computed.  net8.0-maccatalyst was computed.  net8.0-macos was computed.  net8.0-tvos was computed.  net8.0-windows was computed.  net9.0 was computed.  net9.0-android was computed.  net9.0-browser was computed.  net9.0-ios was computed.  net9.0-maccatalyst was computed.  net9.0-macos was computed.  net9.0-tvos was computed.  net9.0-windows was computed.  net10.0 was computed.  net10.0-android was computed.  net10.0-browser was computed.  net10.0-ios was computed.  net10.0-maccatalyst was computed.  net10.0-macos was computed.  net10.0-tvos was computed.  net10.0-windows was computed. 
Compatible target framework(s)
Included target framework(s) (in package)
Learn more about Target Frameworks and .NET Standard.

NuGet packages

This package is not used by any NuGet packages.

GitHub repositories

This package is not used by any popular GitHub repositories.

Version Downloads Last updated
0.21.4 1,509 4 months ago
0.21.4-pullrequest0155-0003 67 4 months ago
0.21.3 84 4 months ago
0.21.3-pullrequest0155-0002 66 5 months ago
0.21.1 607 5 months ago
0.21.1-pullrequest0153-0003 52 5 months ago
0.21.0 376 6 months ago
0.21.0-pullrequest0152-0005 68 6 months ago
0.21.0-pullrequest0152-0004 65 6 months ago
0.21.0-pullrequest0152-0003 60 6 months ago
0.21.0-pullrequest0152-0002 63 6 months ago
0.20.7 235 7 months ago
0.20.7-pullrequest0151-0002 67 7 months ago
0.20.6 373 8 months ago
0.20.6-pullrequest0150-0003 59 8 months ago
0.20.5 278 9 months ago
0.20.5-pullrequest0148-0004 75 9 months ago
0.20.5-pullrequest0148-0003 68 9 months ago
0.20.5-pullrequest0148-0002 89 9 months ago
0.20.3 122 10 months ago
0.20.3-pullrequest0146-0005 63 10 months ago
0.20.3-pullrequest0146-0004 64 10 months ago
0.20.2 88 10 months ago
0.20.2-pullrequest0145-0004 55 10 months ago
0.20.0 114 8/17/2024
0.20.0-pullrequest0143-0013 79 8/17/2024
0.20.0-pullrequest0143-0012 81 8/17/2024
0.20.0-pullrequest0143-0011 79 8/17/2024
0.20.0-pullrequest0143-0010 79 8/17/2024
0.20.0-pullrequest0143-0009 78 8/17/2024
0.19.6-pullrequest0143-0008 87 8/17/2024
0.19.5 81 8/7/2024
0.19.5-pullrequest0141-0002 58 8/7/2024
0.19.4 85 8/1/2024
0.19.4-pullrequest0140-0003 60 8/1/2024
0.19.4-pullrequest0139-0001 59 7/29/2024
0.19.3 79 7/29/2024
0.19.2 90 7/29/2024
0.19.2-pullrequest0138-0004 60 7/29/2024
0.19.1 80 7/29/2024
0.19.1-pullrequest0137-0003 65 7/29/2024
0.19.0 76 7/29/2024
0.19.0-pullrequest0136-0014 57 7/29/2024
0.19.0-pullrequest0136-0013 57 7/29/2024
0.19.0-pullrequest0136-0012 147 7/29/2024
0.19.0-pullrequest0136-0011 59 7/29/2024
0.19.0-pullrequest0136-0010 62 7/29/2024
0.19.0-pullrequest0136-0009 72 7/27/2024
0.19.0-pullrequest0136-0008 66 7/27/2024
0.19.0-pullrequest0136-0007 66 7/27/2024
0.18.6 96 7/15/2024
0.18.6-pullrequest0135-0002 63 7/15/2024
0.18.5 81 7/15/2024
0.18.5-pullrequest0134-0003 59 7/15/2024
0.18.4 83 7/15/2024
0.18.4-pullrequest0133-0009 63 7/15/2024
0.18.4-pullrequest0133-0008 62 7/15/2024
0.18.4-pullrequest0133-0007 64 7/15/2024
0.18.3 90 7/12/2024
0.18.3-pullrequest0132-0005 63 7/12/2024
0.18.2 82 7/12/2024
0.18.2-pullrequest0131-0003 69 7/11/2024
0.18.2-pullrequest0130-0003 62 7/11/2024
0.18.1 83 7/10/2024
0.18.1-pullrequest0129-0007 61 7/10/2024
0.18.0 127 7/10/2024
0.18.0-pullrequest0125-0014 62 7/10/2024
0.18.0-pullrequest0125-0013 68 7/10/2024
0.18.0-pullrequest0125-0012 57 7/10/2024
0.18.0-pullrequest0125-0011 59 7/10/2024
0.18.0-pullrequest0125-0010 62 7/10/2024
0.18.0-pullrequest0125-0009 73 7/10/2024
0.17.2-pullrequest0125-0008 63 7/10/2024
0.17.2-pullrequest0125-0007 70 7/10/2024
0.17.2-pullrequest0125-0006 64 7/10/2024
0.17.2-pullrequest0125-0005 65 7/10/2024
0.17.2-pullrequest0125-0004 70 7/10/2024
0.17.2-pullrequest0125-0003 66 7/9/2024
0.17.1 83 7/9/2024
0.17.1-pullrequest0124-0005 67 7/9/2024
0.17.0 88 7/8/2024
0.17.0-pullrequest0123-0024 54 7/8/2024
0.17.0-pullrequest0123-0023 57 7/8/2024
0.17.0-pullrequest0123-0021 69 7/8/2024
0.17.0-pullrequest0123-0019 62 7/8/2024
0.17.0-pullrequest0123-0018 52 7/8/2024
0.17.0-pullrequest0123-0017 66 7/8/2024
0.17.0-pullrequest0123-0015 61 7/8/2024
0.17.0-pullrequest0123-0014 56 7/8/2024
0.17.0-pullrequest0123-0013 62 7/8/2024
0.17.0-pullrequest0123-0012 57 7/8/2024
0.17.0-pullrequest0123-0011 63 7/8/2024
0.17.0-pullrequest0123-0009 68 7/6/2024
0.17.0-pullrequest0123-0008 62 7/6/2024
0.17.0-pullrequest0123-0007 99 7/6/2024
0.16.23 90 6/12/2024
0.16.23-pullrequest0118-0005 68 6/12/2024
0.16.23-pullrequest0118-0004 61 6/12/2024
0.16.23-pullrequest0118-0003 61 6/12/2024
0.16.23-pullrequest0118-0002 60 6/12/2024
0.16.21 143 5/29/2024
0.16.21-pullrequest0114-0002 67 5/29/2024
0.16.20 89 5/28/2024
0.16.20-pullrequest0112-0002 62 5/28/2024
0.16.19 94 5/13/2024
0.16.19-pullrequest0110-0004 75 5/13/2024
0.16.19-pullrequest0110-0003 64 5/13/2024
0.16.17 112 4/22/2024
0.16.17-pullrequest0108-0005 77 4/19/2024
0.16.17-pullrequest0108-0004 68 4/19/2024
0.16.17-pullrequest0108-0003 70 4/19/2024
0.16.16 102 4/16/2024
0.16.16-pullrequest0107-0003 81 4/16/2024
0.16.15 100 4/16/2024
0.16.15-pullrequest0106-0004 69 4/16/2024
0.16.15-pullrequest0106-0003 66 4/16/2024
0.16.15-pullrequest0106-0002 78 4/16/2024
0.16.14 162 4/16/2024
0.16.13 112 3/23/2024
0.16.13-pullrequest0103-0003 84 3/23/2024
0.16.12 95 3/23/2024
0.16.12-pullrequest0102-0006 76 3/22/2024
0.16.11 97 3/21/2024
0.16.11-pullrequest0101-0004 66 3/21/2024
0.16.11-pullrequest0101-0003 73 3/21/2024
0.16.11-pullrequest0101-0002 67 3/21/2024
0.16.9 105 3/16/2024
0.16.9-pullrequest0098-0008 74 3/16/2024
0.16.9-pullrequest0098-0007 76 3/16/2024
0.16.8 123 3/15/2024
0.16.8-pullrequest0097-0002 71 3/15/2024
0.16.7 101 3/10/2024
0.16.7-pullrequest0095-0002 81 3/10/2024
0.16.6 183 3/9/2024
0.16.6-pullrequest0094-0003 72 3/9/2024
0.16.6-pullrequest0094-0002 68 3/9/2024
0.16.2 107 3/5/2024
0.16.2-pullrequest0089-0009 76 3/5/2024
0.16.1 106 3/3/2024
0.16.1-pullrequest0088-0003 73 3/2/2024
0.16.1-pullrequest0088-0002 62 3/2/2024
0.16.0 151 2/28/2024
0.16.0-pullrequest0087-0010 66 2/28/2024
0.16.0-pullrequest0087-0009 70 2/28/2024
0.16.0-pullrequest0087-0007 81 2/28/2024
0.16.0-pullrequest0087-0006 73 2/28/2024
0.16.0-pullrequest0087-0005 66 2/28/2024
0.16.0-pullrequest0087-0004 77 2/28/2024
0.16.0-pullrequest0087-0003 71 2/28/2024
0.15.2 94 2/26/2024
0.15.2-pullrequest0086-0007 69 2/26/2024
0.15.2-pullrequest0086-0006 60 2/26/2024
0.15.1 102 2/23/2024
0.15.1-pullrequest0085-0006 69 2/23/2024
0.15.0 103 2/22/2024
0.15.0-pullrequest0084-0004 72 2/22/2024
0.15.0-pullrequest0084-0003 103 2/22/2024
0.14.1-pullrequest0084-0002 84 2/22/2024
0.14.0 115 2/21/2024
0.14.0-pullrequest0083-0011 95 2/8/2024
0.14.0-pullrequest0083-0010 74 2/8/2024
0.14.0-pullrequest0083-0009 76 2/8/2024
0.13.7 122 2/8/2024
0.13.7-pullrequest0082-0002 82 2/8/2024
0.13.6-pullrequest0081-0006 74 2/8/2024
0.13.5 127 2/7/2024
0.13.5-pullrequest0080-0002 75 2/7/2024
0.13.4 94 2/7/2024
0.13.4-pullrequest0079-0003 68 2/7/2024
0.13.1 104 2/1/2024
0.13.1-pullrequest0076-0003 78 2/1/2024
0.13.1-pullrequest0076-0002 76 2/1/2024
0.13.0 121 2/1/2024
0.13.0-pullrequest0075-0005 75 2/1/2024
0.13.0-pullrequest0075-0004 70 2/1/2024
0.12.2-pullrequest0075-0003 71 2/1/2024
0.12.1 98 2/1/2024
0.12.1-pullrequest0074-0002 72 2/1/2024
0.12.0 95 2/1/2024
0.12.0-pullrequest0073-0002 71 2/1/2024
0.11.0 97 1/31/2024
0.11.0-pullrequest0072-0011 75 1/31/2024
0.11.0-pullrequest0072-0010 71 1/31/2024
0.11.0-pullrequest0072-0008 69 1/31/2024
0.10.0 98 1/31/2024
0.10.0-pullrequest0070-0005 76 1/29/2024
0.10.0-pullrequest0070-0004 72 1/29/2024
0.9.1-pullrequest0069-0001 71 1/29/2024
0.9.0 93 1/29/2024
0.9.0-pullrequest0069-0002 69 1/29/2024
0.8.7-pullrequest0068-0001 71 1/29/2024
0.8.6 96 1/29/2024
0.8.4 114 1/19/2024
0.8.4-pullrequest0066-0009 72 1/19/2024
0.8.4-pullrequest0066-0008 64 1/19/2024
0.8.4-pullrequest0066-0007 73 1/19/2024
0.8.4-pullrequest0066-0005 66 1/19/2024
0.8.4-pullrequest0066-0004 74 1/19/2024
0.8.4-pullrequest0066-0003 69 1/19/2024
0.8.3 93 1/19/2024
0.8.3-pullrequest0065-0003 70 1/19/2024
0.8.3-pullrequest0065-0002 71 1/19/2024
0.8.1 94 1/19/2024
0.8.1-pullrequest0063-0013 69 1/19/2024
0.8.1-pullrequest0063-0011 72 1/19/2024
0.7.16-pullrequest0063-0010 78 1/19/2024
0.7.15 101 1/18/2024
0.7.15-pullrequest0062-0002 73 1/18/2024
0.7.12 108 1/16/2024
0.7.12-pullrequest0059-0006 77 1/16/2024
0.7.10 102 1/15/2024
0.7.10-pullrequest0059-0007 72 1/16/2024
0.7.10-pullrequest0058-0004 67 1/15/2024
0.7.10-pullrequest0057-0002 72 1/15/2024
0.7.9 101 1/15/2024
0.7.9-pullrequest0057-0003 69 1/15/2024
0.7.7 99 1/15/2024
0.7.7-pullrequest0056-0003 82 1/15/2024
0.7.4 101 1/15/2024
0.7.4-pullrequest0055-0012 75 1/15/2024
0.7.4-pullrequest0055-0011 76 1/15/2024
0.7.4-pullrequest0055-0010 80 1/15/2024
0.7.4-pullrequest0055-0009 79 1/15/2024
0.7.4-pullrequest0055-0008 75 1/15/2024
0.7.4-pullrequest0055-0007 77 1/15/2024
0.7.4-pullrequest0054-0005 76 1/15/2024
0.7.4-pullrequest0053-0004 88 1/15/2024
0.7.4-pullrequest0052-0002 69 1/15/2024
0.7.2 98 1/15/2024
0.7.2-pullrequest0051-0013 79 1/15/2024
0.7.2-pullrequest0050-0010 69 1/15/2024
0.7.1 107 1/13/2024
0.7.0 105 1/13/2024
0.7.0-pullrequest0046-0079 87 1/13/2024
0.6.12 124 1/3/2024
0.6.12-pullrequest0045-0018 73 1/13/2024
0.6.12-pullrequest0042-0010 77 1/12/2024
0.6.12-pullrequest0041-0007 75 1/12/2024
0.6.12-pullrequest0039-0002 81 1/3/2024
0.6.10 120 12/22/2023
0.6.10-pullrequest0038-0007 99 1/2/2024
0.6.10-pullrequest0038-0006 84 1/2/2024
0.6.10-pullrequest0038-0005 86 1/2/2024
0.6.8 114 12/21/2023
0.6.6 109 12/18/2023
0.6.6-pullrequest0035-0007 89 12/21/2023
0.6.6-pullrequest0035-0006 91 12/21/2023
0.6.6-pullrequest0035-0005 81 12/21/2023
0.6.3 108 12/18/2023
0.6.3-pullrequest0034-0018 85 12/18/2023
0.6.1 103 12/15/2023
0.6.1-pullrequest0029-0008 89 12/15/2023
0.6.1-pullrequest0028-0006 95 12/15/2023
0.6.1-pullrequest0027-0004 72 12/15/2023
0.5.3 111 12/11/2023
0.5.3-pullrequest0025-0049 90 12/14/2023
0.5.3-pullrequest0025-0047 87 12/14/2023
0.5.3-pullrequest0025-0043 85 12/13/2023
0.5.3-pullrequest0022-0038 84 12/13/2023
0.5.1 96 12/7/2023
0.5.0 102 12/6/2023
0.4.3 110 12/4/2023
0.4.2 100 12/4/2023
0.4.1 110 12/4/2023
0.3.10 118 11/22/2023
0.3.10-pullrequest0022-0008 85 11/27/2023
0.3.10-pullrequest0021-0005 86 11/22/2023
0.3.9 95 11/20/2023
0.3.8 107 11/9/2023
0.3.8-pullrequest0020-0005 81 11/9/2023
0.3.6 101 11/9/2023
0.3.5 103 11/9/2023
0.3.4 98 11/8/2023
0.3.3 101 11/8/2023
0.3.1 102 11/8/2023
0.3.1-pullrequest0018-0004 92 11/8/2023
0.2.1 136 11/2/2023
0.2.1-pullrequest0017-0050 100 11/8/2023
0.2.1-pullrequest0016-0048 100 11/8/2023
0.2.1-pullrequest0015-0045 90 11/8/2023
0.2.1-pullrequest0014-0035 102 11/7/2023
0.2.1-pullrequest0013-0033 94 11/7/2023
0.2.1-pullrequest0012-0031 95 11/7/2023
0.2.1-pullrequest0011-0024 94 11/6/2023
0.2.1-pullrequest0010-0015 109 11/2/2023
0.2.1-pullrequest0009-0011 96 11/2/2023
0.2.1-pullrequest0008-0009 99 11/2/2023
0.2.1-pullrequest0007-0007 104 11/2/2023
0.2.1-pullrequest0006-0005 102 11/2/2023
0.2.1-pullrequest0005-0001 90 11/2/2023
0.1.0 126 10/27/2023
0.1.0-pullrequest0004-0024 99 11/1/2023
0.1.0-pullrequest0003-0022 100 11/1/2023
0.1.0-pullrequest0002-0019 98 11/1/2023
0.1.0-pullrequest0001-0017 102 11/1/2023
0.1.0-pullrequest0001-0010 102 10/30/2023

releasenotes.md