Compendium.Adapters.LiteLLM 1.0.0-preview.0

Compendium.Adapters.LiteLLM

IAIProvider adapter for the LiteLLM proxy — a self-hostable gateway that exposes 100+ LLM providers behind an OpenAI-compatible REST API.

LiteLLM lets you put one endpoint in front of OpenAI, Anthropic, Bedrock, Azure OpenAI, Vertex / Gemini, Cohere, Mistral, Ollama, and dozens more, plus its own routing primitives: budget caps, virtual keys per team, per-model rate limits, semantic caching, and request tagging. This adapter speaks that endpoint from Compendium.

Why LiteLLM (vs OpenRouter, vs direct provider adapters)

Pick this adapter when you want self-hostable multi-provider routing with first-class observability hooks. Pick compendium-adapter-openrouter when you'd rather not host anything yourself. Pick a direct adapter (compendium-adapter-openai, -anthropic, …) when you need just one provider with no gateway in the path.

Quick start

1. Run a LiteLLM proxy

pip install "litellm[proxy]"

# Single-model dev mode (uses your real provider API keys)
export OPENAI_API_KEY=sk-...
export ANTHROPIC_API_KEY=sk-ant-...
litellm --model gpt-4o-mini --model claude-haiku-4.5
# Listening on http://localhost:4000

For production, point the proxy at a config file with virtual keys, fallbacks, rate limits, and observability. See LiteLLM docs.

2. Install the adapter

dotnet add package Compendium.Adapters.LiteLLM

3. Register the provider

using Compendium.Adapters.LiteLLM.DependencyInjection;

services.AddCompendiumLiteLLM(opt =>
{
    opt.BaseUrl     = "http://localhost:4000";        // your proxy
    opt.VirtualKey  = "sk-litellm-team-eng";          // optional team key
    opt.DefaultModel = "openai/gpt-4o-mini";          // provider-prefixed
});

…or bind from configuration:

{
  "LiteLLM": {
    "BaseUrl": "https://litellm.internal/v1",
    "VirtualKey": "sk-litellm-team-eng",
    "DefaultModel": "openai/gpt-4o-mini",
    "PassThroughHeaders": {
      "x-litellm-tags": "env=prod,team=ai"
    }
  }
}

services.AddCompendiumLiteLLM(builder.Configuration);

4. Use it

public sealed class AssistantService(IAIProvider ai)
{
    public async Task<string> AnswerAsync(string question, CancellationToken ct)
    {
        var result = await ai.CompleteAsync(new CompletionRequest
        {
            Model = "anthropic/claude-haiku-4.5", // route this call to Anthropic
            Messages = new() { Message.User(question) },
            MaxTokens = 512
        }, ct);

        return result.IsSuccess
            ? result.Value.Content
            : $"AI error: {result.Error.Code}";
    }
}

The same IAIProvider instance can hit openai/gpt-4o-mini, anthropic/claude-sonnet-4, bedrock/anthropic.claude-3-haiku, gemini/gemini-1.5-pro — anything your proxy is configured for. Routing happens server-side; your code only changes the model id.

See samples/01-multi-provider-routing for a runnable demo.

Configuration reference

Pass-through headers — common LiteLLM primitives

The pass-through bag is case-insensitive and is silently overridden by the configured ApiKey / VirtualKey for the Authorization header — you can't shoot yourself in the foot by clobbering credentials from configuration.

Tool / function calling

using Compendium.Adapters.LiteLLM.Tools;

var tools = new List<AgentTool>
{
    new("get_weather", "Returns the weather for a city.",
        """{"type":"object","properties":{"city":{"type":"string"}},"required":["city"]}""")
};

var request = new CompletionRequest
{
    Model = "openai/gpt-4o-mini",
    Messages = new() { Message.User("What's the weather in Paris?") }
}.WithTools(tools, toolChoice: "auto");

var result = await ai.CompleteAsync(request);

foreach (var call in result.Value.GetToolCalls())
{
    Console.WriteLine($"{call.ToolName}({call.ArgumentsJson})");
}

Whether a given upstream model honours the tool list depends on the model. The LiteLLM proxy passes the OpenAI-style tools / tool_choice fields straight through.

Structured outputs

using Compendium.Adapters.LiteLLM.StructuredOutputs;

var schema = """
{ "type":"object","properties":{"answer":{"type":"string"}},"required":["answer"] }
""";

var result = await ai.CompleteAsync(
    new CompletionRequest { Model = "openai/gpt-4o-mini", Messages = [...] }
        .WithStructuredOutput(schema, schemaName: "Answer", strict: true));

…or plain JSON mode:

var result = await ai.CompleteAsync(request.WithJsonMode());

Production checklist

• TLS termination — never run the proxy on plain HTTP across a public network. Front it with NGINX / Envoy / your cloud LB.
• Virtual-key rotation — rotate sk-litellm-... keys per team on a schedule. They live in your proxy DB; the adapter just forwards them as Authorization: Bearer.
• Observability — turn on LiteLLM's Langfuse / Helicone / Prometheus hooks server-side. Do NOT set EnableLogging = true in prod — that dumps raw prompts to your app logs.
• Per-tenant tagging — set x-litellm-tags via PassThroughHeaders so per-tenant spend is queryable on the proxy.
• Budget caps — configure proxy-side budget limits per virtual key. The adapter surfaces 402 / AI.InsufficientCredits when a key is over budget.
• Fallbacks — define provider fallbacks (openai/gpt-4o-mini → azure/gpt-4o-mini) in the proxy config; the adapter is unaware and benefits automatically.

Error mapping

Caller cancellation always re-throws TaskCanceledException so the surrounding code knows the shutdown wasn't a server-side fault.

Development

dotnet build  -c Release
dotnet test   -c Release --filter "FullyQualifiedName!~IntegrationTests"

CI gate: ≥ 90 % line coverage. The unit tests use RichardSzalay.MockHttp — no LiteLLM proxy required to run them. The optional integration tests (skipped when LITELLM_BASE_URL is unset) exercise the real proxy.

License

MIT — same as Compendium itself.