LocalAI.Embedder 0.4.0

LocalAI

.NET libraries for on-device AI inference with zero external API dependencies. Run embeddings, reranking, and more entirely on your local machine with automatic GPU acceleration.

Features

• Zero Configuration: Works out of the box with sensible defaults
• Automatic GPU Acceleration: Detects and uses CUDA, DirectML, or CoreML automatically
• HuggingFace Compatible: Downloads models from HuggingFace Hub with standard caching
• Cross-Platform: Windows, Linux, macOS support
• Production Ready: Thread-safe, async-first, IAsyncDisposable support

Packages

Package	Description	NuGet
LocalAI.Embedder	Text embeddings with sentence-transformers models
LocalAI.Reranker	Semantic reranking with cross-encoder models

Quick Start

Text Embeddings

using LocalAI.Embedder;

// Load a model (downloads automatically on first use)
await using var model = await LocalEmbedder.LoadAsync("default");

// Generate embeddings
float[] embedding = await model.EmbedAsync("Hello, world!");

// Batch processing
float[][] embeddings = await model.EmbedBatchAsync(new[]
{
    "First document",
    "Second document",
    "Third document"
});

// Calculate similarity
float similarity = model.CosineSimilarity(embeddings[0], embeddings[1]);

Semantic Reranking

using LocalAI.Reranker;

// Load a reranker model
await using var reranker = await LocalReranker.LoadAsync("default");

// Rerank documents by relevance to a query
var results = await reranker.RerankAsync(
    query: "What is machine learning?",
    documents: new[]
    {
        "Machine learning is a subset of artificial intelligence...",
        "The weather today is sunny and warm...",
        "Deep learning uses neural networks..."
    },
    topK: 2
);

foreach (var result in results)
{
    Console.WriteLine($"[{result.Score:F4}] {result.Document}");
}

Available Models

Embedder Models

Reranker Models

GPU Acceleration

GPU acceleration is automatic when available:

// Auto-detect best provider (default)
var options = new EmbedderOptions { Provider = ExecutionProvider.Auto };

// Force specific provider
var options = new EmbedderOptions { Provider = ExecutionProvider.Cuda };
var options = new EmbedderOptions { Provider = ExecutionProvider.DirectML }; // Windows
var options = new EmbedderOptions { Provider = ExecutionProvider.CoreML };   // macOS
var options = new EmbedderOptions { Provider = ExecutionProvider.Cpu };

For GPU support, install the appropriate ONNX Runtime package:

# NVIDIA CUDA
dotnet add package Microsoft.ML.OnnxRuntime.Gpu

# Windows DirectML (AMD, Intel, NVIDIA)
dotnet add package Microsoft.ML.OnnxRuntime.DirectML

# macOS CoreML
dotnet add package Microsoft.ML.OnnxRuntime.CoreML

Model Caching

Models are cached following HuggingFace Hub standard:

• Default: ~/.cache/huggingface/hub
• Override with HF_HUB_CACHE, HF_HOME, or XDG_CACHE_HOME environment variables
• Or specify directly: new EmbedderOptions { CacheDirectory = "/path/to/cache" }

Requirements

• .NET 8.0 or later
• Windows, Linux, or macOS

Documentation

• Embedder Guide
• Reranker Guide
• API Reference
• Performance Tips

License

MIT License - see LICENSE for details.

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

Release Process

Releases are automated via GitHub Actions when Directory.Build.props is updated:

1. Update the <Version> in Directory.Build.props
2. Commit and push to main
3. CI automatically publishes all packages to NuGet and creates a GitHub release

Requires NUGET_API_KEY secret configured in GitHub repository settings.