LMSupply.Generator 0.65.1

LMSupply.Generator

Local text generation and chat via GGUF (llama-server), with an optional ONNX Runtime GenAI backend.

Features

• Zero-config: Models download automatically from HuggingFace
• GPU Acceleration: CUDA, Vulkan, DirectML, CoreML, Metal
• GGUF Support: Load any GGUF model via llama-server (auto-downloaded)
• Server Pooling: Reuses llama-server instances for fast model switching
• MIT Models: Phi-4 and Phi-3.5 models with no usage restrictions
• Chat Support: Built-in chat formatters for various models

ONNX Runtime GenAI models

This package alone covers GGUF/llama-server. Loading an ONNX Runtime GenAI model (a HuggingFace repo whose name or path indicates ONNX, or a local genai_config.json + model.onnx directory) requires the separate LMSupply.Generator.Onnx package, registered once at startup:

dotnet add package LMSupply.Generator.Onnx

LMSupply.Generator.Onnx.OnnxGeneratorBackend.Register();

Without it, loading an ONNX model throws a NotSupportedException naming the package. The hardware-aware WithDefaultModel()/"auto" path is unaffected either way — it only ever selects ONNX on a discrete non-NVIDIA Windows GPU, and only when this backend is registered; otherwise it falls back to its GGUF selection, same as on any other hardware profile.

Quick Start

using LMSupply.Generator;

// Using the builder pattern
var generator = await TextGeneratorBuilder.Create()
    .WithDefaultModel()
    .BuildAsync();

// Generate text
string response = await generator.GenerateCompleteAsync("What is AI?");
Console.WriteLine(response);

await generator.DisposeAsync();

Downloading ahead of time

// Same resolution as LoadAsync (aliases, default/auto, repo ids), no model load:
var path = await LocalGenerator.DownloadModelAsync("gguf:gemma4-default");

Chat Completion

var messages = new[]
{
    new ChatMessage(ChatRole.System, "You are a helpful assistant."),
    new ChatMessage(ChatRole.User, "Explain quantum computing.")
};

string response = await generator.GenerateChatCompleteAsync(messages);

Available Models

GPU Acceleration

The GGUF/llama-server path auto-downloads the right llama-server binary for the detected GPU (CUDA, Vulkan, Metal) — no extra NuGet package needed. GPU acceleration for ONNX Runtime GenAI models is a LMSupply.Generator.Onnx concern; see that package's README.

Advanced GGUF Tuning

Fine-grained llama-server control (GPU offload, batch size, RoPE/YaRN scaling, KV cache quantization, speculative decoding, LoRA, the startup wait policy — StartupStallTimeout / StartupTimeout, progress-based rather than a fixed deadline — and more) is available via LlamaOptions:

var generator = await TextGeneratorBuilder.Create()
    .WithDefaultModel()
    .WithLlamaOptions(new LlamaOptions
    {
        GpuLayerCount = -1,
        AdditionalArgs = ["--verbose"], // raw llama-server CLI passthrough for flags not modeled above
    })
    .BuildAsync();

Pinning or pre-provisioning the llama-server binary

By default, the llama-server binary itself (not the model) is acquired from an unauthenticated, unpinned GitHub Releases "latest" lookup on first use. For offline, air-gapped, or security-reviewed deployments, ServerUpdateOptions lets a consumer take over that acquisition:

var generator = await TextGeneratorBuilder.Create()
    .WithDefaultModel()
    .WithServerUpdateOptions(new LlamaServerUpdateOptions
    {
        // Pin an exact release tag — a cache hit makes zero network calls; a cache miss downloads
        // exactly this tagged asset instead of "latest". A pinned installation also never
        // auto-checks for or applies a newer version.
        PinnedVersion = "b7898",

        // Or bypass acquisition entirely by pointing at an already-provisioned binary — takes
        // precedence over PinnedVersion when both are set.
        // ServerBinaryPath = "/opt/myapp/bin/llama-server",
    })
    .BuildAsync();

Reuse the same LlamaServerUpdateOptions instance across loads (rather than constructing a new one each time) so they share one background-update timer and state file instead of each spawning its own. EmbedderOptions.ServerUpdateOptions and RerankerOptions.ServerUpdateOptions follow the same shape for the embedding and reranking GGUF paths.

An unpinned load starts on the cached build and fetches a newer one in the background for a later load. UpdateOnWarmup = true makes each load check first and start on the newest build, at the cost of waiting for the check (and the download, when there is one).