Themis.Inference.Gguf 2.0.3

Themis.Inference.Gguf

Themis.Inference.Gguf is an optional .NET 10 package for RLCD GGUF decision inference, GGUF model packing, and conversational inference through an external llama.cpp server. It does not contain model weights, start the server, or add ONNX Runtime dependencies.

Install it when using the extracted APIs:

dotnet add package Themis.Inference.Gguf

Applications using GGUF functionality should reference this package directly and use Themis.Inference.Gguf.Runtime. The main Themis.Inference package no longer carries a transitive GGUF dependency.

llama.cpp chat

The chat provider implements IInferenceProvider and uses the OpenAI-compatible /v1/chat/completions endpoint. Configure the endpoint and the model alias used by the server:

using Themis.Inference.Gguf.Runtime;

services.AddThemisLlamaCppChat(options =>
{
    options.ChatCompletionsEndpoint =
        new Uri("http://localhost:8080/v1/chat/completions");
    options.DefaultModel = "Qwen3-0.6B-GGUF";
});

The provider maps Themis chat messages, sampling options, tools, JSON response formats, usage, and cancellation to the endpoint contract. It does not convert arbitrary GGUF output into typed RLCD decisions.

Pinned Qwen3 proof-of-concept

The Qwen3-0.6B-GGUF model card declares Apache-2.0. This example pins the model repository revision and the llama.cpp source commit; check the upstream terms and host requirements before redistribution or deployment. Model weights are downloaded separately and are not part of this package:

$model = Join-Path $env:LOCALAPPDATA 'Themis\models\Qwen3-0.6B-Q8_0.gguf'
Invoke-WebRequest `
  'https://huggingface.co/Qwen/Qwen3-0.6B-GGUF/resolve/23749fefcc72300e3a2ad315e1317431b06b590a/Qwen3-0.6B-Q8_0.gguf' `
  -OutFile $model

# Build llama.cpp from commit 620d2225cc436ab0b001e3200e590e3b286ef9ab
llama-server --model $model --alias Qwen3-0.6B-GGUF --port 8080

The provider is tested with deterministic fake HTTP responses in the normal test suite. Live server/model testing is opt-in and does not establish accelerator execution or support for every model, server build, or device. Configure THEMIS_LLAMACPP_CHAT_ENDPOINT and optionally THEMIS_LLAMACPP_CHAT_MODEL to run that integration test:

dotnet test .\tests\Themis.Tests\Themis.Tests.csproj `
  --artifacts-path .\artifacts `
  --filter "TestCategory=LlamaCppChatIntegration"

The Qwen model card and the pinned llama.cpp server documentation are available at Qwen/Qwen3-0.6B-GGUF and llama.cpp server README.