Starlings.Generation.OnnxGenAI 0.5.0

dotnet add package Starlings.Generation.OnnxGenAI --version 0.5.0
                    
NuGet\Install-Package Starlings.Generation.OnnxGenAI -Version 0.5.0
                    
This command is intended to be used within the Package Manager Console in Visual Studio, as it uses the NuGet module's version of Install-Package.
<PackageReference Include="Starlings.Generation.OnnxGenAI" Version="0.5.0" />
                    
For projects that support PackageReference, copy this XML node into the project file to reference the package.
<PackageVersion Include="Starlings.Generation.OnnxGenAI" Version="0.5.0" />
                    
Directory.Packages.props
<PackageReference Include="Starlings.Generation.OnnxGenAI" />
                    
Project file
For projects that support Central Package Management (CPM), copy this XML node into the solution Directory.Packages.props file to version the package.
paket add Starlings.Generation.OnnxGenAI --version 0.5.0
                    
#r "nuget: Starlings.Generation.OnnxGenAI, 0.5.0"
                    
#r directive can be used in F# Interactive and Polyglot Notebooks. Copy this into the interactive tool or source code of the script to reference the package.
#:package Starlings.Generation.OnnxGenAI@0.5.0
                    
#:package directive can be used in C# file-based apps starting in .NET 10 preview 4. Copy this into a .cs file before any lines of code to reference the package.
#addin nuget:?package=Starlings.Generation.OnnxGenAI&version=0.5.0
                    
Install as a Cake Addin
#tool nuget:?package=Starlings.Generation.OnnxGenAI&version=0.5.0
                    
Install as a Cake Tool

Starlings.Generation.OnnxGenAI

Text generation on ONNX Runtime GenAI 0.17.1 behind a Starling adapter, with an owned decode loop. Prompt ids come from the owned tokenizer (Starlings.Tokenizers); GenAI's tokenizer and template engine are not used. The API may still change.

  • GenAIModel.Load(directory, options): explicit provider (CPU, or CUDA with TF32 off by default), no silent fallback, config overlays (threads, pad id) recorded in AppliedOverlays, reference-counted disposal, package version check. On CUDA the native calls of concurrent sequences are serialized by default (SerializeNativeCalls): concurrent GenAI 0.17.1 generators on the CUDA provider produced wrong tokens.
  • TextGenerator: greedy or seeded sampling, MaxNewTokens, stop token ids (never part of the completion) and stop strings (text cut before the match), streaming with UTF-8-safe incremental detokenization (IncrementalDetokenizer), optional chunked prefill, cancellation checked before the prefill, between prefill chunks and between decode steps (not inside a step), usage (PromptTokens, CompletionTokens excluding the stop token) and stage timings. Synchronous by design: each step is blocking native work.
  • StopSequenceMatcher: the stop-string matcher TextGenerator uses, for applications with their own decode loop. Push each detokenized piece and emit what Push returns; text that could still start a stop string is held back, so no emitted piece contains part of one. When Matched is set, the text is cut before the match and the request ends; otherwise emit Flush() at the end. Ordinal comparison; the earliest match wins, the longest at the same start. One instance per request.
  • Experimental, opt-in: guided decoding (GenerationOptions.Guidance): a JSON Schema, Lark grammar or regex constraint passed to GenAI's llguidance integration (GeneratorParams.SetGuidance), optionally with fast-forward tokens.
  • Advanced: GenAISequence (append ids, step, step with forced tokens, copy the last logits); it also implements the asynchronous IGenerationSequence seam of Starlings.Generation.Abstractions (each call completes synchronously).

The request and result types (GenerationOptions, SamplingOptions, GenerationGuidance, GenerationResult, GenerationUpdate, GenerationUsage, TokenLogprob, GenerationContextLengthException), StopSequenceMatcher and IncrementalDetokenizer are in Starlings.Generation.Abstractions (namespace Starlings.Generation), which this package references; they moved there unchanged in behaviour. Code in the namespace Starlings.Generation.OnnxGenAI sees them without a using; other code adds using Starlings.Generation;.

Most applications use it through a model package such as Starlings.Models.Qwen35.OnnxGenAI.

Use

using Starlings.Generation;
using Starlings.Generation.OnnxGenAI;
using Starlings.Tokenizers;

using var model = GenAIModel.Load(modelDirectory, new GenAIModelOptions { Provider = GenAIExecutionProvider.Cpu });
var tokenizer = Tokenizer.FromFile(tokenizerPath);
var generator = new TextGenerator(model, tokenizer);
var promptIds = tokenizer.Encode(promptText, addSpecialTokens: false).Ids;   // the model package renders the template
GenerationResult result = generator.Generate(promptIds, new GenerationOptions { MaxNewTokens = 256 });
Console.WriteLine($"{result.Text} ({result.FinishReason}, {result.Usage.CompletionTokens} tokens)");
  • GenAIModelOptions: Provider (default Cpu), CudaDeviceId (default 0), UseTf32 (default false), IntraOpThreads (null: the CPU quota or affinity count in a constrained container, else the GenAI default), PadTokenId (null keeps the model's), ConfigOverlays (advanced JSON overlays on genai_config.json), SerializeNativeCalls (null: serialized on CUDA, concurrent on CPU). The loaded GenAIModel reports ModelType, ContextLength, VocabularySize, EosTokenIds, PadTokenId, AppliedOverlays, ProviderOptions, SerializesNativeCalls, and GuidanceDirectory and GuidanceMissingFiles (guided decoding).
  • GenerationOptions: MaxNewTokens (default 512, at least 1), StopSequences (non-empty strings), AdditionalStopTokenIds, Sampling (null: greedy), PrefillChunkTokens (null: one prefill), ReturnLogprobs and TopLogprobs (opt-in per-token log probabilities, top 0 to 20; experimental), Guidance (null: unconstrained; experimental; not combinable with ReturnLogprobs).
  • SamplingOptions: Temperature (default 1.0, greater than 0), TopK (default 50), TopP (default 1.0, in (0, 1]), Seed (required, non-negative). The same seed, prompt, settings, model and device reproduce the same tokens.
  • GenerationResult: Text, TokenIds, FinishReason (Stop, StopSequence or Length; cancellation throws instead), StopTokenId, StopSequence, Usage (PromptTokens, CompletionTokens, TotalTokens), Timings (Prefill, Decode), Logprobs (with ReturnLogprobs). GenerateStreaming yields GenerationUpdates (Text, TokenIds, Logprobs); the last has IsFinal and Result.
  • TokenLogprob (TokenId, Text, Logprob, TopLogprobs of TopTokenLogprob): one per completion token (the stop token has none), the natural-log probability from the log-softmax of the raw logits of the step that selected the token (before temperature, top-k and top-p). After a step, ONNX Runtime GenAI 0.17.1 reports the logits that selected the returned token, because it runs the model on that token only at the next step; the decode loop reads them right after the step. Each step then copies one logits row (vocabulary size x 4 bytes, about 1 MB for Qwen3.5).
  • Guided decoding (experimental, opt-in): GenerationGuidance.JsonSchema(schema), Lark(grammar) or Regex(pattern) as GenerationOptions.Guidance. GenAI masks the tokens the constraint does not allow before it selects each one, and allows only an end-of-sequence id once the constraint is complete. GenAI builds the token table from tokenizer.json and tokenizer_config.json in the model directory (GenAIModel.GuidanceFiles; model.tokenizer_dir when set). GenAIModel.GuidanceMissingFiles lists the ones missing, and a guided request on such a directory fails with NotSupportedException before any native call. WithFastForward(maxTokensPerStep) turns on GenAI's forced tokens: when the constraint leaves one continuation, GenAI appends it without a decode step per token. The loop returns those tokens in order, before the step's selected token, through the same token limit, stop ids, detokenization and stop sequences. GenAI does not check the sequence's maximum length for forced tokens, so the loop sizes the sequence for MaxNewTokens plus maxTokensPerStep (counted in the context-length check), and a step that appends more fails the request. The bound is the caller's to derive from the constraint. Without Guidance the loop is unchanged.
  • Lifetime: dispose GenAIModel, GenAISequence and streaming enumerators (foreach does). As a safety net, a sequence or model that becomes unreachable undisposed (for example an abandoned GenerateStreaming enumerator) is released after a garbage collection: GenAI's own finalizers on these objects are suppressed, and the adapter's finalizer queues the same release as Dispose (generator under the native-call lock, then the model reference) to a thread-pool thread, never the finalizer thread. An abandoned request therefore no longer pins the model, but its native memory stays allocated until a collection finds it.
  • GenAIRuntimeInfo: PinnedVersion (0.17.1), and ManagedVersion and NativePackage read from the application's .deps.json (null in single-file or AOT hosts).
  • Exceptions: GenAIRuntimeUnavailableException (no native library, or packages other than 0.17.1), GenAIProviderUnavailableException (the provider cannot be used; no fallback), GenAIModelException (GenAI rejected the directory, its configuration or a step), GenerationContextLengthException (prompt plus MaxNewTokens exceed the context length; PromptTokens, MaxNewTokens, ContextLength).

Maturity

Maturity is per profile, generated from the registry manifests by model-pipeline catalog (do not edit this table by hand). Gates, evidence and capabilities per profile: the model catalog.

Profile Maturity Platforms
qwen2.5-0.5b-instruct/ortgenai-fp32-cpu experimental linux-x64 cpu: experimental; windows-x64 cpu: untested
qwen3-0.6b/ortgenai-fp32-cpu experimental linux-x64 cpu: experimental; windows-x64 cpu: untested
qwen3.5-0.8b/ortgenai-fp16x-cuda stable linux-x64 cuda: supported; windows-x64 cuda: untested
qwen3.5-0.8b/ortgenai-fp32x-cpu stable linux-x64 cpu: supported; windows-x64 cpu: untested
qwen3.5-0.8b/ortgenai-fp32x-cuda stable linux-x64 cuda: supported; windows-x64 cuda: untested
qwen3.5-0.8b/ortgenai-int4-cpu experimental linux-x64 cpu: experimental; linux-x64 cuda: unsupported; windows-x64 cpu: untested

Native runtime

The package references Microsoft.ML.OnnxRuntimeGenAI.Managed (exactly 0.17.1; it asks for Microsoft.Extensions.AI.Abstractions 9.8.0 or later, which Starlings.Generation.Abstractions lifts to 10.10.1), Starlings.Generation.Abstractions and the owned tokenizer. The application references exactly one native package of the same version:

Deployment Package Notes
CPU Microsoft.ML.OnnxRuntimeGenAI 0.17.1 Brings Microsoft.ML.OnnxRuntime 1.30.0. Natives for linux-x64, linux-arm64, win-x64, win-arm64, osx-arm64; exercised on linux-x64 only.
CUDA Microsoft.ML.OnnxRuntimeGenAI.Cuda 0.17.1 Brings Microsoft.ML.OnnxRuntime.Gpu 1.30.0. linux-x64 and win-x64 natives (CUDA 13 build); the CUDA runtime and cuBLAS are not shipped. Exercised on linux-x64 only.

Native package check

The package ships MSBuild targets in buildTransitive/, so every executable or test project that references it, directly or through Starlings.Models.Qwen35.OnnxGenAI or Starlings.Models.QwenChat.OnnxGenAI (the definition packages Starlings.Models.Qwen35 and Starlings.Models.QwenChat do not bring it), checks its package graph after restore: warning STARLINGS003 when no GenAI native package is present (loading a model would throw GenAIRuntimeUnavailableException), error STARLINGS004 when both Microsoft.ML.OnnxRuntimeGenAI and Microsoft.ML.OnnxRuntimeGenAI.Cuda are. Class libraries and browser projects are not checked. Turn it off with <StarlingsOnnxRuntimeGenAINativeCheck>false</StarlingsOnnxRuntimeGenAINativeCheck>. CUDA build recipe and Linux GPU dependencies: native runtimes guide.

The model directory must come from the onnxruntime-genai builder of the same version. Documentation: ai-dotnet/README.md.

Product Compatible and additional computed target framework versions.
.NET net10.0 is compatible.  net10.0-android was computed.  net10.0-browser was computed.  net10.0-ios was computed.  net10.0-maccatalyst was computed.  net10.0-macos was computed.  net10.0-tvos was computed.  net10.0-windows was computed. 
Compatible target framework(s)
Included target framework(s) (in package)
Learn more about Target Frameworks and .NET Standard.

NuGet packages (2)

Showing the top 2 NuGet packages that depend on Starlings.Generation.OnnxGenAI:

Package Downloads
Starlings.Models.Qwen35.OnnxGenAI

Qwen/Qwen3.5-0.8B (revision 2fc06364) chat generation in-process on ONNX Runtime GenAI 0.17.1: Qwen35ChatModel over the model definition Starlings.Models.Qwen35 (the pinned chat template rendered in C#, the tool-call parser, prompt ids from the owned tokenizer Starlings.Tokenizers profile qwen35-bpe-chat: no Microsoft.ML.Tokenizers, no GenAI tokenizer) and Starling's decode loop (greedy or seeded sampling, stop tokens and strings, streaming, cancellation between steps, usage). Ships the asset lock of the ONNX Runtime GenAI bundle, not the weights. Profiles qwen3.5-0.8b/ortgenai-fp32x (CPU; CUDA with TF32 off) and ortgenai-fp16x (CUDA), Linux x64, Stable. The application adds Microsoft.ML.OnnxRuntimeGenAI or Microsoft.ML.OnnxRuntimeGenAI.Cuda. Maturity is per model profile (Stable, Preview or Experimental): see the README, generated from the model registry.

Starlings.Models.QwenChat.OnnxGenAI

Qwen/Qwen3-0.6B (revision c1899de2, thinking disabled) and Qwen/Qwen2.5-0.5B-Instruct (revision 7ae55760) text chat generation in-process on ONNX Runtime GenAI 0.17.1: QwenChatModel over the model definitions Starlings.Models.QwenChat (the pinned chat templates rendered in C#, the tool-call parser, prompt ids from the owned tokenizer Starlings.Tokenizers: no Microsoft.ML.Tokenizers, no GenAI tokenizer) and Starling's decode loop (greedy, stop tokens and strings, streaming, cancellation between steps, usage). Ships no weights: QwenChatModelAssets downloads the profile's bundle explicitly from Starling's public Hugging Face hosting (or it is built locally from the pinned original), and Load verifies every file against the profile's digests. Profiles qwen3-0.6b/ortgenai-fp32-cpu and qwen2.5-0.5b-instruct/ortgenai-fp32-cpu, Linux x64. The application adds Microsoft.ML.OnnxRuntimeGenAI. Maturity is per model profile (Stable, Preview or Experimental): see the README, generated from the model registry.

GitHub repositories

This package is not used by any popular GitHub repositories.

Version Downloads Last Updated
0.5.0 149 11/3/2026
0.4.0 115 11/3/2026
0.3.0 248 10/19/2026
0.2.0 356 10/17/2026
0.1.0-preview.6 115 10/14/2026