Starlings.Generation.OnnxGenAI
0.5.0
dotnet add package Starlings.Generation.OnnxGenAI --version 0.5.0
NuGet\Install-Package Starlings.Generation.OnnxGenAI -Version 0.5.0
<PackageReference Include="Starlings.Generation.OnnxGenAI" Version="0.5.0" />
<PackageVersion Include="Starlings.Generation.OnnxGenAI" Version="0.5.0" />
<PackageReference Include="Starlings.Generation.OnnxGenAI" />
paket add Starlings.Generation.OnnxGenAI --version 0.5.0
#r "nuget: Starlings.Generation.OnnxGenAI, 0.5.0"
#:package Starlings.Generation.OnnxGenAI@0.5.0
#addin nuget:?package=Starlings.Generation.OnnxGenAI&version=0.5.0
#tool nuget:?package=Starlings.Generation.OnnxGenAI&version=0.5.0
Starlings.Generation.OnnxGenAI
Text generation on ONNX Runtime GenAI 0.17.1 behind a Starling adapter, with an owned decode loop. Prompt ids come from the owned tokenizer (Starlings.Tokenizers); GenAI's tokenizer and template engine are not used. The API may still change.
GenAIModel.Load(directory, options): explicit provider (CPU, or CUDA with TF32 off by default), no silent fallback, config overlays (threads, pad id) recorded inAppliedOverlays, reference-counted disposal, package version check. On CUDA the native calls of concurrent sequences are serialized by default (SerializeNativeCalls): concurrent GenAI 0.17.1 generators on the CUDA provider produced wrong tokens.TextGenerator: greedy or seeded sampling,MaxNewTokens, stop token ids (never part of the completion) and stop strings (text cut before the match), streaming with UTF-8-safe incremental detokenization (IncrementalDetokenizer), optional chunked prefill, cancellation checked before the prefill, between prefill chunks and between decode steps (not inside a step), usage (PromptTokens,CompletionTokensexcluding the stop token) and stage timings. Synchronous by design: each step is blocking native work.StopSequenceMatcher: the stop-string matcherTextGeneratoruses, for applications with their own decode loop. Push each detokenized piece and emit whatPushreturns; text that could still start a stop string is held back, so no emitted piece contains part of one. WhenMatchedis set, the text is cut before the match and the request ends; otherwise emitFlush()at the end. Ordinal comparison; the earliest match wins, the longest at the same start. One instance per request.- Experimental, opt-in: guided decoding (
GenerationOptions.Guidance): a JSON Schema, Lark grammar or regex constraint passed to GenAI's llguidance integration (GeneratorParams.SetGuidance), optionally with fast-forward tokens. - Advanced:
GenAISequence(append ids, step, step with forced tokens, copy the last logits); it also implements the asynchronousIGenerationSequenceseam ofStarlings.Generation.Abstractions(each call completes synchronously).
The request and result types (GenerationOptions, SamplingOptions, GenerationGuidance, GenerationResult, GenerationUpdate, GenerationUsage, TokenLogprob, GenerationContextLengthException), StopSequenceMatcher and IncrementalDetokenizer are in Starlings.Generation.Abstractions (namespace Starlings.Generation), which this package references; they moved there unchanged in behaviour. Code in the namespace Starlings.Generation.OnnxGenAI sees them without a using; other code adds using Starlings.Generation;.
Most applications use it through a model package such as Starlings.Models.Qwen35.OnnxGenAI.
Use
using Starlings.Generation;
using Starlings.Generation.OnnxGenAI;
using Starlings.Tokenizers;
using var model = GenAIModel.Load(modelDirectory, new GenAIModelOptions { Provider = GenAIExecutionProvider.Cpu });
var tokenizer = Tokenizer.FromFile(tokenizerPath);
var generator = new TextGenerator(model, tokenizer);
var promptIds = tokenizer.Encode(promptText, addSpecialTokens: false).Ids; // the model package renders the template
GenerationResult result = generator.Generate(promptIds, new GenerationOptions { MaxNewTokens = 256 });
Console.WriteLine($"{result.Text} ({result.FinishReason}, {result.Usage.CompletionTokens} tokens)");
GenAIModelOptions:Provider(defaultCpu),CudaDeviceId(default 0),UseTf32(default false),IntraOpThreads(null: the CPU quota or affinity count in a constrained container, else the GenAI default),PadTokenId(null keeps the model's),ConfigOverlays(advanced JSON overlays ongenai_config.json),SerializeNativeCalls(null: serialized on CUDA, concurrent on CPU). The loadedGenAIModelreportsModelType,ContextLength,VocabularySize,EosTokenIds,PadTokenId,AppliedOverlays,ProviderOptions,SerializesNativeCalls, andGuidanceDirectoryandGuidanceMissingFiles(guided decoding).GenerationOptions:MaxNewTokens(default 512, at least 1),StopSequences(non-empty strings),AdditionalStopTokenIds,Sampling(null: greedy),PrefillChunkTokens(null: one prefill),ReturnLogprobsandTopLogprobs(opt-in per-token log probabilities, top 0 to 20; experimental),Guidance(null: unconstrained; experimental; not combinable withReturnLogprobs).SamplingOptions:Temperature(default 1.0, greater than 0),TopK(default 50),TopP(default 1.0, in (0, 1]),Seed(required, non-negative). The same seed, prompt, settings, model and device reproduce the same tokens.GenerationResult:Text,TokenIds,FinishReason(Stop,StopSequenceorLength; cancellation throws instead),StopTokenId,StopSequence,Usage(PromptTokens,CompletionTokens,TotalTokens),Timings(Prefill,Decode),Logprobs(withReturnLogprobs).GenerateStreamingyieldsGenerationUpdates (Text,TokenIds,Logprobs); the last hasIsFinalandResult.TokenLogprob(TokenId,Text,Logprob,TopLogprobsofTopTokenLogprob): one per completion token (the stop token has none), the natural-log probability from the log-softmax of the raw logits of the step that selected the token (before temperature, top-k and top-p). After a step, ONNX Runtime GenAI 0.17.1 reports the logits that selected the returned token, because it runs the model on that token only at the next step; the decode loop reads them right after the step. Each step then copies one logits row (vocabulary size x 4 bytes, about 1 MB for Qwen3.5).- Guided decoding (experimental, opt-in):
GenerationGuidance.JsonSchema(schema),Lark(grammar)orRegex(pattern)asGenerationOptions.Guidance. GenAI masks the tokens the constraint does not allow before it selects each one, and allows only an end-of-sequence id once the constraint is complete. GenAI builds the token table fromtokenizer.jsonandtokenizer_config.jsonin the model directory (GenAIModel.GuidanceFiles;model.tokenizer_dirwhen set).GenAIModel.GuidanceMissingFileslists the ones missing, and a guided request on such a directory fails withNotSupportedExceptionbefore any native call.WithFastForward(maxTokensPerStep)turns on GenAI's forced tokens: when the constraint leaves one continuation, GenAI appends it without a decode step per token. The loop returns those tokens in order, before the step's selected token, through the same token limit, stop ids, detokenization and stop sequences. GenAI does not check the sequence's maximum length for forced tokens, so the loop sizes the sequence forMaxNewTokensplusmaxTokensPerStep(counted in the context-length check), and a step that appends more fails the request. The bound is the caller's to derive from the constraint. WithoutGuidancethe loop is unchanged. - Lifetime: dispose
GenAIModel,GenAISequenceand streaming enumerators (foreachdoes). As a safety net, a sequence or model that becomes unreachable undisposed (for example an abandonedGenerateStreamingenumerator) is released after a garbage collection: GenAI's own finalizers on these objects are suppressed, and the adapter's finalizer queues the same release asDispose(generator under the native-call lock, then the model reference) to a thread-pool thread, never the finalizer thread. An abandoned request therefore no longer pins the model, but its native memory stays allocated until a collection finds it. GenAIRuntimeInfo:PinnedVersion(0.17.1), andManagedVersionandNativePackageread from the application's.deps.json(null in single-file or AOT hosts).- Exceptions:
GenAIRuntimeUnavailableException(no native library, or packages other than 0.17.1),GenAIProviderUnavailableException(the provider cannot be used; no fallback),GenAIModelException(GenAI rejected the directory, its configuration or a step),GenerationContextLengthException(prompt plusMaxNewTokensexceed the context length;PromptTokens,MaxNewTokens,ContextLength).
Maturity
Maturity is per profile, generated from the registry manifests by model-pipeline catalog (do not edit this table by hand). Gates, evidence and capabilities per profile: the model catalog.
| Profile | Maturity | Platforms |
|---|---|---|
qwen2.5-0.5b-instruct/ortgenai-fp32-cpu |
experimental | linux-x64 cpu: experimental; windows-x64 cpu: untested |
qwen3-0.6b/ortgenai-fp32-cpu |
experimental | linux-x64 cpu: experimental; windows-x64 cpu: untested |
qwen3.5-0.8b/ortgenai-fp16x-cuda |
stable | linux-x64 cuda: supported; windows-x64 cuda: untested |
qwen3.5-0.8b/ortgenai-fp32x-cpu |
stable | linux-x64 cpu: supported; windows-x64 cpu: untested |
qwen3.5-0.8b/ortgenai-fp32x-cuda |
stable | linux-x64 cuda: supported; windows-x64 cuda: untested |
qwen3.5-0.8b/ortgenai-int4-cpu |
experimental | linux-x64 cpu: experimental; linux-x64 cuda: unsupported; windows-x64 cpu: untested |
Native runtime
The package references Microsoft.ML.OnnxRuntimeGenAI.Managed (exactly 0.17.1; it asks for Microsoft.Extensions.AI.Abstractions 9.8.0 or later, which Starlings.Generation.Abstractions lifts to 10.10.1), Starlings.Generation.Abstractions and the owned tokenizer. The application references exactly one native package of the same version:
| Deployment | Package | Notes |
|---|---|---|
| CPU | Microsoft.ML.OnnxRuntimeGenAI 0.17.1 |
Brings Microsoft.ML.OnnxRuntime 1.30.0. Natives for linux-x64, linux-arm64, win-x64, win-arm64, osx-arm64; exercised on linux-x64 only. |
| CUDA | Microsoft.ML.OnnxRuntimeGenAI.Cuda 0.17.1 |
Brings Microsoft.ML.OnnxRuntime.Gpu 1.30.0. linux-x64 and win-x64 natives (CUDA 13 build); the CUDA runtime and cuBLAS are not shipped. Exercised on linux-x64 only. |
Native package check
The package ships MSBuild targets in buildTransitive/, so every executable or test project that references it, directly or through Starlings.Models.Qwen35.OnnxGenAI or Starlings.Models.QwenChat.OnnxGenAI (the definition packages Starlings.Models.Qwen35 and Starlings.Models.QwenChat do not bring it), checks its package graph after restore: warning STARLINGS003 when no GenAI native package is present (loading a model would throw GenAIRuntimeUnavailableException), error STARLINGS004 when both Microsoft.ML.OnnxRuntimeGenAI and Microsoft.ML.OnnxRuntimeGenAI.Cuda are. Class libraries and browser projects are not checked. Turn it off with <StarlingsOnnxRuntimeGenAINativeCheck>false</StarlingsOnnxRuntimeGenAINativeCheck>. CUDA build recipe and Linux GPU dependencies: native runtimes guide.
The model directory must come from the onnxruntime-genai builder of the same version. Documentation: ai-dotnet/README.md.
| Product | Versions Compatible and additional computed target framework versions. |
|---|---|
| .NET | net10.0 is compatible. net10.0-android was computed. net10.0-browser was computed. net10.0-ios was computed. net10.0-maccatalyst was computed. net10.0-macos was computed. net10.0-tvos was computed. net10.0-windows was computed. |
-
net10.0
- Microsoft.ML.OnnxRuntimeGenAI.Managed (= 0.17.1)
- Starlings.Generation.Abstractions (= 0.5.0)
- Starlings.Tokenizers (= 0.3.4)
NuGet packages (2)
Showing the top 2 NuGet packages that depend on Starlings.Generation.OnnxGenAI:
| Package | Downloads |
|---|---|
|
Starlings.Models.Qwen35.OnnxGenAI
Qwen/Qwen3.5-0.8B (revision 2fc06364) chat generation in-process on ONNX Runtime GenAI 0.17.1: Qwen35ChatModel over the model definition Starlings.Models.Qwen35 (the pinned chat template rendered in C#, the tool-call parser, prompt ids from the owned tokenizer Starlings.Tokenizers profile qwen35-bpe-chat: no Microsoft.ML.Tokenizers, no GenAI tokenizer) and Starling's decode loop (greedy or seeded sampling, stop tokens and strings, streaming, cancellation between steps, usage). Ships the asset lock of the ONNX Runtime GenAI bundle, not the weights. Profiles qwen3.5-0.8b/ortgenai-fp32x (CPU; CUDA with TF32 off) and ortgenai-fp16x (CUDA), Linux x64, Stable. The application adds Microsoft.ML.OnnxRuntimeGenAI or Microsoft.ML.OnnxRuntimeGenAI.Cuda. Maturity is per model profile (Stable, Preview or Experimental): see the README, generated from the model registry. |
|
|
Starlings.Models.QwenChat.OnnxGenAI
Qwen/Qwen3-0.6B (revision c1899de2, thinking disabled) and Qwen/Qwen2.5-0.5B-Instruct (revision 7ae55760) text chat generation in-process on ONNX Runtime GenAI 0.17.1: QwenChatModel over the model definitions Starlings.Models.QwenChat (the pinned chat templates rendered in C#, the tool-call parser, prompt ids from the owned tokenizer Starlings.Tokenizers: no Microsoft.ML.Tokenizers, no GenAI tokenizer) and Starling's decode loop (greedy, stop tokens and strings, streaming, cancellation between steps, usage). Ships no weights: QwenChatModelAssets downloads the profile's bundle explicitly from Starling's public Hugging Face hosting (or it is built locally from the pinned original), and Load verifies every file against the profile's digests. Profiles qwen3-0.6b/ortgenai-fp32-cpu and qwen2.5-0.5b-instruct/ortgenai-fp32-cpu, Linux x64. The application adds Microsoft.ML.OnnxRuntimeGenAI. Maturity is per model profile (Stable, Preview or Experimental): see the README, generated from the model registry. |
GitHub repositories
This package is not used by any popular GitHub repositories.
| Version | Downloads | Last Updated |
|---|---|---|
| 0.5.0 | 149 | 11/3/2026 |
| 0.4.0 | 115 | 11/3/2026 |
| 0.3.0 | 248 | 10/19/2026 |
| 0.2.0 | 356 | 10/17/2026 |
| 0.1.0-preview.6 | 115 | 10/14/2026 |