Starlings.Models.Qwen35.Extensions.AI 0.5.0

dotnet add package Starlings.Models.Qwen35.Extensions.AI --version 0.5.0
                    
NuGet\Install-Package Starlings.Models.Qwen35.Extensions.AI -Version 0.5.0
                    
This command is intended to be used within the Package Manager Console in Visual Studio, as it uses the NuGet module's version of Install-Package.
<PackageReference Include="Starlings.Models.Qwen35.Extensions.AI" Version="0.5.0" />
                    
For projects that support PackageReference, copy this XML node into the project file to reference the package.
<PackageVersion Include="Starlings.Models.Qwen35.Extensions.AI" Version="0.5.0" />
                    
Directory.Packages.props
<PackageReference Include="Starlings.Models.Qwen35.Extensions.AI" />
                    
Project file
For projects that support Central Package Management (CPM), copy this XML node into the solution Directory.Packages.props file to version the package.
paket add Starlings.Models.Qwen35.Extensions.AI --version 0.5.0
                    
#r "nuget: Starlings.Models.Qwen35.Extensions.AI, 0.5.0"
                    
#r directive can be used in F# Interactive and Polyglot Notebooks. Copy this into the interactive tool or source code of the script to reference the package.
#:package Starlings.Models.Qwen35.Extensions.AI@0.5.0
                    
#:package directive can be used in C# file-based apps starting in .NET 10 preview 4. Copy this into a .cs file before any lines of code to reference the package.
#addin nuget:?package=Starlings.Models.Qwen35.Extensions.AI&version=0.5.0
                    
Install as a Cake Addin
#tool nuget:?package=Starlings.Models.Qwen35.Extensions.AI&version=0.5.0
                    
Install as a Cake Tool

Starlings.Models.Qwen35.Extensions.AI

Microsoft.Extensions.AI IChatClient over the Qwen/Qwen3.5-0.8B chat model in Starlings.Models.Qwen35.OnnxGenAI (model definition Starlings.Models.Qwen35): the same template, tokenizer, decode loop and tokens, usable with Microsoft.Extensions.AI middleware and dependency injection. The API may still change.

Maturity

Maturity is per profile, generated from the registry manifests by model-pipeline catalog (do not edit this table by hand). Gates, evidence and capabilities per profile: the model catalog.

Profile Maturity Platforms
qwen3.5-0.8b/ortgenai-fp16x-cuda stable linux-x64 cuda: supported; windows-x64 cuda: untested
qwen3.5-0.8b/ortgenai-fp32x-cpu stable linux-x64 cpu: supported; windows-x64 cpu: untested
qwen3.5-0.8b/ortgenai-fp32x-cuda stable linux-x64 cuda: supported; windows-x64 cuda: untested
qwen3.5-0.8b/ortgenai-int4-cpu experimental linux-x64 cpu: experimental; linux-x64 cuda: unsupported; windows-x64 cpu: untested

Install

dotnet add package Starlings.Models.Qwen35.Extensions.AI
dotnet add package Microsoft.ML.OnnxRuntimeGenAI --version 0.17.1        # CPU natives; or Microsoft.ML.OnnxRuntimeGenAI.Cuda (exactly one)

Use

using Starlings.Models.Qwen35;
using Starlings.Models.Qwen35.Extensions.AI;
using Microsoft.Extensions.AI;

Qwen35ModelFiles files = Qwen35ModelAssets.OpenLocal(Qwen35Variant.Fp32x);   // offline, verified, never downloads
using IChatClient client = Qwen35ChatClient.Create(files);

ChatResponse response = await client.GetResponseAsync("What is the capital of France?", new ChatOptions { MaxOutputTokens = 128 });
Console.WriteLine($"{response.Text} [{response.FinishReason}, {response.Usage?.InputTokenCount} + {response.Usage?.OutputTokenCount} tokens]");

await foreach (ChatResponseUpdate update in client.GetStreamingResponseAsync("Why is the sky blue?"))
{
    Console.Write(update.Text);
}

// Or with DI: services.AddQwen35ChatClient(files);

Tool calling (experimental)

var getWeather = AIFunctionFactory.Create((string city) => $"18 C and sunny in {city}.", "get_weather", "Get the current weather for a city.");
using IChatClient agent = new ChatClientBuilder(client).UseFunctionInvocation().Build();
ChatResponse answer = await agent.GetResponseAsync("What is the weather in Paris today?", new ChatOptions { Tools = [getWeather] });
Console.WriteLine(answer.Text);   // on the 0.8B model, greedy: "Today in Paris, the weather is 18 degrees Celsius and sunny."
  • ChatOptions.Tools: each AIFunctionDeclaration (any AIFunction) is rendered into the template's tools block as {"type": "function", "function": {"name", "description", "parameters": JsonSchema}}. Other tool types and duplicate names are rejected.
  • ToolMode: Auto (or unset) and None are supported; None renders no tools and parses nothing. RequireAny and RequireSpecific need guided decoding (below). On a directory without the guidance files (a cache acquired before hosting revision r2, or a local build without them) they are rejected with NotSupportedException: the template cannot require a call, and a constrained decode needs those files. AllowMultipleToolCalls = false is rejected with Auto.
  • History: FunctionCallContent in assistant messages is rendered as the template's <tool_call> blocks after the message text; each FunctionResultContent of a tool message becomes one template tool message (strings as is, other results as compact JSON), so several results render as consecutive <tool_response> blocks.
  • Responses: tool-call blocks become FunctionCallContent (a new CallId, Name, Arguments as JsonElement values cast with the tool's schema; RawRepresentation is the Qwen35ToolCall), and the finish reason is ToolCalls. Text before a call is TextContent. A malformed or unfinished block is returned as text, verbatim, with finish reason Stop or Length; the client never throws on model output. With thinking enabled, the reasoning up to </think> stays text and is not parsed (or becomes TextReasoningContent with SplitReasoning).
  • Streaming: text deltas are emitted as usual; text that could start a tool call is held back, and each FunctionCallContent is emitted once its block is complete, so no <tool_call> markup leaks for a valid call. The updates reassemble (ToChatResponse) to the non-streamed response.
  • A 0.8B model does not always call tools when it should, and may produce a malformed call or call an undeclared function; check calls before running them.

Guided decoding (experimental, opt-in)

// The hosted variant directories carry tokenizer.json and tokenizer_config.json since hosting revision r2.
using var guided = Qwen35ChatClient.Create(await Qwen35ModelAssets.DownloadAsync(Qwen35Variant.Fp32x));
ChatResponse call = await guided.GetResponseAsync("Weather in Porto?", new ChatOptions { Tools = [getWeather], ToolMode = ChatToolMode.RequireAny });
ChatResponse json = await guided.GetResponseAsync("6 times 7, as JSON.", new ChatOptions { ResponseFormat = ChatResponseFormat.ForJsonSchema(schema) });
  • When the model directory has ONNX Runtime GenAI's guidance files (Qwen35ModelInfo.GuidanceMissingFiles empty), ChatToolMode.RequireAny constrains the completion to calls of the offered functions with Qwen35ToolGrammar. RequireSpecific(name) constrains it to calls of that function (all functions are still rendered into the prompt). The grammar admits one call with AllowMultipleToolCalls = false, and one or more otherwise. A JSON ResponseFormat constrains the completion to its schema, or to a JSON object without one (json_schema guidance).
  • Rejected with NotSupportedException: any of these on a directory without the guidance files (the message names the missing files), a JSON format with offered tools, a JSON format with a required tool mode, guidance with enable_thinking or logprobs, and a required function that is not offered.
  • Qwen35ChatClientOptions.GuidanceFastForward (default false) builds the tool grammar with fast-forward tokens, so its fixed tags are appended without a decode step each.
  • Capabilities guidance.json_schema, guidance.lark and guidance.regex (in Qwen35Capabilities.All): experimental. Scripted sequences in CI; on the real fp32x model on CPU in a directory with the two files added, one RequireAny request returned a get_weather call and one JSON schema request returned matching JSON (docs/evidence/qwen35-guidance-2026-10-26/). extensions_ai.tool_mode.required and extensions_ai.structured_output: experimental since hosting revision r2, whose variant directories carry both files; on fp32x on CPU, loaded from the downloaded r2 files, GuidedRealModelTests passed (docs/evidence/qwen35-hosting-r2-2026-11-02/README.md).

Reasoning split (experimental, opt-in)

using var thinker = new Qwen35ChatClient(model, new Qwen35ChatClientOptions { SplitReasoning = true });
var options = new ChatOptions { AdditionalProperties = new() { ["enable_thinking"] = true } };
ChatResponse r = await thinker.GetResponseAsync("Why is the sky blue?", options);
// r.Messages[0].Contents: TextReasoningContent (the reasoning), then TextContent (the answer); r.Text is the answer.
  • Off by default: with thinking enabled, the completion stays one TextContent holding the reasoning, </think> and the answer, as before.
  • With SplitReasoning = true and thinking enabled, the text before the first </think> becomes a TextReasoningContent and the rest the answer (TextContent, or text and FunctionCallContent when tools are rendered). The tag is dropped; whitespace around it is kept verbatim. A completion that ends before the tag (for example at MaxOutputTokens) is reasoning only. Requests without thinking are unaffected.
  • Streaming: reasoning is emitted as TextReasoningContent deltas while it is generated; only text that could still start </think> is held back until the tag resolves, as text that could start a tool call is. Any split of a completion into deltas gives the same reasoning and answer as the non-streamed response.
  • Capability extensions_ai.reasoning_split: experimental (scripted sequences with the real template and tokenizer; on the real fp32x model on CPU, one test checks that the split partitions the unsplit completion, streamed and not).

Per-token log probabilities (experimental, opt-in)

var options = new ChatOptions { AdditionalProperties = new() { ["logprobs"] = true, ["top_logprobs"] = 5 } };
ChatResponse r = await client.GetResponseAsync("Name a primary colour.", options);
var logprobs = (IReadOnlyList<TokenLogprob>)r.AdditionalProperties!["logprobs"]!;
foreach (TokenLogprob t in logprobs) { /* t.TokenId, t.Text, t.Logprob, t.TopLogprobs (TokenId, Text, Logprob) */ }
  • Request keys: AdditionalProperties["logprobs"] (bool or JSON true/false; Qwen35ChatClient.LogprobsPropertyName) and AdditionalProperties["top_logprobs"] (0 to 20; TopLogprobsPropertyName, needs logprobs). Defaults come from Qwen35ChatClientOptions.Defaults.ReturnLogprobs and TopLogprobs; off by default.
  • Payload: under the same key "logprobs", an IReadOnlyList<TokenLogprob> (from Starlings.Generation.Abstractions, namespace Starlings.Generation): one entry per completion token in order (the stop token has none) with the token id, its text decoded on its own, its natural-log probability, and the top alternatives of that step, most likely first. ChatResponse.AdditionalProperties holds the whole list; each streamed ChatResponseUpdate.AdditionalProperties holds the entries of the tokens since the previous update, so concatenated they give the response's list (ToChatResponse does not concatenate them for you).
  • Values are the log-softmax of the raw logits of the step that selected each token, before temperature, top-k and top-p. Each step copies one logits row (about 1 MB for this model's vocabulary), so the option costs time per token.
  • Capability extensions_ai.logprobs: experimental (scripted sequences in CI; on the real fp32x model on CPU, greedy tokens are each step's most likely and the first step's top 5 equal the reference within the profile's step-0 top-5 bound).

Messages and options

  • Roles system, user, assistant and tool. Text content; TextReasoningContent and FunctionCallContent in assistant messages; FunctionResultContent in tool messages. Images and other content are rejected with NotSupportedException.
  • MaxOutputTokens, StopSequences, Temperature/TopP/TopK with an explicit Seed (greedy without them; sampling without a seed is rejected), and AdditionalProperties["enable_thinking"], ["logprobs"] and ["top_logprobs"]. Every other option that would change the result is rejected instead of being ignored.
  • Sampling: temperature, then top-k, then top-p through the ONNX Runtime GenAI sampler. When TopK is not set (by the request or a sampling default), the sampler's default top-k of 50 applies, also to a request that sets only Temperature or TopP with a Seed; ChatOptions has no way to switch top-k off, so set TopK explicitly when a different cut-off is wanted.
  • client.CountTokens(messages, options) renders a request exactly as GetResponseAsync would (defaults, instructions, tools, thinking mode) and returns its prompt token count, which equals the request's Usage.InputTokenCount, without generating. The mapping it uses is public: Qwen35ChatRequestMapper.MapMessages and MapOptions, with Qwen35ChatModel.CreatePrompt(messages, options).
  • Streaming updates are complete-character text deltas; they concatenate to the non-streamed text, and the last update carries the finish reason (Stop, Length or ToolCalls) and usage (stop token excluded).
  • Qwen35ChatClient.Create(files, modelOptions, options) and AddQwen35ChatClient take Qwen35ModelOptions (for example Provider = GenAIExecutionProvider.Cuda) and Qwen35ChatClientOptions: Defaults (a Qwen35ChatOptions used where a request's ChatOptions sets nothing; default 512 max new tokens, greedy, thinking off), StreamingBufferCapacity (updates buffered ahead of a slow consumer, default 16), MaxConcurrentRequests (admission control, below; default unbounded), SplitReasoning (default false) and GuidanceFastForward (default false).
  • Two opt-in hardening settings live on Defaults because ChatOptions cannot express them, and both default to off so that today's output is unchanged byte for byte: EscapeSpecialTokens rewrites every <|name|> in message and tool-result content to <¦name¦> before rendering, so content containing <|im_end|> or <|im_start|> cannot rewrite the conversation (off is faithful to upstream apply_chat_template, which renders content verbatim); StopAtRoleMarker adds <|im_start|> to the stop token ids, so a model that runs on after its answer into <|im_start|>user ... stops instead of streaming the role markup as text.
  • Each request runs on a dedicated background thread (ONNX Runtime GenAI steps are blocking native calls). Cancellation takes effect between prefill chunks and decode steps.
  • Admission control: MaxConcurrentRequests = n lets at most n requests (non-streamed and streamed together) generate at once; further requests wait in order before any thread or native generator is created for them, a wait honours the request's cancellation token, and a streaming request holds its slot from its first MoveNextAsync until its enumeration completes or is disposed. Unset (the default), every request starts at once, as before.

Status: supported on the Stable profiles qwen3.5-0.8b/ortgenai-fp32x-cpu (Linux x64 CPU) and ortgenai-fp32x-cuda and ortgenai-fp16x-cuda (Linux x64, RTX 3090), under the contracts in the Starlings.Models.Qwen35.OnnxGenAI README; tool calling (ChatOptions.Tools, function invocation), the reasoning split, per-token log probabilities and guided decoding are experimental. See the documentation and evidence.

Product Compatible and additional computed target framework versions.
.NET net10.0 is compatible.  net10.0-android was computed.  net10.0-browser was computed.  net10.0-ios was computed.  net10.0-maccatalyst was computed.  net10.0-macos was computed.  net10.0-tvos was computed.  net10.0-windows was computed. 
Compatible target framework(s)
Included target framework(s) (in package)
Learn more about Target Frameworks and .NET Standard.

NuGet packages

This package is not used by any NuGet packages.

GitHub repositories

This package is not used by any popular GitHub repositories.

Version Downloads Last Updated
0.5.0 127 11/3/2026
0.4.0 84 11/3/2026
0.3.0 132 10/19/2026
0.2.0 157 10/17/2026
0.1.0-preview.6 108 10/14/2026