Starlings.Models.Qwen35.Extensions.AI
0.4.0
See the version list below for details.
dotnet add package Starlings.Models.Qwen35.Extensions.AI --version 0.4.0
NuGet\Install-Package Starlings.Models.Qwen35.Extensions.AI -Version 0.4.0
<PackageReference Include="Starlings.Models.Qwen35.Extensions.AI" Version="0.4.0" />
<PackageVersion Include="Starlings.Models.Qwen35.Extensions.AI" Version="0.4.0" />
<PackageReference Include="Starlings.Models.Qwen35.Extensions.AI" />
paket add Starlings.Models.Qwen35.Extensions.AI --version 0.4.0
#r "nuget: Starlings.Models.Qwen35.Extensions.AI, 0.4.0"
#:package Starlings.Models.Qwen35.Extensions.AI@0.4.0
#addin nuget:?package=Starlings.Models.Qwen35.Extensions.AI&version=0.4.0
#tool nuget:?package=Starlings.Models.Qwen35.Extensions.AI&version=0.4.0
Starlings.Models.Qwen35.Extensions.AI
Microsoft.Extensions.AI IChatClient over the Qwen/Qwen3.5-0.8B chat model in Starlings.Models.Qwen35.OnnxGenAI (model definition Starlings.Models.Qwen35): the same template, tokenizer, decode loop and tokens, usable with Microsoft.Extensions.AI middleware and dependency injection. The API may still change.
Maturity
Maturity is per profile, generated from the registry manifests by model-pipeline catalog (do not edit this table by hand). Gates, evidence and capabilities per profile: the model catalog.
| Profile | Maturity | Platforms |
|---|---|---|
qwen3.5-0.8b/ortgenai-fp16x-cuda |
stable | linux-x64 cuda: supported; windows-x64 cuda: untested |
qwen3.5-0.8b/ortgenai-fp32x-cpu |
stable | linux-x64 cpu: supported; windows-x64 cpu: untested |
qwen3.5-0.8b/ortgenai-fp32x-cuda |
stable | linux-x64 cuda: supported; windows-x64 cuda: untested |
Install
dotnet add package Starlings.Models.Qwen35.Extensions.AI
dotnet add package Microsoft.ML.OnnxRuntimeGenAI --version 0.17.1 # CPU natives; or Microsoft.ML.OnnxRuntimeGenAI.Cuda (exactly one)
Use
using Starlings.Models.Qwen35;
using Starlings.Models.Qwen35.Extensions.AI;
using Microsoft.Extensions.AI;
Qwen35ModelFiles files = Qwen35ModelAssets.OpenLocal(Qwen35Variant.Fp32x); // offline, verified, never downloads
using IChatClient client = Qwen35ChatClient.Create(files);
ChatResponse response = await client.GetResponseAsync("What is the capital of France?", new ChatOptions { MaxOutputTokens = 128 });
Console.WriteLine($"{response.Text} [{response.FinishReason}, {response.Usage?.InputTokenCount} + {response.Usage?.OutputTokenCount} tokens]");
await foreach (ChatResponseUpdate update in client.GetStreamingResponseAsync("Why is the sky blue?"))
{
Console.Write(update.Text);
}
// Or with DI: services.AddQwen35ChatClient(files);
Tool calling (experimental)
var getWeather = AIFunctionFactory.Create((string city) => $"18 C and sunny in {city}.", "get_weather", "Get the current weather for a city.");
using IChatClient agent = new ChatClientBuilder(client).UseFunctionInvocation().Build();
ChatResponse answer = await agent.GetResponseAsync("What is the weather in Paris today?", new ChatOptions { Tools = [getWeather] });
Console.WriteLine(answer.Text); // on the 0.8B model, greedy: "Today in Paris, the weather is 18 degrees Celsius and sunny."
ChatOptions.Tools: eachAIFunctionDeclaration(anyAIFunction) is rendered into the template's tools block as{"type": "function", "function": {"name", "description", "parameters": JsonSchema}}. Other tool types and duplicate names are rejected.ToolMode:Auto(or unset) andNoneare supported;Nonerenders no tools and parses nothing.RequireAnyandRequireSpecificneed guided decoding (below). On the published bundle they are rejected withNotSupportedException: the template cannot require a call, and the bundle lacks the files a constrained decode needs.AllowMultipleToolCalls = falseis rejected withAuto.- History:
FunctionCallContentin assistant messages is rendered as the template's<tool_call>blocks after the message text; eachFunctionResultContentof a tool message becomes one template tool message (strings as is, other results as compact JSON), so several results render as consecutive<tool_response>blocks. - Responses: tool-call blocks become
FunctionCallContent(a newCallId,Name,ArgumentsasJsonElementvalues cast with the tool's schema;RawRepresentationis theQwen35ToolCall), and the finish reason isToolCalls. Text before a call isTextContent. A malformed or unfinished block is returned as text, verbatim, with finish reasonStoporLength; the client never throws on model output. With thinking enabled, the reasoning up to</think>stays text and is not parsed (or becomesTextReasoningContentwithSplitReasoning). - Streaming: text deltas are emitted as usual; text that could start a tool call is held back, and each
FunctionCallContentis emitted once its block is complete, so no<tool_call>markup leaks for a valid call. The updates reassemble (ToChatResponse) to the non-streamed response. - A 0.8B model does not always call tools when it should, and may produce a malformed call or call an undeclared function; check calls before running them.
Guided decoding (experimental, opt-in)
// A directory with tokenizer.json and tokenizer_config.json next to the variant's files (the published bundle has neither).
using var guided = Qwen35ChatClient.Create(filesWithGuidanceFiles);
ChatResponse call = await guided.GetResponseAsync("Weather in Porto?", new ChatOptions { Tools = [getWeather], ToolMode = ChatToolMode.RequireAny });
ChatResponse json = await guided.GetResponseAsync("6 times 7, as JSON.", new ChatOptions { ResponseFormat = ChatResponseFormat.ForJsonSchema(schema) });
- When the model directory has ONNX Runtime GenAI's guidance files (
Qwen35ModelInfo.GuidanceMissingFilesempty),ChatToolMode.RequireAnyconstrains the completion to calls of the offered functions withQwen35ToolGrammar.RequireSpecific(name)constrains it to calls of that function (all functions are still rendered into the prompt). The grammar admits one call withAllowMultipleToolCalls = false, and one or more otherwise. A JSONResponseFormatconstrains the completion to its schema, or to a JSON object without one (json_schemaguidance). - Rejected with
NotSupportedException: any of these on a directory without the guidance files (the message names the missing files), a JSON format with offered tools, a JSON format with a required tool mode, guidance withenable_thinkingorlogprobs, and a required function that is not offered. Qwen35ChatClientOptions.GuidanceFastForward(default false) builds the tool grammar with fast-forward tokens, so its fixed tags are appended without a decode step each.- Capabilities
guidance.json_schema,guidance.larkandguidance.regex(inQwen35Capabilities.All): experimental. Scripted sequences in CI; on the real fp32x model on CPU in a directory with the two files added, oneRequireAnyrequest returned aget_weathercall and one JSON schema request returned matching JSON (docs/evidence/qwen35-guidance-2026-10-26/). On the published bundle,extensions_ai.tool_mode.requiredandextensions_ai.structured_outputstay unsupported.
Reasoning split (experimental, opt-in)
using var thinker = new Qwen35ChatClient(model, new Qwen35ChatClientOptions { SplitReasoning = true });
var options = new ChatOptions { AdditionalProperties = new() { ["enable_thinking"] = true } };
ChatResponse r = await thinker.GetResponseAsync("Why is the sky blue?", options);
// r.Messages[0].Contents: TextReasoningContent (the reasoning), then TextContent (the answer); r.Text is the answer.
- Off by default: with thinking enabled, the completion stays one
TextContentholding the reasoning,</think>and the answer, as before. - With
SplitReasoning = trueand thinking enabled, the text before the first</think>becomes aTextReasoningContentand the rest the answer (TextContent, or text andFunctionCallContentwhen tools are rendered). The tag is dropped; whitespace around it is kept verbatim. A completion that ends before the tag (for example atMaxOutputTokens) is reasoning only. Requests without thinking are unaffected. - Streaming: reasoning is emitted as
TextReasoningContentdeltas while it is generated; only text that could still start</think>is held back until the tag resolves, as text that could start a tool call is. Any split of a completion into deltas gives the same reasoning and answer as the non-streamed response. - Capability
extensions_ai.reasoning_split: experimental (scripted sequences with the real template and tokenizer; on the real fp32x model on CPU, one test checks that the split partitions the unsplit completion, streamed and not).
Per-token log probabilities (experimental, opt-in)
var options = new ChatOptions { AdditionalProperties = new() { ["logprobs"] = true, ["top_logprobs"] = 5 } };
ChatResponse r = await client.GetResponseAsync("Name a primary colour.", options);
var logprobs = (IReadOnlyList<TokenLogprob>)r.AdditionalProperties!["logprobs"]!;
foreach (TokenLogprob t in logprobs) { /* t.TokenId, t.Text, t.Logprob, t.TopLogprobs (TokenId, Text, Logprob) */ }
- Request keys:
AdditionalProperties["logprobs"](boolor JSONtrue/false;Qwen35ChatClient.LogprobsPropertyName) andAdditionalProperties["top_logprobs"](0 to 20;TopLogprobsPropertyName, needslogprobs). Defaults come fromQwen35ChatClientOptions.Defaults.ReturnLogprobsandTopLogprobs; off by default. - Payload: under the same key
"logprobs", anIReadOnlyList<TokenLogprob>(fromStarlings.Generation.Abstractions, namespaceStarlings.Generation): one entry per completion token in order (the stop token has none) with the token id, its text decoded on its own, its natural-log probability, and the top alternatives of that step, most likely first.ChatResponse.AdditionalPropertiesholds the whole list; each streamedChatResponseUpdate.AdditionalPropertiesholds the entries of the tokens since the previous update, so concatenated they give the response's list (ToChatResponsedoes not concatenate them for you). - Values are the log-softmax of the raw logits of the step that selected each token, before temperature, top-k and top-p. Each step copies one logits row (about 1 MB for this model's vocabulary), so the option costs time per token.
- Capability
extensions_ai.logprobs: experimental (scripted sequences in CI; on the real fp32x model on CPU, greedy tokens are each step's most likely and the first step's top 5 equal the reference within the profile's step-0 top-5 bound).
Messages and options
- Roles system, user, assistant and tool. Text content;
TextReasoningContentandFunctionCallContentin assistant messages;FunctionResultContentin tool messages. Images and other content are rejected withNotSupportedException. MaxOutputTokens,StopSequences,Temperature/TopP/TopKwith an explicitSeed(greedy without them; sampling without a seed is rejected), andAdditionalProperties["enable_thinking"],["logprobs"]and["top_logprobs"]. Every other option that would change the result is rejected instead of being ignored.- Sampling: temperature, then top-k, then top-p through the ONNX Runtime GenAI sampler. When
TopKis not set (by the request or a sampling default), the sampler's default top-k of 50 applies, also to a request that sets onlyTemperatureorTopPwith aSeed;ChatOptionshas no way to switch top-k off, so setTopKexplicitly when a different cut-off is wanted. client.CountTokens(messages, options)renders a request exactly asGetResponseAsyncwould (defaults, instructions, tools, thinking mode) and returns its prompt token count, which equals the request'sUsage.InputTokenCount, without generating. The mapping it uses is public:Qwen35ChatRequestMapper.MapMessagesandMapOptions, withQwen35ChatModel.CreatePrompt(messages, options).- Streaming updates are complete-character text deltas; they concatenate to the non-streamed text, and the last update carries the finish reason (
Stop,LengthorToolCalls) and usage (stop token excluded). Qwen35ChatClient.Create(files, modelOptions, options)andAddQwen35ChatClienttakeQwen35ModelOptions(for exampleProvider = GenAIExecutionProvider.Cuda) andQwen35ChatClientOptions:Defaults(aQwen35ChatOptionsused where a request'sChatOptionssets nothing; default 512 max new tokens, greedy, thinking off),StreamingBufferCapacity(updates buffered ahead of a slow consumer, default 16),MaxConcurrentRequests(admission control, below; default unbounded),SplitReasoning(default false) andGuidanceFastForward(default false).- Two opt-in hardening settings live on
DefaultsbecauseChatOptionscannot express them, and both default to off so that today's output is unchanged byte for byte:EscapeSpecialTokensrewrites every<|name|>in message and tool-result content to<¦name¦>before rendering, so content containing<|im_end|>or<|im_start|>cannot rewrite the conversation (off is faithful to upstreamapply_chat_template, which renders content verbatim);StopAtRoleMarkeradds<|im_start|>to the stop token ids, so a model that runs on after its answer into<|im_start|>user ...stops instead of streaming the role markup as text. - Each request runs on a dedicated background thread (ONNX Runtime GenAI steps are blocking native calls). Cancellation takes effect between prefill chunks and decode steps.
- Admission control:
MaxConcurrentRequests = nlets at mostnrequests (non-streamed and streamed together) generate at once; further requests wait in order before any thread or native generator is created for them, a wait honours the request's cancellation token, and a streaming request holds its slot from its firstMoveNextAsyncuntil its enumeration completes or is disposed. Unset (the default), every request starts at once, as before.
Status: supported on the Stable profiles qwen3.5-0.8b/ortgenai-fp32x-cpu (Linux x64 CPU) and ortgenai-fp32x-cuda and ortgenai-fp16x-cuda (Linux x64, RTX 3090), under the contracts in the Starlings.Models.Qwen35.OnnxGenAI README; tool calling (ChatOptions.Tools, function invocation), the reasoning split, per-token log probabilities and guided decoding are experimental. See the documentation and evidence.
| Product | Versions Compatible and additional computed target framework versions. |
|---|---|
| .NET | net10.0 is compatible. net10.0-android was computed. net10.0-browser was computed. net10.0-ios was computed. net10.0-maccatalyst was computed. net10.0-macos was computed. net10.0-tvos was computed. net10.0-windows was computed. |
-
net10.0
- Microsoft.Extensions.AI.Abstractions (>= 10.10.1)
- Microsoft.Extensions.DependencyInjection.Abstractions (>= 10.0.12)
- Starlings.Models.Qwen35.OnnxGenAI (= 0.4.0)
NuGet packages
This package is not used by any NuGet packages.
GitHub repositories
This package is not used by any popular GitHub repositories.
| Version | Downloads | Last Updated |
|---|---|---|
| 0.5.0 | 127 | 11/3/2026 |
| 0.4.0 | 84 | 11/3/2026 |
| 0.3.0 | 132 | 10/19/2026 |
| 0.2.0 | 157 | 10/17/2026 |
| 0.1.0-preview.6 | 108 | 10/14/2026 |