TensorStack.OnnxRuntime.LLM
0.15.1
Prefix Reserved
dotnet add package TensorStack.OnnxRuntime.LLM --version 0.15.1
NuGet\Install-Package TensorStack.OnnxRuntime.LLM -Version 0.15.1
<PackageReference Include="TensorStack.OnnxRuntime.LLM" Version="0.15.1" />
<PackageVersion Include="TensorStack.OnnxRuntime.LLM" Version="0.15.1" />
<PackageReference Include="TensorStack.OnnxRuntime.LLM" />
paket add TensorStack.OnnxRuntime.LLM --version 0.15.1
#r "nuget: TensorStack.OnnxRuntime.LLM, 0.15.1"
#:package TensorStack.OnnxRuntime.LLM@0.15.1
#addin nuget:?package=TensorStack.OnnxRuntime.LLM&version=0.15.1
#tool nuget:?package=TensorStack.OnnxRuntime.LLM&version=0.15.1
TensorStack.OnnxRuntime.LLM
ONNX pipelines for small language, vision and speech models: text generation, speech recognition, text to speech, image captioning/OCR/detection and summarization, with greedy and beam search decoding.
Nuget Package
For GGUF language models with tool calls and image/audio input see TensorStack.LlamaCpp.
Loading and saving images and audio uses TensorStack.Media.
Pipelines
| Pipeline | Models | Use | Model files |
|---|---|---|---|
LlamaPipeline |
Llama 3.2 | Text generation | model.onnx, tokenizer.json |
Phi3Pipeline |
Phi-3 mini, medium | Text generation | model.onnx, tokenizer.json (or tokenizer.model) |
WhisperPipeline |
Whisper tiny to large-v3, turbo | Speech recognition and translation | encoder_model.onnx, decoder_model_merged.onnx, tokenizer.json (or vocab.json + merges.txt + tokenizer_config.json) |
SupertonicPipeline |
Supertonic, Supertonic 3 | Text to speech | duration_predictor.onnx, text_encoder.onnx, vector_estimator.onnx, vocoder.onnx, unicode_indexer.json, voice_styles/*.json |
FlorencePipeline |
Florence-2 base, large | Captioning, OCR, object detection, grounding and segmentation | vision_encoder.onnx, embed_tokens.onnx, encoder_model.onnx, decoder_model_merged.onnx, tokenizer.json (or vocab.json + merges.txt + tokenizer_config.json) |
SummaryPipeline |
T5 | Text summarization | encoder_model.onnx, decoder_model_merged.onnx, tokenizer.json (or spiece.model) |
Model sizes (layers, heads, cache dimensions, mel bins) are read from the ONNX models, the Create model type is only a fallback.
Tokenizers are read from the Hugging Face tokenizer.json when the model folder has one, falling back to the older vocabulary/merges or SentencePiece files.
Execution Providers
Each Create has an overload taking a provider per model. Run vision and audio encoders on the GPU and language model decoders on the CPU, DirectML returns incorrect tokens for some decoders.
var gpu = Provider.GetProvider(DeviceType.GPU);
var cpu = Provider.GetProvider(DeviceType.CPU);
var pipeline = WhisperPipeline.Create(gpu, cpu, "M:\\Models\\Whisper-Base", WhisperType.Base);
Decoding
Each text pipeline has two RunAsync overloads:
GenerateOptions(WhisperOptions,FlorenceOptions) returns a singleGenerateResultusing greedy decodingSearchOptions(WhisperSearchOptions,FlorenceSearchOptions) returns aGenerateResult[], best first, using beam search
| Option | Description |
|---|---|
MinLength / MaxLength |
The generated length in tokens |
Beams |
Number of beams for beam search |
TopK / TopP / Temperature |
Beam search candidate sampling |
LengthPenalty |
Beam score length penalty (1 = none, higher favours longer results) |
NoRepeatNgramSize |
Blocks a repeat of any n-gram of this size, 0 to disable |
EarlyStopping |
None, BestBeam or BeamCount |
Seed |
Random seed for beam search sampling, negative for random |
GenerateResult.Score is the sum of the token log probabilities. Progress callbacks report the text as it is generated, IsReset is set when the best beam changes and the full text is reported again.
Llama
var provider = Provider.GetProvider(DeviceType.CPU);
using (var pipeline = LlamaPipeline.Create(provider, "M:\\Models\\Llama-3.2-1B-Instruct"))
{
var result = await pipeline.RunAsync(new GenerateOptions
{
Prompt = "<|start_header_id|>user<|end_header_id|>\n\nWhat is an apple?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n",
MaxLength = 200,
MinLength = 0
});
Console.WriteLine(result.Result);
}
BOS is added to the prompt, generation stops at <|eot_id|>, <|eom_id|> or <|end_of_text|>.
GetLastHiddenStateAsync returns the prompt's last hidden state for use as a text encoder.
Phi-3
var provider = Provider.GetProvider(DeviceType.CPU);
using (var pipeline = Phi3Pipeline.Create(provider, "M:\\Models\\Phi-3-medium-128k-instruct", PhiType.Medium))
{
var results = await pipeline.RunAsync(new SearchOptions
{
Prompt = "<|user|>\nWhat is an apple?<|end|>\n<|assistant|>\n",
Beams = 3,
TopK = 3,
MaxLength = 200,
MinLength = 0
});
foreach (var result in results)
Console.WriteLine(result.Result);
}
Whisper
var gpu = Provider.GetProvider(DeviceType.GPU);
var cpu = Provider.GetProvider(DeviceType.CPU);
using (var pipeline = WhisperPipeline.Create(gpu, cpu, "M:\\Models\\Whisper-Base", WhisperType.Base))
{
var result = await pipeline.RunAsync(new WhisperOptions
{
Task = TaskType.Transcribe,
Language = LanguageType.EN,
AudioInput = await AudioInput.CreateAsync("Speech.mp3"), // 16 kHz mono
MinLength = 0,
MaxLength = 448,
NoRepeatNgramSize = 0
});
Console.WriteLine(result.Result);
}
- Audio must be 16 kHz,
AudioInput.CreateAsyncloads at 16 kHz by default, other audio can be converted withaudioTensor.Resample(16000) - Long audio is split into chunks of up to
ChunkSizeseconds (max 30) at the quietest point near each boundary TaskType.Translatetranslates to English- The mel filters are computed, large-v3 and turbo use 128 mel bins (read from
preprocessor_config.jsonor the model)
Supertonic
var provider = Provider.GetProvider(DeviceType.GPU);
using (var pipeline = SupertonicPipeline.Create("M:\\Models\\Supertonic", provider))
{
var audio = await pipeline.RunAsync(new SupertonicOptions
{
TextInput = "On a quiet morning in the old town, a clockmaker named Ellis unlocked his tiny shop.",
VoiceStyle = "Female1",
Language = "en", // Supertonic 3, null for Supertonic 1
Steps = 5,
Speed = 1f
});
await audio.SaveAsync("Output.wav");
}
Text is split into sentences of up to 300 characters (120 for Korean) with SilenceDuration seconds between them. pipeline.VoiceStyles lists the available voices.
Florence-2
var gpu = Provider.GetProvider(DeviceType.GPU);
var cpu = Provider.GetProvider(DeviceType.CPU);
using (var pipeline = FlorencePipeline.Create(gpu, cpu, gpu, gpu, "M:\\Models\\Florence-2-large", FlorenceType.Large))
{
var result = await pipeline.RunAsync(new FlorenceOptions
{
TaskType = TaskType.OD,
Image = new ImageInput("Image.png"),
MinLength = 0,
MaxLength = 1024
});
foreach (var coordinate in result.CoordinateResults)
Console.WriteLine($"{coordinate.Label}: {coordinate.CoordinateBox}");
}
- Text tasks (
CAPTION,DETAILED_CAPTION,MORE_DETAILED_CAPTION,OCR) returnResult - Region tasks (
OD,DENSE_REGION_CAPTION,REGION_PROPOSAL,CAPTION_TO_PHRASE_GROUNDING,OCR_WITH_REGIONetc) returnCoordinateResultsin source image pixels - Region input tasks (
REGION_TO_CATEGORY,REGION_TO_DESCRIPTION,REGION_TO_OCR,REGION_TO_SEGMENTATION) takeRegion CAPTION_TO_PHRASE_GROUNDING,OPEN_VOCABULARY_DETECTIONandREFERRING_EXPRESSION_SEGMENTATIONtakePrompt
Summary
var gpu = Provider.GetProvider(DeviceType.GPU);
var cpu = Provider.GetProvider(DeviceType.CPU);
using (var pipeline = SummaryPipeline.Create(gpu, cpu, "M:\\Models\\text_summarization"))
{
var result = await pipeline.RunAsync(new GenerateOptions
{
Prompt = "summarize: " + text,
MinLength = 10,
MaxLength = 200,
NoRepeatNgramSize = 3
});
Console.WriteLine(result.Result);
}
T5 models expect the task prefix (summarize: ) on the prompt.
| Product | Versions Compatible and additional computed target framework versions. |
|---|---|
| .NET | net10.0-windows10.0.17763 is compatible. |
-
net10.0-windows10.0.17763
- MathNet.Numerics (>= 5.0.0)
- Microsoft.ML.Tokenizers (>= 3.0.0)
- TensorStack.Common (>= 0.15.1)
- TensorStack.OnnxRuntime (>= 0.15.1)
NuGet packages
This package is not used by any NuGet packages.
GitHub repositories
This package is not used by any popular GitHub repositories.
| Version | Downloads | Last Updated |
|---|---|---|
| 0.15.1 | 0 | 11/21/2026 |
| 0.15.0 | 50 | 11/18/2026 |
| 0.14.14 | 54 | 11/17/2026 |
| 0.14.13 | 67 | 11/16/2026 |
| 0.14.12 | 61 | 11/15/2026 |
| 0.14.11 | 55 | 11/15/2026 |
| 0.14.10 | 65 | 11/14/2026 |
| 0.14.9 | 65 | 11/13/2026 |
| 0.14.8 | 61 | 11/10/2026 |
| 0.14.7 | 63 | 11/8/2026 |
| 0.14.6 | 69 | 11/5/2026 |
| 0.14.5 | 90 | 11/1/2026 |
| 0.14.4 | 134 | 10/29/2026 |
| 0.14.3 | 79 | 10/28/2026 |
| 0.14.2 | 111 | 10/20/2026 |
| 0.14.1 | 87 | 10/16/2026 |
| 0.13.13 | 79 | 10/11/2026 |
| 0.13.12 | 74 | 10/10/2026 |
| 0.13.10 | 91 | 10/6/2026 |
| 0.13.9 | 72 | 10/5/2026 |