ModelSharp.Hub 1.0.3

<div align="center"> <img src="modelsharp_logo.png" alt="ModelSharp" width="120" height="120" />

<h1>ModelSharp</h1>

<p><strong>Universal, manifest-driven model inference for .NET — pure-managed by default, that <em>compiles</em> your model instead of interpreting it, with an optional native fast path.</strong></p>

<p> <a href="https://www.nuget.org/packages/ModelSharp"><img src="https://img.shields.io/nuget/v/ModelSharp.svg?label=nuget&color=512BD4" alt="NuGet" /></a> <a href="https://www.nuget.org/packages/ModelSharp"><img src="https://img.shields.io/nuget/dt/ModelSharp.svg?label=downloads&color=512BD4" alt="NuGet downloads" /></a> <a href="LICENSE"><img src="https://img.shields.io/badge/license-Apache--2.0-green.svg" alt="License: Apache-2.0" /></a> <img src="https://img.shields.io/badge/.NET-10.0-512BD4.svg" alt=".NET 10" /> <img src="https://img.shields.io/badge/core%20dependencies-zero-brightgreen.svg" alt="Zero core dependencies" /> <img src="https://img.shields.io/badge/platforms-Windows%20%7C%20Linux%20%7C%20macOS-informational.svg" alt="Cross-platform" /> <img src="https://img.shields.io/badge/x64%20%7C%20ARM64-single%20build-informational.svg" alt="x64 and ARM64" /> </p> </div>

ModelSharp runs real machine-learning models — vision, text, and audio — entirely in managed .NET. No Python, no native DLLs to ship, no per-model glue code. Point it at an ONNX, GGUF, or safetensors model and call one method: a small manifest describes how to feed and decode the model, so the same Pipeline API handles embeddings, image classification, object detection, speech recognition, and quantized LLM generation.

Under that one API, ModelSharp compiles the model once and runs the compiled plan — it doesn't re-walk the graph on every call. That compiled plan can be interpreted, JIT-lowered to a delegate, or even baked out to a standalone C# file with zero dependencies. A single build runs on Windows, Linux, and macOS, x64 and ARM64 — on CPU, and on the GPU through an optional backend.

using ModelSharp.Hub;

// Downloads the model + tokenizer, then runs it — one line.
using var pipeline = HubPipeline.Load("qwen2.5-0.5b-int4");
string answer = pipeline.Run<string>("The capital of France is");
// → " Paris. It is the largest city in the world by population…"

Compile, don't interpret

A naïve runtime walks the graph node-by-node on every Run(), looking up a kernel and dispatching it each time. ModelSharp instead compiles the graph once into an optimized plan, then executes that plan — and lets you choose how far to compile:

   ONNX / GGUF / safetensors
              │
              ▼
     ┌─────────────────┐
     │    GraphPlan     │   compile once: pre-resolve kernels, constant-fold,
     │  (fuse + fold)   │   fuse ops (Conv+BN, binary+ReLU, in-place activations),
     └─────────────────┘   compute liveness so dead tensors free immediately
              │
     ┌────────┼───────────────────────────┐
     ▼        ▼                            ▼
 interpret   JIT (Expression.Compile)   AOT bake (ModelBaker)
 the plan    → one straight-line          → standalone zero-dependency
 (default)     delegate, opt-in              C# source you can ship, opt-in

Every path drives the same kernels in the same order, so results stay bit-identical (or within documented float tolerance) — you trade nothing for the speedup. See How it runs for the details of each stage.

Contents

• Why ModelSharp
• Features
• Installation
• Quick start
• How it runs
• Supported tasks & formats
• Performance & optional native acceleration
• Verified on real models
• Architecture
• Packages
• Requirements
• License
• Contributing

Why ModelSharp

ONNX Runtime gives you tensor in → tensor out, but every model still needs its own pre/post-processing, and the native runtime can't run everywhere. ModelSharp closes both gaps:

• Self-describing models. A small manifest — embedded ONNX metadata, a sidecar JSON, or a built-in registry — describes how to feed and decode a model. One Pipeline API runs any model: text in or image in, typed result out, no per-model glue code.
• Pure-managed core. Managed kernels mean a single build runs everywhere .NET runs, with no native binaries to ship. Even the ONNX parser is a hand-rolled protobuf reader — there is no Google.Protobuf dependency either.
• Compiled, not interpreted. The graph is compiled once into a GraphPlan — kernels pre-resolved, constants folded, ops fused, dead tensors freed on their last use — then executed. Optionally JIT it to a single delegate, or bake it to standalone C#.
• Optional native speed. When you want more throughput, drop in the optional native kernel libraries (AVX-512 on CPU, cuBLAS on NVIDIA GPUs). The engine uses them when present and transparently falls back to managed otherwise — so you never trade away portability to get speed.
• Bit-verified. Outputs are validated against ONNX Runtime on real models, down to exact next-token logits on quantized LLMs — and every optimization is checked against the plain interpreter.

Features

• 🧩 One-line inference — Pipeline.Load("model.onnx").Run<T>(input) for any supported task.
• ⚙️ Compiles the graph once — a GraphPlan pre-resolves kernels, constant-folds, fuses ops, and frees dead tensors, so Run() isn't re-interpreting the model every call.
• 🚀 Optional JIT & AOT bake — lower the plan to a single delegate (jitCompile: true) or emit it as standalone, zero-dependency C# (ModelBaker.EmitCSharp).
• 📦 Zero dependencies in the core package — no native runtime, no Python, no protobuf library.
• 🌍 Runs everywhere .NET runs — single managed build, x64 / ARM64, all OSes.
• ⚡ Optional native fast path — opt-in AVX-512 / AVX512-VNNI CPU kernels and a cuBLAS GPU path, with automatic managed fallback. (See Performance.)
• 🔢 Multi-dtype engine — float32 / int64 / int32 / bool flow through as their real types, so token ids, masks, and shape tensors all work natively.
• 🧠 Broad operator coverage — CNNs, transformers, RNNs (LSTM/GRU), signal ops (DFT/STFT/MelWeightMatrix), control flow (If/Loop/Scan), sequence/optional ops, and quantized QLinear* / MatMulNBits ops.
• 🧮 Real quantized LLMs — runs INT4 / INT8 / fp16 ONNX LLMs end to end, including a 7B model (MatMulNBits INT4 + genai GroupQueryAttention) loaded from multi-gigabyte external-data files.
• ⏩ Greedy-equivalent speculative decoding — opt-in prompt-lookup drafting verifies repeated n-grams in one forward pass, with output identical to plain greedy decode.
• 🔁 Encoder-decoder & decoder-only generation — T5 / BART / MarianMT-style seq2seq (KV-cached decode) alongside decoder-only LLM generation.
• 🔤 Built-in tokenizers — WordPiece (BERT) and byte-level BPE (GPT-2 / RoBERTa), pure managed.
• 🎙️ Audio front end — FFT, log-mel spectrograms, and CTC decoding (greedy + prefix-beam) for ASR.
• ♻️ Runs any model on the GPU — the optional ILGPU backend executes supported ops on-device (with bias-add→activation epilogue fusion) and falls back to the CPU kernel for the rest, so anything that runs on CPU also runs through the GPU engine.
• ⬇️ Optional model hub — HubPipeline.Load("qwen2.5-0.5b-int4") downloads a model (plus its external-data shards and tokenizer) from Hugging Face, GGUF, safetensors, or any URL and runs it, with a local cache. Pure-managed (HttpClient only).

Installation

dotnet add package ModelSharp              # core: load + run any ONNX model on CPU, plus text & audio front ends
dotnet add package ModelSharp.ImageSharp   # optional: image decoding & classification
dotnet add package ModelSharp.Gpu          # optional: ILGPU GPU backend
dotnet add package ModelSharp.Hub          # optional: download models from Hugging Face / URLs

Requires .NET 10. The core ModelSharp package has no external dependencies.

Quick start

Download & run from the hub

using ModelSharp.Hub;

// Downloads the model + tokenizer/config (and any external-data shards) into a local cache, then runs it.
using var pipeline = HubPipeline.Load("qwen2.5-0.5b-int4");
string answer = pipeline.Run<string>("The capital of France is");

// …or resolve any Hugging Face repo / file, GGUF, safetensors, or direct URL:
ResolvedModel m = ModelHub.Get("onnx-community/Qwen2.5-0.5B-Instruct/onnx/model_q4.onnx");
// m.ModelPath is the local file; m.Files lists the model + tokenizer + config that came with it.

Text embeddings

using ModelSharp.Pipeline;

// The manifest is resolved automatically (sidecar JSON → ONNX metadata → built-in registry).
using var pipeline = Pipeline.Load("all-MiniLM-L6-v2.onnx");

float[] a = pipeline.Run<float[]>("A man is playing a guitar.");
float[] b = pipeline.Run<float[]>("Someone strums an acoustic guitar.");
float[] c = pipeline.Run<float[]>("The stock market fell sharply today.");

// a·b ≈ 0.70 (paraphrase)   a·c ≈ -0.05 (unrelated)

Image classification

using ModelSharp.Pipeline;
using ModelSharp.ImageSharp;

ImageSharpRegistration.Ensure();   // wire the image processors (or just reference the assembly)

using var pipeline = Pipeline.Load("resnet50.onnx");
var results = pipeline.Run<List<Classification>>("cat.jpg");   // also accepts byte[], Stream, or Image<Rgb24>

foreach (var r in results.Take(3))
    Console.WriteLine(r);   // e.g. "tiger cat (82%)"

LLM text generation

using ModelSharp.Pipeline;

using var pipeline = Pipeline.Load("Qwen2.5-0.5B-Instruct-q4.onnx");
string text = pipeline.Run<string>("The capital of France is");
// → " Paris. It is the largest city in the world by population…"

Speech recognition

using ModelSharp.Pipeline;

using var pipeline = Pipeline.Load("whisper-tiny.onnx");
string transcript = pipeline.Run<string>("audio.wav");
// → "Mr. Quilter is the apostle of the middle classes and we are glad to welcome his gospel."

Raw graph execution (full control)

When you want tensors in and tensors out — no manifest, no processors:

using ModelSharp.Onnx;
using ModelSharp.Cpu;
using ModelSharp.Tensors;

ModelGraph graph = OnnxModelLoader.LoadModel("model.onnx");

// The engine compiles the graph once (GraphPlan). Pass jitCompile: true to also
// lower the compiled plan to a single delegate via Expression.Compile().
using var engine = new ManagedCpuEngine(graph, jitCompile: false);

var feeds = new Dictionary<string, NamedTensor>
{
    ["input_ids"] = new NamedTensor(
        "input_ids",
        Tensor<long>.FromArray(new TensorShape(1, 5), new long[] { 101, 2054, 2003, 2009, 102 })),
};

IReadOnlyDictionary<string, NamedTensor> outputs = engine.Run(feeds);
Tensor<float> hidden = outputs["last_hidden_state"].Tensor.AsFloat();

How it runs

ModelSharp compiles a model once and then executes the compiled plan. You choose how far to take that compilation — from the default interpreter up to a standalone C# file — and every step is checked against the plain interpreter.

1. GraphPlan — compile once (on by default)

When you construct a ManagedCpuEngine, the graph is compiled into a GraphPlan:

• Kernels are pre-resolved into a flat array — no per-node dictionary lookup while running.
• Initializer-only subgraphs are constant-folded at compile time (shape math, reshapes, casts).
• Ops are fused (see below).
• Value liveness is computed, so a dead intermediate is released the moment its last consumer runs, keeping the working set small.

It's bit-identical to the old interpreter: anything that can't be folded or fused simply stays on the dynamic path (control flow, sequence/optional ops, unsupported shapes).

2. Operator fusion (on by default)

The fusion pass runs inside GraphPlan.Compile and rewrites common patterns into single kernels — each bit-identical to the unfused version:

On the GPU engine, a broadcasting binary op feeding a ReLU/Sigmoid/Tanh/GELU/LeakyReLU fuses into one device kernel (y = act(op(a, b))), collapsing two launches and the intermediate device buffer into one.

3. Model JIT — lower to a delegate (opt-in)

using var engine = new ManagedCpuEngine(graph, jitCompile: true);

The compiled plan is lowered to a single straight-line delegate via Expression.Compile(): each node becomes a direct bound-kernel call, each dead value a direct release, each constant a bound reference. This removes the interpreter loop's indexing, bounds checks, and per-node release branches. It drives the same kernels in the same order, so the output is identical to the interpreter.

4. AOT bake — emit standalone C# (opt-in)

using ModelSharp.Compile;

string code = ModelBaker.EmitCSharp(graph, className: "MyModel", @namespace: "Acme.Inference");
// → a self-contained .cs file: an inline float32 tensor struct, baked initializers,
//   and `Run(...)` — using only System.*, with no reference to ModelSharp.

ModelBaker emits the model as a zero-dependency C# source file you can drop into any project (great for AOT / trimming / auditing). It covers a core op subset (elementwise, MatMul/Gemm, shape/view ops); any op it can't emit throws at bake time rather than producing broken code. Baked numerics mirror the CPU kernels within float tolerance.

5. Speculative decoding — greedy-equivalent drafting (opt-in)

var config = new GenerationConfig
{
    UsePromptLookupDecoding = true,   // draft repeated n-grams from the prompt/context
    PromptLookupNgramSize = 3,
    PromptLookupMaxDraftTokens = 3,
};

Prompt-lookup speculative decoding matches the last n tokens against earlier occurrences, drafts the tokens that followed, and verifies them in a single multi-token forward pass. The emitted token sequence is identical to ordinary greedy (argmax) decoding — speculation only saves forward passes, it never changes the output. It engages only for greedy decode over a KV-cached model and falls back silently otherwise.

Supported tasks & formats

Performance & optional native acceleration

ModelSharp's default engine is pure managed and SIMD-tuned. Two things drive its throughput:

1. Compilation over interpretation. The GraphPlan pre-resolves kernels, folds constants, fuses ops, and frees dead tensors — and the optional JIT collapses the whole run into a single delegate (see How it runs).
2. Tuned managed kernels. A register-tiled, multithreaded, cache-blocked BlockedGemm (built on System.Numerics.Vector<T>, with real hardware FMA) backs MatMul, Gemm, and Conv. It runs ~2–5× faster than a naïve managed kernel — with zero native code and identical behavior on every platform — and adds direct depthwise / 1×1 conv fast paths and an online-softmax (flash) attention path that skips the full scores matrix.

For maximum throughput on supported hardware, ModelSharp ships an optional native kernel layer (under native/, built separately — it is not part of the NuGet packages, which stay pure-managed). The engine loads it when present and falls back to the managed kernels when it is absent, the CPU lacks the required ISA, or the shape isn't supported — so enabling it never costs you portability or correctness.

Highlights of the native layer:

• Portable and safe. The CPU library is built on a portable -mavx2 baseline and chooses AVX-512 / AVX512-VNNI / AVX2 / scalar paths at runtime, so it never executes an unsupported instruction on older x86.
• Resident weights on GPU. Model weights (including quantized weights) stay on the device across Run() calls instead of being re-uploaded each time, removing the dominant per-call PCIe cost for repeated inference.
• Opt-in via environment flags. MODELSHARP_NATIVE, MODELSHARP_W4A8, MODELSHARP_CUBLAS, MODELSHARP_TF32, MODELSHARP_RESIDENT_WEIGHTS. See native/README.md and native/GPU.md for build and tuning details.

Scope & honesty. The "~2–5×" and "~2.5–3×" figures are measured relative to ModelSharp's own baselines (a naïve managed kernel, and the managed kernel respectively), not claims of beating a vendor runtime. The managed engine remains the default; the native libraries are an opt-in build for users who want extra throughput on AVX-512 CPUs or NVIDIA GPUs, and are verified against the managed engine.

Verified on real models

Every supported task is validated end to end on real, exported models — with no Python at inference time and no native dependencies:

Beyond real-model parity, the optimizations themselves are pinned down by tests: the JIT output is asserted identical to the interpreter, baked C# is compiled and run and checked against the engine, speculative decode is asserted to match plain greedy, and GPU fusion kernels are compared against their CPU equivalents.

Architecture

ModelSharp is built around a single seam: a manifest-driven Pipeline on top of a swappable execution engine that compiles each model.

            +-----------------------------------------------+
  input --> |  Pipeline  (manifest-driven, engine-agnostic) | --> typed result
            +------+--------------------------+--------------+
             IPreprocessor             IPostprocessor
                          |
                          v
                 IExecutionEngine               <-- swappable backend
                    +-- ManagedCpuEngine  (ModelSharp core)
                    |     GraphPlan (fold + fuse + liveness) -> interpret | JIT delegate | AOT-baked C#
                    |     pure-managed kernels (+ optional native fast path)
                    +-- IlgpuEngine       (ModelSharp.Gpu)
                          C# kernels -> CUDA / OpenCL / CPU (+ epilogue fusion, optional cuBLAS)

• Manifest-driven pipeline. A manifest resolves automatically — sidecar JSON next to the model, then embedded ONNX metadata_props, then a built-in registry of filename heuristics — or you pass one explicitly. It selects the right pre/post-processors so one API serves every task.
• Compile-then-execute engine. The graph is compiled once into a GraphPlan (fold + fuse + liveness); the plan is then interpreted, JIT-lowered to a delegate, or baked to standalone C#. Tensors carry their real dtype end to end.
• Swappable backends. The public API and all processing are fixed behind IExecutionEngine; the CPU and GPU engines plug in underneath without changing caller code, and the optional native libraries plug in beneath them with automatic managed fallback.
• Hand-rolled ONNX reader. The loader is a custom protobuf parser, which is why the core package pulls in nothing — no Google.Protobuf, no native runtime.

Need a task ModelSharp doesn't ship? Register your own pre/post-processors with ProcessorRegistry.RegisterPreprocessor / RegisterPostprocessor — exactly how the ImageSharp package plugs itself in.

Packages

Packaging is split by dependency, not by feature — so the common case is a single download.

The optional native acceleration layer (native/) is a separate, opt-in build and is not distributed through any NuGet package — the published packages remain pure-managed.

Requirements

• .NET 10 or later.
• The core ModelSharp package has no external or native dependencies. Optional packages add only the managed dependencies listed above.
• The optional native layer requires a C++17 compiler (and the CUDA toolkit for the GPU library); see native/.

License

Apache License 2.0 — permissive, with an explicit patent grant (the prevailing license across the ML inference ecosystem: ONNX, ONNX Runtime, PyTorch, TensorFlow).

Contributing

Issues and pull requests are welcome. The project targets net10.0; dotnet build and dotnet test build the solution and run the test suite. New operator kernels and model-task processors are wired in behind the stable IExecutionEngine and ProcessorRegistry seams, so contributions extend coverage without breaking the public API.