ModelSharp 1.0.2

<div align="center"> <img src="modelsharp_logo.png" alt="ModelSharp" width="120" height="120" />

<h1>ModelSharp</h1>

<p><strong>Universal, manifest-driven model inference for .NET — pure-managed by default, with an optional native fast path.</strong></p>

<p> <a href="https://www.nuget.org/packages/ModelSharp"><img src="https://img.shields.io/nuget/v/ModelSharp.svg?label=nuget&color=512BD4" alt="NuGet" /></a> <a href="https://www.nuget.org/packages/ModelSharp"><img src="https://img.shields.io/nuget/dt/ModelSharp.svg?label=downloads&color=512BD4" alt="NuGet downloads" /></a> <a href="LICENSE"><img src="https://img.shields.io/badge/license-Apache--2.0-green.svg" alt="License: Apache-2.0" /></a> <img src="https://img.shields.io/badge/.NET-10.0-512BD4.svg" alt=".NET 10" /> <img src="https://img.shields.io/badge/core%20dependencies-zero-brightgreen.svg" alt="Zero core dependencies" /> <img src="https://img.shields.io/badge/platforms-Windows%20%7C%20Linux%20%7C%20macOS-informational.svg" alt="Cross-platform" /> </p> </div>

ModelSharp runs real machine-learning models — vision, text, and audio — entirely in managed .NET. No Python, no native DLLs to ship, no per-model glue code. Point it at an ONNX, GGUF, or safetensors model and call one method: a small manifest describes how to feed and decode the model, so the same Pipeline API handles embeddings, image classification, object detection, speech recognition, and quantized LLM generation. A single build runs on Windows, Linux, and macOS, x64 and ARM64 — on CPU, and on the GPU through an optional backend.

using ModelSharp.Hub;

// Downloads the model + tokenizer, then runs it — one line.
using var pipeline = HubPipeline.Load("qwen2.5-0.5b-int4");
string answer = pipeline.Run<string>("The capital of France is");
// → " Paris. It is the largest city in the world by population…"

Contents

• Why ModelSharp
• Features
• Installation
• Quick start
• Supported tasks & formats
• Performance & optional native acceleration
• Verified on real models
• Architecture
• Packages
• Requirements
• License
• Contributing

Why ModelSharp

ONNX Runtime gives you tensor in → tensor out, but every model still needs its own pre/post-processing, and the native runtime can't run everywhere. ModelSharp closes both gaps:

• Self-describing models. A small manifest — embedded ONNX metadata, a sidecar JSON, or a built-in registry — describes how to feed and decode a model. One Pipeline API runs any model: text in or image in, typed result out, no per-model glue code.
• Pure-managed core. Managed kernels mean a single build runs everywhere .NET runs, with no native binaries to ship. Even the ONNX parser is a hand-rolled protobuf reader — there is no Google.Protobuf dependency either.
• Optional native speed. When you want more throughput, drop in the optional native kernel libraries (AVX-512 on CPU, cuBLAS on NVIDIA GPUs). The engine uses them when present and transparently falls back to managed otherwise — so you never trade away portability to get speed.
• Bit-verified. Outputs are validated against ONNX Runtime on real models, down to exact next-token logits on quantized LLMs.

Features

• 🧩 One-line inference — Pipeline.Load("model.onnx").Run<T>(input) for any supported task.
• 📦 Zero dependencies in the core package — no native runtime, no Python, no protobuf library.
• 🌍 Runs everywhere .NET runs — single managed build, x64 / ARM64, all OSes.
• ⚡ Optional native fast path — opt-in AVX-512 / AVX512-VNNI CPU kernels and a cuBLAS GPU path, with automatic managed fallback. (See Performance.)
• 🔢 Multi-dtype engine — float32 / int64 / int32 / bool flow through as their real types, so token ids, masks, and shape tensors all work natively.
• 🧠 Broad operator coverage — CNNs, transformers, RNNs (LSTM/GRU), signal ops (DFT/STFT/MelWeightMatrix), control flow (If/Loop/Scan), sequence/optional ops, and quantized QLinear* / MatMulNBits ops.
• 🧮 Real quantized LLMs — runs INT4 / INT8 / fp16 ONNX LLMs end to end, including a 7B model (MatMulNBits INT4 + genai GroupQueryAttention) loaded from multi-gigabyte external-data files.
• 🔁 Encoder-decoder & decoder-only generation — T5 / BART / MarianMT-style seq2seq (KV-cached decode) alongside decoder-only LLM generation.
• 🔤 Built-in tokenizers — WordPiece (BERT) and byte-level BPE (GPT-2 / RoBERTa), pure managed.
• 🎙️ Audio front end — FFT, log-mel spectrograms, and CTC decoding (greedy + prefix-beam) for ASR.
• ♻️ Runs any model on the GPU — the optional ILGPU backend executes supported ops on-device and falls back to the CPU kernel for the rest, so anything that runs on CPU also runs through the GPU engine.
• ⬇️ Optional model hub — HubPipeline.Load("qwen2.5-0.5b-int4") downloads a model (plus its external-data shards and tokenizer) from Hugging Face, GGUF, safetensors, or any URL and runs it, with a local cache. Pure-managed (HttpClient only).

Installation

dotnet add package ModelSharp              # core: load + run any ONNX model on CPU, plus text & audio front ends
dotnet add package ModelSharp.ImageSharp   # optional: image decoding & classification
dotnet add package ModelSharp.Gpu          # optional: ILGPU GPU backend
dotnet add package ModelSharp.Hub          # optional: download models from Hugging Face / URLs

Requires .NET 10. The core ModelSharp package has no external dependencies.

Quick start

Download & run from the hub

using ModelSharp.Hub;

// Downloads the model + tokenizer/config (and any external-data shards) into a local cache, then runs it.
using var pipeline = HubPipeline.Load("qwen2.5-0.5b-int4");
string answer = pipeline.Run<string>("The capital of France is");

// …or resolve any Hugging Face repo / file, GGUF, safetensors, or direct URL:
ResolvedModel m = ModelHub.Get("onnx-community/Qwen2.5-0.5B-Instruct/onnx/model_q4.onnx");
// m.ModelPath is the local file; m.Files lists the model + tokenizer + config that came with it.

Text embeddings

using ModelSharp.Pipeline;

// The manifest is resolved automatically (sidecar JSON → ONNX metadata → built-in registry).
using var pipeline = Pipeline.Load("all-MiniLM-L6-v2.onnx");

float[] a = pipeline.Run<float[]>("A man is playing a guitar.");
float[] b = pipeline.Run<float[]>("Someone strums an acoustic guitar.");
float[] c = pipeline.Run<float[]>("The stock market fell sharply today.");

// a·b ≈ 0.70 (paraphrase)   a·c ≈ -0.05 (unrelated)

Image classification

using ModelSharp.Pipeline;
using ModelSharp.ImageSharp;

ImageSharpRegistration.Ensure();   // wire the image processors (or just reference the assembly)

using var pipeline = Pipeline.Load("resnet50.onnx");
var results = pipeline.Run<List<Classification>>("cat.jpg");   // also accepts byte[], Stream, or Image<Rgb24>

foreach (var r in results.Take(3))
    Console.WriteLine(r);   // e.g. "tiger cat (82%)"

LLM text generation

using ModelSharp.Pipeline;

using var pipeline = Pipeline.Load("Qwen2.5-0.5B-Instruct-q4.onnx");
string text = pipeline.Run<string>("The capital of France is");
// → " Paris. It is the largest city in the world by population…"

Speech recognition

using ModelSharp.Pipeline;

using var pipeline = Pipeline.Load("whisper-tiny.onnx");
string transcript = pipeline.Run<string>("audio.wav");
// → "Mr. Quilter is the apostle of the middle classes and we are glad to welcome his gospel."

Raw graph execution (full control)

When you want tensors in and tensors out — no manifest, no processors:

using ModelSharp.Onnx;
using ModelSharp.Cpu;
using ModelSharp.Tensors;

ModelGraph graph = OnnxModelLoader.LoadModel("model.onnx");
using var engine = new ManagedCpuEngine(graph);

var feeds = new Dictionary<string, NamedTensor>
{
    ["input_ids"] = new NamedTensor(
        "input_ids",
        Tensor<long>.FromArray(new TensorShape(1, 5), new long[] { 101, 2054, 2003, 2009, 102 })),
};

IReadOnlyDictionary<string, NamedTensor> outputs = engine.Run(feeds);
Tensor<float> hidden = outputs["last_hidden_state"].Tensor.AsFloat();

Supported tasks & formats

Performance & optional native acceleration

ModelSharp's default engine is pure managed and SIMD-tuned. Its register-tiled, multithreaded BlockedGemm (built on System.Numerics.Vector<T>) backs MatMul, Gemm, and Conv, and runs ~2–5× faster than a naïve managed kernel — with zero native code and identical behavior on every platform.

For maximum throughput on supported hardware, ModelSharp ships an optional native kernel layer (under native/, built separately — it is not part of the NuGet packages, which stay pure-managed). The engine loads it when present and falls back to the managed kernels when it is absent, the CPU lacks the required ISA, or the shape isn't supported — so enabling it never costs you portability or correctness.

Highlights of the native layer:

• Portable and safe. The CPU library is built on a portable -mavx2 baseline and chooses AVX-512 / AVX512-VNNI / AVX2 / scalar paths at runtime, so it never executes an unsupported instruction on older x86.
• Resident weights on GPU. Model weights stay on the device across Run() calls instead of being re-uploaded each time, removing the dominant per-call PCIe cost for repeated inference.
• Opt-in via environment flags. MODELSHARP_NATIVE, MODELSHARP_CUBLAS, MODELSHARP_TF32, MODELSHARP_RESIDENT_WEIGHTS. See native/README.md and native/GPU.md for build and tuning details.

Scope. The native layer accelerates ModelSharp's own hot paths and is verified against the managed engine. The managed engine remains the default; the native libraries are an opt-in build for users who want extra throughput on AVX-512 CPUs or NVIDIA GPUs.

Verified on real models

Every supported task is validated end to end on real, exported models — with no Python at inference time and no native dependencies:

Architecture

ModelSharp is built around a single seam: a manifest-driven Pipeline on top of a swappable execution engine.

            +-----------------------------------------------+
  input --> |  Pipeline  (manifest-driven, engine-agnostic) | --> typed result
            +------+--------------------------+--------------+
             IPreprocessor             IPostprocessor
                          |
                          v
                 IExecutionEngine               <-- swappable backend
                    +-- ManagedCpuEngine  (ModelSharp core)  -- pure-managed kernels (+ optional native fast path)
                    +-- IlgpuEngine       (ModelSharp.Gpu)   -- C# kernels -> CUDA / OpenCL / CPU (+ optional cuBLAS)

• Manifest-driven pipeline. A manifest resolves automatically — sidecar JSON next to the model, then embedded ONNX metadata_props, then a built-in registry of filename heuristics — or you pass one explicitly. It selects the right pre/post-processors so one API serves every task.
• Pure-managed, multi-dtype engine. Tensors carry their real dtype end to end. Kernels are written in plain managed C# and are SIMD-friendly, with no native code in the core.
• Swappable backends. The public API and all processing are fixed behind IExecutionEngine; the CPU and GPU engines plug in underneath without changing caller code, and the optional native libraries plug in beneath them with automatic managed fallback.
• Hand-rolled ONNX reader. The loader is a custom protobuf parser, which is why the core package pulls in nothing — no Google.Protobuf, no native runtime.

Need a task ModelSharp doesn't ship? Register your own pre/post-processors with ProcessorRegistry.RegisterPreprocessor / RegisterPostprocessor — exactly how the ImageSharp package plugs itself in.

Packages

Packaging is split by dependency, not by feature — so the common case is a single download.

The optional native acceleration layer (native/) is a separate, opt-in build and is not distributed through any NuGet package — the published packages remain pure-managed.

Requirements

• .NET 10 or later.
• The core ModelSharp package has no external or native dependencies. Optional packages add only the managed dependencies listed above.
• The optional native layer requires a C++17 compiler (and the CUDA toolkit for the GPU library); see native/.

License

Apache License 2.0 — permissive, with an explicit patent grant (the prevailing license across the ML inference ecosystem: ONNX, ONNX Runtime, PyTorch, TensorFlow).

Contributing

Issues and pull requests are welcome. The project targets net10.0; dotnet build and dotnet test build the solution and run the test suite. New operator kernels and model-task processors are wired in behind the stable IExecutionEngine and ProcessorRegistry seams, so contributions extend coverage without breaking the public API.