TensorSharp.Server 2026.10.3

TensorSharp

<p align="center"> <img src="imgs/banner_1.png" alt="TensorSharp logo" width="320"> </p>

English | 中文

TensorSharp is a .NET 10 inference engine for local GGUF models. Run it on Windows, macOS or Linux through the CLI, browser chat, or Ollama/OpenAI-compatible APIs, or embed it in your own .NET application. Use managed C# CPU kernels or native CUDA, Metal and Vulkan backends, with support that varies by model.

Current source covers text and reasoning, multimodal input, embeddings, image generation/editing, video with audio, and Agent Skills with code tools. TensorSharp also powers TensorAgent, the local app for iPhone, iPad, Mac and Windows. Start below, or check the project status for capabilities and validation limits; source changes may be ahead of published packages.

Highlights

• Local models, several interfaces. One engine for CLI use, browser chat and Ollama/OpenAI-compatible APIs, across managed CPU and native accelerator backends.
• Text and multimodal models. Dense and MoE GGUF models, reasoning, image/audio input and document questions. See Supported models for each family's capabilities.
• Embeddings and media generation. Text/code embeddings, Qwen-Image-2.1 generation and masked editing, and video models including MiniMax-H3 with audio.
• Efficient serving. Continuous batching and a paged, Radix prefix-shared KV cache are on by default. Speculative decoding and multi-GPU placement are available for supported models. See Features.
• Agent Skills and code tools. TensorSharp.AgentHost adds file, shell and document workflows. Server and TensorAgent chats also support bounded sub-agent delegation with private workspaces and read-only defaults.
• Recorded comparisons. Benchmarks against llama.cpp use identical GGUF files and hardware; results apply to the measured model, backend and workload. See Benchmarks.
• TensorAgent apps. Local chat, attachments, skills and saved work on phones and desktops. See the Mac/Windows installation guide and app coverage notes.

Quick Start

TensorSharp CLI and server

The Releases page provides self-contained CLI and Server archives for Windows x64 (CPU/CUDA), Linux x64 (CPU/CUDA), and macOS arm64.

To build from source you need the full .NET 10 SDK (how to install it), git, curl, CMake 3.20+, and the toolchain for your GPU. Then run the verified Gemma 4 E4B model (7.48 GiB). On Windows with an NVIDIA GPU (PowerShell):

git clone https://github.com/zhongkaifu/TensorSharp.git; Set-Location TensorSharp
New-Item -ItemType Directory -Force models | Out-Null
curl.exe -L --fail "https://huggingface.co/ggml-org/gemma-4-E4B-it-GGUF/resolve/main/gemma-4-E4B-it-Q8_0.gguf?download=true" -o models\gemma-4-E4B-it-Q8_0.gguf
'Answer in one short sentence: what is TensorSharp?' | Set-Content prompt.txt
$env:TENSORSHARP_GGML_NATIVE_ENABLE_CUDA = 'ON'
dotnet run --project TensorSharp.Cli -c Release -p:TensorSharpSkipMlxNative=true -- --model models\gemma-4-E4B-it-Q8_0.gguf --input prompt.txt --max-tokens 128 --backend ggml_cuda

On other machines, change the backend (see Pick a backend):

• macOS (Apple Silicon): drop the CUDA environment variable and use --backend ggml_metal.
• Linux + NVIDIA: prefix the dotnet run with TENSORSHARP_GGML_NATIVE_ENABLE_CUDA=ON and use --backend ggml_cuda.
• AMD / Intel / NVIDIA Vulkan: set TENSORSHARP_GGML_NATIVE_ENABLE_VULKAN=ON and use --backend ggml_vulkan.

Host the same model as a server: a browser chat at http://localhost:5000 plus Ollama- and OpenAI-compatible APIs.

dotnet run --project TensorSharp.Server.Host -c Release -p:TensorSharpSkipMlxNative=true -- --model models/gemma-4-E4B-it-Q8_0.gguf --backend ggml_cuda --max-tokens 512

The server listens on 0.0.0.0:5000 with no built-in authentication or TLS; keep it behind a firewall or an authenticated HTTPS reverse proxy.

Pick a backend

The Getting started guide has the rest: installing the SDK on each platform, multi-GPU and multi-node runs, NVIDIA DGX Spark, multimodal input, embeddings, and making it fast. Every option is in the CLI and Server references, and both programs print them with --help.

dotnet build TensorSharp.slnx also builds TensorAgent's available desktop heads and the iOS simulator head on Apple Silicon when the selected SDK has the required MAUI workloads and staged native/Python files. Missing prerequisites skip the affected app head with a warning; see TensorAgent build instructions.

TensorAgent Desktop: download, install, chat

Download the latest release and expand Assets. Choose a file beginning with tensoragent-desktop-:

The app includes its .NET runtime and native engine. Windows needs WebView2 Evergreen Runtime if missing; Python/Node are optional tools for skills. The current built-in catalog needs at least 12 GB system RAM, and model weights download separately. Open ☰ → Models → Download → Use, then type a message. The Desktop user guide covers package verification, unsigned-app prompts, first-run setup, attachments, skills, updates and troubleshooting. Historical releases may have no Desktop assets until a release runs the updated workflow; iPhone/iPad remain source builds.

Supported model families at a glance

• Text, reasoning, and multimodal LLMs: DeepSeek V4 Flash / V4.1 Flash, GLM 5.x, Gemma 4, Qwen 3.5 / 3.6 / 3.8 27B, Qwen 3.8 Flash Next, Bonsai2 (Qwen family), GPT OSS, Nemotron-H, Mistral 3, Hunyuan Dense, and Muse-Glimmer.
• Text diffusion: DiffusionGemma, including Jev typed decision inference at /v1/systemone, over text, images, uploaded documents, sampled video frames and audio transcripts (configured ASR companion).
• Image generation/editing and video generation: Qwen-Image-2.1, MiniMax-H3 (video + stereo audio), and Wan 2.1 / 2.2.
• Text and code embeddings: BERT / XLM-R encoders — Snowflake Arctic Embed L v2.0 and all-MiniLM-L6-v2.

Backend, modality, feature support, and validation coverage vary by model. See the supported models tables, the model cards, and the embedding guide for details.

Recent source additions include Qwen-Image-2.1 masked edits with exact protected pixels and optional processing of the selected region, twelve TensorAgent LoRA plug-ins for speed, style and editing, and Qwen3.8 Flash Next on a 48 GB Mac using SSD-backed weights. Multi-GPU --layer-split and --tp are separate controls; support and performance depend on the architecture and quantization. These source features may be ahead of published packages; Desktop installers appear in releases built with the updated Release Binaries workflow, while older releases may lack them.

See it in action

One engine, four ways to use it, each an unedited capture of a real run.

<table> <tr> <td align="center" width="50%"><img src="website/assets/screenshots/tensorsharp-cli.png" alt="TensorSharp.Cli in a terminal: an interactive chat with Gemma 4 E4B that reads this README and answers questions about it" width="250"><br><b>TensorSharp.Cli</b><br>Models in your terminal</td> <td align="center" width="50%"><img src="website/assets/screenshots/tensorsharp-webui.png" alt="The TensorSharp Web UI: Qwen3.8 27B compared two mortgages by writing and running a Python script" width="400"><br><b>TensorSharp.Server.Host</b><br>Web UI chat and Ollama/OpenAI-compatible APIs</td> </tr> <tr> <td align="center"><img src="website/assets/screenshots/tensoragent-iphone.png" alt="TensorAgent in the iPhone 17 Pro simulator: Gemma 4 E2B uses ggml_cpu and runs an in-app Python script to scale a recipe" width="140"><br><b>TensorAgent · iPhone simulator</b><br>CPU inference and in-app Python; this is a simulator capture</td> <td align="center"><img src="website/assets/screenshots/tensoragent-mac.png" alt="TensorAgent on a Mac: a saved Qwen-Image 2.1 edit changes the TensorSharp banner background to a starry blue night sky" width="400"><br><b>TensorAgent on Mac</b><br>A saved Qwen-Image 2.1 edit in the Mac app</td> </tr> </table>

What each run shows, step by step: Screenshots.

Benchmarks

TensorSharp and llama.cpp run identical GGUF files on the same NVIDIA RTX 3080 Laptop GPU (16 GB), each on its GGML CUDA and Vulkan builds. Each number is TensorSharp's speedup over llama.cpp on the same backend (geomean, single-stream, greedy, MTP off); above 1.0× means TensorSharp is faster.

What these numbers mean, how to rerun them, and the head-to-heads of models too large for this GPU: Benchmarks.

Documentation

New here? The sections above are all you need to get running. Everything else is detailed reference:

Current Status

Actively developed, and the source tree runs ahead of the published packages.

Per-area detail (which architecture runs on which backend, which features each family supports, and the known limits) is in the status matrix.

Author

Zhongkai Fu

License

See LICENSE for details.

Learn with the books

Explore both books and their repository reading paths

<p align="center"><a href="https://buymeacoffee.com/zhongkaifu"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me A Coffee" height="50"></a><br> _{TensorSharp/TensorAgent is free. If you like it, a coffee keeps the work on it going.}</p>