JYPPX.DeploySharp.Backend.TensorRT 2.0.0

<picture> <source media="(prefers-color-scheme: dark)" srcset="docs/images/readme/hero-dark.svg"> <source media="(prefers-color-scheme: light)" srcset="docs/images/readme/hero-light.svg"> <img alt="DeploySharp - reproducible AI inference workflows for .NET" src="docs/images/readme/hero-light.svg" width="100%"> </picture>

<p align="center">A modular .NET model deployment toolkit for reproducible vision, language, and multimodal inference across replaceable backends.</p>

<p align="center"> <a href="https://github.com/guojin-yan/DeploySharp/actions/workflows/ci.yml?query=branch%3ADeploySharpV2.0"><img src="https://github.com/guojin-yan/DeploySharp/actions/workflows/ci.yml/badge.svg?branch=DeploySharpV2.0" alt="Windows CI" /></a> <a href="https://github.com/guojin-yan/DeploySharp/blob/DeploySharpV2.0/LICENSE.txt"><img src="https://img.shields.io/badge/License-Apache%202.0-blue.svg" alt="Apache-2.0 license" /></a> <a href="https://github.com/guojin-yan/DeploySharp/stargazers"><img src="https://img.shields.io/github/stars/guojin-yan/DeploySharp?style=flat&label=stars" alt="GitHub stars" /></a> <a href="https://dotnet.microsoft.com/"><img src="https://img.shields.io/badge/.NET-net46%20to%20net10.0-512BD4" alt=".NET Framework 4.6 through .NET 10" /></a> <a href="docs/articles/platform-support.md"><img src="https://img.shields.io/badge/platform-Windows%20x64-0078D4" alt="Windows x64 release scope" /></a> </p>

<p align="center"> <a href="docs/index.md"><img src="https://img.shields.io/badge/docs-DocFX-2f80ed" alt="DocFX documentation" /></a> <a href="https://www.nuget.org/packages/JYPPX.DeploySharp.Core/"><img src="https://img.shields.io/nuget/vpre/JYPPX.DeploySharp.Core.svg?label=NuGet" alt="Latest DeploySharp NuGet version" /></a> <a href="https://github.com/guojin-yan/DeploySharp/releases"><img src="https://img.shields.io/github/v/release/guojin-yan/DeploySharp?label=GitHub%20Release" alt="Latest stable GitHub Release" /></a> </p>

<p align="center"><strong>English</strong> | <a href="README_cn.md">简体中文</a></p>

DeploySharp

DeploySharp V2 provides explicit contracts for model artifacts, typed tensors, sessions, visual pipelines, language/multimodal workflows, ModelPack integrity, ModelFactory acquisition, and replaceable inference backends. Applications own model files and native runtimes; DeploySharp keeps backend selection and execution boundaries visible.

📖 Introduction

• Stable application contracts: model identity, typed tensors, named inputs/outputs, sessions, diagnostics, cancellation, and disposal.
• Typed workflows and extension points: classification, detection, segmentation, pose, OBB, OCR, anomaly, promptable segmentation, vision-language, LLM, and multimodal paths, with model-level maturity documented separately.
• Explicit backend ownership: ONNX Runtime, OpenVINO, OpenCV DNN, TensorRT/CUDA, and LLamaSharp adapters without silently installing every vendor runtime.
• Reproducible model delivery: ModelPack manifests, artifact size/SHA-256 checks, versioned Release downloads, offline cache reuse, and a runnable model case.

The V2 API is a clean redesign and does not provide V1 source, binary, configuration, or behavior compatibility.

✨ Release Highlights

• Core, Visual, LLM, Multimodal, ModelPack, ModelFactory, five backend families, and seven grouped sample modules.
• The model catalog, model/backend verification matrix, and named-device measurements are maintained as public documents.
• Windows x64 verification for ONNX Runtime, OpenVINO, OpenCV DNN, and named TensorRT/CUDA environments.
• Session pools, batching, asynchronous visual inference, sliding-window detection, and repeatable benchmark tooling.

📢 Latest Update: 2.0.0

2.0.0 is the first stable DeploySharp V2 release target, covering typed inference, visual workflows, PaddleOCR/PP-Structure, model downloads, and replaceable backends. Its release validation scope is Windows 10/11 x64; model and backend limits remain explicit. See the 2.0.0 release notes for the complete scope and upgrade guidance.

The 2.0.0 release comes from the DeploySharpV2.0 branch. The NuGet badges and GitHub Releases show available packages and assets. Upgrade all DeploySharp packages together and recompile applications built against 2.0.0-alpha.1; see the upgrade guidance and preview release notes for version-specific changes.

🚀 Get Started In 30 Seconds

Install the Core layer and the backend you need at the same 2.0.0 version. Contributors can use project references to work with the source.

dotnet add package JYPPX.DeploySharp.Core --version 2.0.0
dotnet add package JYPPX.DeploySharp.Backend.OnnxRuntime --version 2.0.0
dotnet add package Microsoft.ML.OnnxRuntime --version 1.28.0

Create a model artifact, register ONNX Runtime, create a named-tensor session, and run one typed input:

using System.Threading;
using JYPPX.DeploySharp;
using JYPPX.DeploySharp.Backends.OnnxRuntime;
using JYPPX.DeploySharp.Models;
using JYPPX.DeploySharp.Registry;
using JYPPX.DeploySharp.Tensors;

using var backends = new BackendRegistry();
backends.UseOnnxRuntime();

var artifact = new ModelArtifact(
    new ModelId("examples/classifier"),
    "onnx",
    @"models\classifier.onnx",
    preferredBackend: OnnxRuntimeBackendProvider.BackendId);
var request = new BackendRequest(
    BackendCapabilities.TensorInference,
    OnnxRuntimeBackendProvider.BackendId,
    "cpu");
using IInferenceSession session = backends.CreateSession(
    artifact, request, SessionOptions.Default);

var input = new Tensor<float>(new TensorShape(1, 3), new[] { 0.1f, 0.2f, 0.7f });
InferenceOutputs outputs = session.Run(InferenceInputs.Create("images", input), CancellationToken.None);
Console.WriteLine(outputs.Count);

The code-first path, visual preparation, ModelFactory download flow, and model-specific examples are in the usage tutorial and samples.

📦 Package Layout

Recommended Package Combinations

See the detailed package combination and installation guide for commands, TFM/RID notes, and runtime ownership.

🌐 Public Packages And Release Assets

The version column reads NuGet directly and includes previews. It shows the latest published version and updates automatically when 2.0.0 becomes available.

Package	NuGet version	Purpose
JYPPX.DeploySharp.Core		Core contracts and backend registration
JYPPX.DeploySharp.Extensibility		Plugin descriptors, runtime probes, and options schemas
JYPPX.DeploySharp.Visual		Visual profiles, preprocessing, and decoders
JYPPX.DeploySharp.Visual.OpenCV		OpenCV image preparation
JYPPX.DeploySharp.Visual.TensorRT		CUDA preprocessing and TensorRT visual pipelines
JYPPX.DeploySharp.LLM		LLM generation and embedding contracts
JYPPX.DeploySharp.Multimodal		Ordered multimodal orchestration
JYPPX.DeploySharp.ModelPack.Json		Model manifests and integrity validation
JYPPX.DeploySharp.ModelFactory		Catalog selection, download, cache, and offline reuse
JYPPX.DeploySharp.Backend.OnnxRuntime		ONNX Runtime named-tensor adapter
JYPPX.DeploySharp.Backend.OpenVINO		OpenVINO named-tensor adapter
JYPPX.DeploySharp.Backend.OpenCV		OpenCV DNN adapter
JYPPX.DeploySharp.Backend.TensorRT		TensorRT inference and ONNX-to-engine boundary
JYPPX.DeploySharp.Backend.LlamaSharp		LLamaSharp GGUF generation and embeddings

Default Test Images

Examples and benchmark tools use the dedicated test-assets.1 release when no local image is supplied. The task defaults are bus.jpg for detection/segmentation, demo_7.jpg for classification, demo_9.jpg for pose, plane.png for oriented detection, and ocr-demo_1.jpg for PaddleOCR. Files are SHA-256 verified and cached under %LOCALAPPDATA%\DeploySharp\TestImages; the full mapping and maintenance commands are in the default test images guide.

Application-Owned Runtime Packages

These are application dependencies/runtime packages used by Windows x64 applications. Referencing a DeploySharp managed package does not silently install them:

Package	NuGet	Purpose
Microsoft.ML.OnnxRuntime		ONNX Runtime CPU native execution
JYPPX.OpenCV.runtime.win-x64		Windows x64 OpenCV native runtime
OpenVINO.runtime.win		Windows OpenVINO native runtime
JYPPX.TensorRT.CSharp.API		Managed TensorRT/CUDA API; NVIDIA libraries remain application-installed
LLamaSharp.Backend.Cpu		CPU native backend for LLamaSharp GGUF workflows

The managed package tables do not mean native dependencies are installed automatically. See installation and runtime ownership before selecting a deployment RID.

🖥️ Platforms And Frameworks

The complete framework list and backend evidence are in platform and backend support. Build compatibility is not the same as inference verification.

🤖 Supported Models

The table lists public model assets by task. "Backend verified" means that at least one exact model/backend combination has real inference evidence; it never means every backend is supported. Model files are downloaded separately and are not embedded in NuGet packages.

Use the model support guide for the detailed current and planned scope, the official catalog for immutable assets, and the model/backend matrix for exact runtime evidence. PaddleOCR and PP-Structure assets are individually downloadable from models-paddleocr.

🧪 Example Series

See the sample learning path.

📚 Documentation

🔨 Build From Source

dotnet restore DeploySharp.sln --locked-mode
dotnet build DeploySharp.sln -c Release --no-restore -m:1
dotnet test DeploySharp.sln -c Release --no-build --no-restore

⚖️ License

DeploySharp source code is licensed under the Apache License 2.0. Models and vendor runtimes are separate artifacts with their own runtime and distribution terms.

🤝 Contact And Sponsorship

For questions, issue reports, testing feedback, or sponsorship, please use the project homepage and issue tracker.

<p align="center"> <img src="docs/images/readme/contact-support-en.png" width="100%" alt="Developer contact channels, community entry points, and sponsorship QR codes"> </p>

⚠️ Software Notice And Disclaimer

📜 1. Open Source License Notice

All open-source project code authored by the project author follows the Apache License 2.0.

Special note: This project integrates several third-party libraries. If any third-party library uses a license that conflicts with or differs from Apache 2.0, the original license of that third-party library takes precedence. This project does not include or represent the license notices of those third-party libraries. Read and comply with the relevant third-party licenses before use.

🤖 2. Code Development And Quality Notice

• AI-assisted development: AI assistance was used to generate and improve parts of this code during development; it was not written entirely line by line by a human.
• Security statement: The author solemnly declares that this code contains no intentionally planted backdoors, viruses, Trojans, or malicious code intended to damage user devices or steal data.
• Technical limitations: Due to the author's technical level and capabilities, the code may contain basic issues caused by insufficiently rigorous logic, incomplete optimization, or limited experience, including but not limited to memory leaks, occasional crashes, and unreleased resources. These issues are the result of limitations in ability and are not intentional.
• Testing scope: Because the author's time is limited, this software has not been fully tested across every scenario and edge case.

🚨 3. Important Disclaimer

Before applying this code to any real-world project, especially a commercial, industrial, or mission-critical environment, you must perform thorough and rigorous independent testing and validation. In view of the possible defects and limited test coverage described above, the author accepts no responsibility for any direct or indirect loss caused by using this code, including but not limited to equipment failure, data loss, system outage, or loss of profits. By using this code, you acknowledge these risks and agree to bear all consequences independently; related issues are not the responsibility of the author.

🔓 4. Scope Of Open Source Code

The core logic of this project is fully open source. However, the binary files, source code, and related resources of the third-party libraries mentioned above are outside this project's open-source obligations. Obtain and use them according to their respective instructions and licenses.

🤝 5. Community And Feedback

Despite these limitations, everyone is welcome to download and use the project, submit Issues, and participate in testing. If you find a bug, memory overflow, or improvement opportunity, please use the contact channels on the project homepage. We will do our best to provide assistance within the time available.