AiDotNet.Tensors 0.150.0

AiDotNet.Tensors

A high-performance .NET tensor library with hand-written AVX2/AVX-512 SIMD kernels in SimdKernels.cs / SimdGemm.cs / SimdConvHelper.cs. Every hot path runs through our own managed-C# kernels — we do NOT call into System.Numerics.Tensors, MKL.NET, or oneDNN through the standard wrappers. In the latest benchmark run on a Threadripper 3990X (see CPU Benchmarks for every number, the versions compared and the losses), AiDotNet.Tensors was faster than TorchSharp 0.107 (libtorch) in 73 of 82 measurements, than ML.NET 5.0 and TensorFlow.NET 0.150 in all of them, than NumSharp 0.70 in 45 of 48 and than MathNet.Numerics 5.0 in all 33.

Note on dependencies. The AiDotNet.Tensors package depends on AiDotNet.Evolution, Microsoft.Extensions.Logging.Abstractions, K4os.Compression.LZ4 (LZ4 compression for serialized tensor blobs), BouncyCastle.Cryptography and Silk.NET.Shaderc.Native (the shaderc GLSL to SPIR-V compiler the Vulkan backend uses), plus BCL polyfills on .NET Framework and .NET Standard (System.Text.Json, System.Threading.Channels, and on netstandard2.0 System.Reflection.Emit). It ships no native CPU math library: OpenBLAS and Intel MKL were removed under #368, and by default every CPU GEMM runs the managed kernels.

When oneDNN is used. Intel oneDNN is never bundled with AiDotNet.Tensors; it ships separately as AiDotNet.Native.OneDNN (oneDNN 3.2, dnnl.dll for win-x64). On net8.0 and net10.0 only (the oneDNN path is compiled out of the net471, net462 and netstandard2.0 builds), when a dnnl library can be loaded - from that package, the application directory, or the system library search path - the CPU engine tries oneDNN first for float32 Conv2D forward, in-place ReLU and sigmoid, softmax, and large element-wise add and multiply. GEMM stays on the managed kernels unless AIDOTNET_ONEDNN_GEMM=1 is set. Any call oneDNN declines falls back to the managed kernel, and so does every call in deterministic mode or while another thread is already inside oneDNN. AIDOTNET_DISABLE_ONEDNN=1 turns oneDNN off even when the library is present.

Performance numbers above assume net8.0+. On net471 the SIMD/intrinsics helpers are excluded (System.Runtime.Intrinsics is unavailable pre-net6); a custom net471 SIMD path to replace System.Numerics.Vector<T> there is on the roadmap as Phase 5.

Features

• Zero Allocations: In-place operations with ArrayPool<T> and Span<T> for hot paths
• Hand-Tuned SIMD: Custom AVX2/FMA kernels with 4-way loop unrolling, not just Vector<T> wrappers
• JIT-Compiled Kernels: Runtime x86-64 machine code generation for size-specialized operations
• BLIS-Style GEMM: Tiled matrix multiply with FMA micro-kernel, cache-aware panel packing
• GPU Acceleration: Optional CUDA, HIP/ROCm, OpenCL, Metal, Vulkan, and WebGPU support via separate packages, with CPU-vs-GPU op-parity validated across every backend (#775)
• Native ANN Index: Dependency-free approximate-nearest-neighbour search (Flat / IVF / PQ / IVFPQ) via AnnIndex, with fused IAnnBackend GPU kernels across all seven backends — no FAISS / MKL dependency (#824)
• Multi-Target: net10.0, net8.0, net471, net462 and netstandard2.0
• Generic Math: Works with any numeric type via INumericOperations<T> interface

Installation

# Core package (CPU SIMD acceleration)
dotnet add package AiDotNet.Tensors


# Optional: CLBlast for OpenCL GPU acceleration (AMD/Intel/NVIDIA)
dotnet add package AiDotNet.Native.CLBlast

# Optional: CUDA for NVIDIA GPU acceleration (requires NVIDIA GPU)
dotnet add package AiDotNet.Native.CUDA

Published packages

These packages are published to nuget.org:

Each AiDotNet.Native.* package carries the upstream licence text of what it redistributes and a NATIVE-MANIFEST.json listing the upstream version, licence and SHA-256 of every native file. For CUDA, the licence and manifest are in the AiDotNet.Native.CUDA package itself: its AiDotNet.Native.CUDA.Fragments.N dependencies carry only byte chunks of the DLLs, which the meta package reassembles and whose SHA-256 the manifest records. hipblas.dll in AiDotNet.Native.ROCm states no version of its own, so its entry names the HIP SDK release it came from. The one licence exception is the Windows HIP runtime in AiDotNet.Native.ROCm (amdhip64_6.dll): it is closed source under AMD's HIP SDK licence, which AMD does not publish as a text file, so its text is not bundled; the package's LICENSE.txt names the binary, its version and SHA-256, and where AMD shows the licence.

The NuGet licence of each AiDotNet.Native.* GPU package (CUDA, its CUDA.Fragments.N packages, and ROCm) is that bundled LICENSE.txt, not this repository's licence, because the binaries are NVIDIA's and AMD's. Only the packaging glue (MSBuild targets and scripts) is under this repository's licence.

AiDotNet.Tensors.Gpu, AiDotNet.Tensors.Onnx, AiDotNet.Tensors.Cli and AiDotNet.Tensors.Analyzers are built from this repository but are not published to NuGet. The analyzers run only when this repository builds.

Quick Start

using AiDotNet.Tensors.LinearAlgebra;

// Create vectors
var v1 = new Vector<double>(new[] { 1.0, 2.0, 3.0, 4.0 });
var v2 = new Vector<double>(new[] { 5.0, 6.0, 7.0, 8.0 });

// SIMD-accelerated operations
var sum = v1 + v2;
var dot = v1.Dot(v2);

// Create matrices
var m1 = new Matrix<double>(3, 3);
var m2 = Matrix<double>.Identity(3);

// Matrix operations
var product = m1 * m2;
var transpose = m1.Transpose();

CPU Benchmarks

Every number below comes from two BenchmarkDotNet runs on one machine on 2026-10-08: --vs-all at commit dcb0926f and --linalg at commit 587358ed, using the suites in this repository. AiDotNet.Tensors runs its own managed C# SIMD kernels; the comparison libraries run their native backends (libtorch for TorchSharp, the TensorFlow runtime for TensorFlow.NET).

Machine: AMD Ryzen Threadripper 3990X (64 cores / 128 threads, AVX2 + FMA, no AVX-512), Windows 11, .NET 10.0.401, BenchmarkDotNet 0.15.8. BenchmarkDotNet confines each benchmark process to one 64-thread processor group.

Versions compared: TorchSharp 0.107.0 (its bundled libtorch CPU build), ML.NET 5.0.0, TensorFlow.NET 0.150.0 (TensorFlow runtime 2.16.0), NumSharp 0.70.0, MathNet.Numerics 5.0.0.

Summary of this run (a cell is a win when AiDotNet's mean time is lower):

Reproduce:

dotnet run -c Release --project tests/AiDotNet.Tensors.Benchmarks --framework net10.0 -- --vs-all   # TorchSharp, ML.NET, TensorFlow.NET
dotnet run -c Release --project tests/AiDotNet.Tensors.Benchmarks --framework net10.0 -- --linalg   # NumSharp, MathNet

--vs-all-filter <glob> and --linalg-filter <glob> run a subset.

The BenchmarkDotNet reports behind every table are in tests/AiDotNet.Tensors.Benchmarks/Results/2026-10-08-threadripper-3990x/; python tests/AiDotNet.Tensors.Benchmarks/make_readme_tables.py <reports> <reports> regenerates the tables from them.

Updating these numbers. --vs-all and --linalg also write manifest.json (the competitor versions that ran, the commit and runtime) beside the reports. Copy the reports and the manifest into a new Results/<run>/ folder, point the link above at it and paste the regenerated tables. CI runs check_benchmark_claims.py on every pull request: it fails if a table row, count or speedup in this README is not what the reports produce, if a competitor's version differs from the one that ran, or if the package or repository description quotes a speed number.

How to read these numbers

• Steady state is BenchmarkDotNet's normal mode: many calls per iteration, averaged.
• Cold call times one call per iteration with a forced garbage collection before it, which is what an op costs when it runs occasionally. Both libraries' first call (JIT and library start-up) is excluded.
• Each arm matches the competitor's allocation behaviour. Where the other library reuses or frees its output (ML.NET writes into a preallocated destination; TorchSharp disposes its result), the AiDotNet arm returns its result to the tensor pool. Where the other library allocates a fresh result every call (TensorFlow.NET, NumSharp, MathNet), so does the AiDotNet arm. The same op can therefore show very different absolute times in different tables.
• Speedup is the other library's time divided by AiDotNet's; above 1× means AiDotNet is faster.

vs TorchSharp (libtorch CPU)

Where AiDotNet is behind: float and double Log, the double Softmax steady state, ReLU steady state, and the cold call of LayerNorm, Log, Tanh, Mish and the 512×512 MatMul. Most are within 10%; the largest gap is the LayerNorm cold call at 1.32×.

vs ML.NET

vs TensorFlow.NET

vs NumSharp and MathNet.Numerics (double precision)

The three NumSharp losses (Matrix Scalar Multiply and Matrix Subtract at N=500, Transpose at N=1000) are benchmarks that allocate a 2–8 MB result and discard it on every call. Each fresh result lands in memory the OS has to fault in and zero again (about 210 page faults per 2 MB result; the first write costs about twice a rewrite), and the large-object churn triggers gen-2 collections; NumSharp's results are unmanaged and stay out of the GC heap. The computation itself is not the gap: writing into an existing matrix takes 22 µs for the N=500 scalar multiply, and TransposeInPlace at N=1000 takes 125 µs. Recycling discarded result buffers is the follow-up change that addresses this.

Element-wise (vs NumSharp, double):

Small matrix multiply (double):

SIMD Instruction Support

The library automatically detects and uses the best available SIMD instructions:

Check Available Acceleration

using AiDotNet.Tensors.Engines;

var caps = PlatformDetector.Capabilities;

// SIMD capabilities
Console.WriteLine($"AVX2: {caps.HasAVX2}");
Console.WriteLine($"AVX-512: {caps.HasAVX512F}");

// GPU support
Console.WriteLine($"CUDA: {caps.HasCudaSupport}");
Console.WriteLine($"OpenCL: {caps.HasOpenCLSupport}");

// Native library availability
Console.WriteLine($"CLBlast: {caps.HasClBlast}");

// Or get a full status summary
Console.WriteLine(NativeLibraryDetector.GetStatusSummary());

Optional Acceleration Packages

AiDotNet.Native.CLBlast

Provides GPU acceleration via OpenCL (works on AMD, Intel, and NVIDIA GPUs):

dotnet add package AiDotNet.Native.CLBlast

Performance: not benchmarked yet, so no speedup is published for it.

AiDotNet.Native.CUDA

Provides GPU acceleration via NVIDIA CUDA (NVIDIA GPUs only):

dotnet add package AiDotNet.Native.CUDA

Performance: not benchmarked yet, so no speedup is published for it.

Requirements:

• NVIDIA GPU (GeForce, Quadro, or Tesla)
• NVIDIA display driver 525.60+ (includes CUDA driver)

Usage with helpful error messages:

using AiDotNet.Tensors.Engines.DirectGpu.CUDA;

// Recommended: throws beginner-friendly exception if CUDA unavailable
using var cuda = CudaBackend.CreateOrThrow();

// Or check availability first
if (CudaBackend.IsCudaAvailable)
{
    using var backend = new CudaBackend();
    // Use CUDA acceleration
}

If CUDA is not available, you'll get detailed troubleshooting steps explaining exactly what's missing and how to fix it.

Requirements

• .NET 8.0 or later, .NET Framework 4.6.2 or later, or another runtime that implements .NET Standard 2.0
• Windows x64, Linux x64, or macOS x64/arm64

License

AiDotNet.Tensors is licensed under the Business Source License 1.1 (BSL 1.1), with CrowdTrainer LLC as Licensor. Its Additional Use Grant allows any non-commercial use (education, research, personal projects and internal evaluation), and training and inference operations are unrestricted regardless of license status. Each version converts to the Apache License 2.0 four years after its release. See aidotnet.dev/license for the license tiers, including the free Community license.

The BSL applies to versions released after this change (0.145.0 and later). Versions published before it remain under the Apache License 2.0.

Parts derived from CLBlast and ONNX remain under the Apache License 2.0; see THIRD-PARTY-NOTICES.txt. The AiDotNet.Native.* packages carry their vendors' licences.

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.