AiDotNet.Tensors 0.136.0

AiDotNet.Tensors

A high-performance .NET tensor library with hand-written AVX2/AVX-512 SIMD kernels in SimdKernels.cs / SimdGemm.cs / SimdConvHelper.cs. Every hot path runs through our own managed-C# kernels — we do NOT call into System.Numerics.Tensors, MKL.NET, or oneDNN through the standard wrappers. In the latest benchmark run on a Threadripper 3990X (see CPU Benchmarks for every number, the versions compared and the losses), AiDotNet.Tensors was faster than TorchSharp 0.107 (libtorch) in 73 of 82 measurements, than ML.NET 5.0 and TensorFlow.NET 0.150 in all of them, than NumSharp 0.70 in 45 of 48 and than MathNet.Numerics 5.0 in all 33.

Note on dependencies. The .nupkg ships with the following PackageReferences: Microsoft.Extensions.Logging.Abstractions, System.Text.Json, System.Threading.Channels, K4os.Compression.LZ4 (LZ4 compression for serialized tensor blobs), AiDotNet.Native.OpenBLAS (transitive native OpenBLAS for fallback paths only — the hot paths run SimdGemm), and MKL via Microsoft.ML.Mkl.Redist (~66 MB on win-x64) + intelmkl.redist.win-x64 (~500 MB on win-x64) for the FP64 kernels that haven't yet been ported to pure-managed AVX2 (Phase 0 remediation work tracks the port). For air-gapped / federal deployments we ship a custom build with MKL/OpenBLAS removed and the entire telemetry namespace compiled out — see aidotnet.dev/enterprise for the Enterprise tier including air-gapped builds.

Performance numbers above assume net8.0+. On net471 the SIMD/intrinsics helpers are excluded (System.Runtime.Intrinsics is unavailable pre-net6); a custom net471 SIMD path to replace System.Numerics.Vector<T> there is on the roadmap as Phase 5.

Features

• Zero Allocations: In-place operations with ArrayPool<T> and Span<T> for hot paths
• Hand-Tuned SIMD: Custom AVX2/FMA kernels with 4-way loop unrolling, not just Vector<T> wrappers
• JIT-Compiled Kernels: Runtime x86-64 machine code generation for size-specialized operations
• BLIS-Style GEMM: Tiled matrix multiply with FMA micro-kernel, cache-aware panel packing
• GPU Acceleration: Optional CUDA, HIP/ROCm, OpenCL, Metal, Vulkan, and WebGPU support via separate packages, with CPU-vs-GPU op-parity validated across every backend (#775)
• Native ANN Index: Dependency-free approximate-nearest-neighbour search (Flat / IVF / PQ / IVFPQ) via AnnIndex, with fused IAnnBackend GPU kernels across all seven backends — no FAISS / MKL dependency (#824)
• Multi-Target: Supports .NET 10.0 and .NET Framework 4.7.1
• Generic Math: Works with any numeric type via INumericOperations<T> interface

Installation

# Core package (CPU SIMD acceleration)
dotnet add package AiDotNet.Tensors

# Optional: OpenBLAS for optimized CPU BLAS operations
dotnet add package AiDotNet.Native.OpenBLAS

# Optional: CLBlast for OpenCL GPU acceleration (AMD/Intel/NVIDIA)
dotnet add package AiDotNet.Native.CLBlast

# Optional: CUDA for NVIDIA GPU acceleration (requires NVIDIA GPU)
dotnet add package AiDotNet.Native.CUDA

Quick Start

using AiDotNet.Tensors.LinearAlgebra;

// Create vectors
var v1 = new Vector<double>(new[] { 1.0, 2.0, 3.0, 4.0 });
var v2 = new Vector<double>(new[] { 5.0, 6.0, 7.0, 8.0 });

// SIMD-accelerated operations
var sum = v1 + v2;
var dot = v1.Dot(v2);

// Create matrices
var m1 = new Matrix<double>(3, 3);
var m2 = Matrix<double>.Identity(3);

// Matrix operations
var product = m1 * m2;
var transpose = m1.Transpose();

CPU Benchmarks

Every number below comes from two BenchmarkDotNet runs on one machine on 2026-10-08: --vs-all at commit dcb0926f and --linalg at commit 587358ed, using the suites in this repository. AiDotNet.Tensors runs its own managed C# SIMD kernels; the comparison libraries run their native backends (libtorch for TorchSharp, the TensorFlow runtime for TensorFlow.NET).

Machine: AMD Ryzen Threadripper 3990X (64 cores / 128 threads, AVX2 + FMA, no AVX-512), Windows 11, .NET 10.0.401, BenchmarkDotNet 0.15.8. BenchmarkDotNet confines each benchmark process to one 64-thread processor group.

Versions compared: TorchSharp 0.107.0 (its bundled libtorch CPU build), ML.NET 5.0.0, TensorFlow.NET 0.150.0 (TensorFlow runtime 2.16.0), NumSharp 0.70.0, MathNet.Numerics 5.0.0.

Summary of this run (a cell is a win when AiDotNet's mean time is lower):

Reproduce:

dotnet run -c Release --project tests/AiDotNet.Tensors.Benchmarks --framework net10.0 -- --vs-all   # TorchSharp, ML.NET, TensorFlow.NET
dotnet run -c Release --project tests/AiDotNet.Tensors.Benchmarks --framework net10.0 -- --linalg   # NumSharp, MathNet

--vs-all-filter <glob> and --linalg-filter <glob> run a subset.

The BenchmarkDotNet reports behind every table are in tests/AiDotNet.Tensors.Benchmarks/Results/2026-10-08-threadripper-3990x/; python tests/AiDotNet.Tensors.Benchmarks/make_readme_tables.py <reports> <reports> regenerates the tables from them.

Updating these numbers. --vs-all and --linalg also write manifest.json (the competitor versions that ran, the commit and runtime) beside the reports. Copy the reports and the manifest into a new Results/<run>/ folder, point the link above at it and paste the regenerated tables. CI runs check_benchmark_claims.py on every pull request: it fails if a table row, count or speedup in this README is not what the reports produce, if a competitor's version differs from the one that ran, or if the package or repository description quotes a speed number.

How to read these numbers

• Steady state is BenchmarkDotNet's normal mode: many calls per iteration, averaged.
• Cold call times one call per iteration with a forced garbage collection before it, which is what an op costs when it runs occasionally. Both libraries' first call (JIT and library start-up) is excluded.
• Each arm matches the competitor's allocation behaviour. Where the other library reuses or frees its output (ML.NET writes into a preallocated destination; TorchSharp disposes its result), the AiDotNet arm returns its result to the tensor pool. Where the other library allocates a fresh result every call (TensorFlow.NET, NumSharp, MathNet), so does the AiDotNet arm. The same op can therefore show very different absolute times in different tables.
• Speedup is the other library's time divided by AiDotNet's; above 1× means AiDotNet is faster.

vs TorchSharp (libtorch CPU)

Where AiDotNet is behind: float and double Log, the double Softmax steady state, ReLU steady state, and the cold call of LayerNorm, Log, Tanh, Mish and the 512×512 MatMul. Most are within 10%; the largest gap is the LayerNorm cold call at 1.32×.

vs ML.NET

vs TensorFlow.NET

vs NumSharp and MathNet.Numerics (double precision)

The three NumSharp losses (Matrix Scalar Multiply and Matrix Subtract at N=500, Transpose at N=1000) are benchmarks that allocate a 2–8 MB result and discard it on every call. Each fresh result lands in memory the OS has to fault in and zero again (about 210 page faults per 2 MB result; the first write costs about twice a rewrite), and the large-object churn triggers gen-2 collections; NumSharp's results are unmanaged and stay out of the GC heap. The computation itself is not the gap: writing into an existing matrix takes 22 µs for the N=500 scalar multiply, and TransposeInPlace at N=1000 takes 125 µs. Recycling discarded result buffers is the follow-up change that addresses this.

Element-wise (vs NumSharp, double):

Small matrix multiply (double):

SIMD Instruction Support

The library automatically detects and uses the best available SIMD instructions:

Check Available Acceleration

using AiDotNet.Tensors.Engines;

var caps = PlatformDetector.Capabilities;

// SIMD capabilities
Console.WriteLine($"AVX2: {caps.HasAVX2}");
Console.WriteLine($"AVX-512: {caps.HasAVX512F}");

// GPU support
Console.WriteLine($"CUDA: {caps.HasCudaSupport}");
Console.WriteLine($"OpenCL: {caps.HasOpenCLSupport}");

// Native library availability
Console.WriteLine($"OpenBLAS: {caps.HasOpenBlas}");
Console.WriteLine($"CLBlast: {caps.HasClBlast}");

// Or get a full status summary
Console.WriteLine(NativeLibraryDetector.GetStatusSummary());

Optional Acceleration Packages

AiDotNet.Native.OpenBLAS

Provides optimized CPU BLAS operations using OpenBLAS:

dotnet add package AiDotNet.Native.OpenBLAS

Performance: Accelerated BLAS operations for matrix multiply and decompositions.

AiDotNet.Native.CLBlast

Provides GPU acceleration via OpenCL (works on AMD, Intel, and NVIDIA GPUs):

dotnet add package AiDotNet.Native.CLBlast

Performance: not benchmarked yet, so no speedup is published for it.

AiDotNet.Native.CUDA

Provides GPU acceleration via NVIDIA CUDA (NVIDIA GPUs only):

dotnet add package AiDotNet.Native.CUDA

Performance: 30,000+ GFLOPS for matrix operations on modern NVIDIA GPUs.

Requirements:

• NVIDIA GPU (GeForce, Quadro, or Tesla)
• NVIDIA display driver 525.60+ (includes CUDA driver)

Usage with helpful error messages:

using AiDotNet.Tensors.Engines.DirectGpu.CUDA;

// Recommended: throws beginner-friendly exception if CUDA unavailable
using var cuda = CudaBackend.CreateOrThrow();

// Or check availability first
if (CudaBackend.IsCudaAvailable)
{
    using var backend = new CudaBackend();
    // Use CUDA acceleration
}

If CUDA is not available, you'll get detailed troubleshooting steps explaining exactly what's missing and how to fix it.

Requirements

• .NET 10.0 or .NET Framework 4.7.1+
• Windows x64, Linux x64, or macOS x64/arm64

License

Apache 2.0 - See LICENSE for details.

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.