Polars.NET.ML 0.5.0

Polars.NET

High-Performance, DataFrame Engine for .NET, powered by Rust & Apache Arrow. With cloud and deltalake features.

Supported Platforms: Windows (x64), Linux (x64/ARM64, glibc/musl), macOS (ARM64). Cloud: AWS, Azure and GCP Data Lake: Delta Lake

<p style="font-size:1.3em; font-weight:bold; background-color:#ffff99;"> <strong>ALMOST</strong> All API in python Polars now is available in both Polars.NET and Polars.FSharp.<br/> The only module left is <strong>config module</strong> which is still under refactoring in rust core.<br/> <strong>Config module will be introduced in next major release.</strong> </p>

Why Polars.NET exists

This is the game I'd like to play: binding the lightning-fast Polars engine to the .NET ecosystem. And it brings a lot of fun.

• Polars.NET vs Python Ecosystem

  <picture> <source media="(prefers-color-scheme: dark)" srcset="assets/benchmark_python_dark.png"> <img alt="Polars.NET vs Python" src="assets/benchmark_python_light.png"> </picture>

• Speedup vs Legacy .NET

  <picture> <source media="(prefers-color-scheme: dark)" srcset="assets/benchmark_summary_dark.png"> <img alt="Speedup Summary" src="assets/benchmark_summary_light.png"> </picture>

• Delta Lake Read

• Delta Lake Write

Installation

C# Users:

dotnet add package Polars.NET 
# And then add the native runtime for your current environment:
dotnet add package Polars.NET.Native.win-x64
# Add LINQ extension package once you need to write LINQ
dotnet add package Polars.NET.Linq

F# Users:

dotnet add package Polars.FSharp
# And then add the native runtime for your current environment:
dotnet add package Polars.NET.Native.win-x64
# Add LINQ extension package once you need to write LINQ or computation expressions
dotnet add package Polars.NET.Linq

• Requirements: .NET 8+.
• Hardware: CPU with AVX2 support (x86-64-v3). Roughly Intel Haswell (2013+) or AMD Excavator (2015+). If you have AVX-512 supported CPU, please try to compile Rust core on your machine use RUSTFLAGS='-C target-cpu=native'

Built Specially for .NET

Bringing .NET to Polars is not enough, it is the time to bring Polars to .NET.

• ML.NET & Tensor InterOp

Prepare data with polars, then train it with ML.NET & ONNX, finally analyze results back with polars.

// ==========================================
// Data Loading
// ==========================================
var hfUrl = "https://huggingface.co/datasets/scikit-learn/iris/resolve/refs%2Fconvert%2Fparquet/default/train/0000.parquet";
var options = CloudOptions.Http(new Dictionary<string, string>
{
    { "User-Agent", "Polars.NET-Test" }
});
using var lf = LazyFrame.ScanParquet(hfUrl, cloudOptions: options);

// sepal length (cm), sepal width (cm), petal length (cm), petal width (cm)     
using var cleanlf = lf.Cast((typeof(double),typeof(float)));
using var cleanDf = cleanlf.WithColumns(Pl.ConcatArray(Cs.Float().ToExpr().Alias("Features"))).Collect();
// ==========================================
// Polars -> ML.NET
// ==========================================
var dataView = cleanDf.AsDataView();

var mlContext = new MLContext(seed: 42);

// ==========================================
// ML.NET Pipeline
// ==========================================
// Form VBuffer<float> tensor
var pipeline = mlContext.Clustering.Trainers.KMeans("Features", numberOfClusters: 3);
var model = pipeline.Fit(dataView);
// ==========================================
// ML.NET Transform and Read Back
// ==========================================
var predictions = model.Transform(dataView);

// ML.NET -> Polars
using var resultDf = predictions.ToDataFrame();

// ==========================================
// TensorInterop
// ==========================================
float[,] matrix = new float[,]
{
    { 1.1f, 1.2f, 1.3f },
    { 2.1f, 2.2f, 2.3f }
};

using var series = Series.From("ffi_matrix", matrix);

var (ptr, shape) = series.AsDangerousUnmanagedTensor<float>();

int totalElements = (int)(shape[0] * shape[1]); 

float* rawFloatPtr = (float*)ptr.ToPointer();

var nativeSpan = new ReadOnlySpan<float>(rawFloatPtr, totalElements);

• ADO.NET

Polars.NET DataReader is generic typed without boxing/unboxing on hot path.

// To DataReader
using var bulkReader = df.AsDataReader(bufferSize: 100, typeOverrides: overrides);
// From DataReader
using var sourceReader = sourceTable.CreateDataReader();
var df = DataFrame.ReadDatabase(sourceReader);

• C# LINQ & F# Computation Expression

With Polars.NET.Linq Extension package(Thanks to Linq2DB), playing DataFrame/Series with LINQ/Query block is available now.

using var dfDepts = DataFrame.From(depts);
using var dfEmps = DataFrame.From(emps);

using var db = new PolarsDataContext(new SqlContext(), ownsContext: true);
var deptQuery = dfDepts.AsQueryable<DeptDto>(db);
var empQuery = empQuery.AsQueryable<EmpDto>(db);

var query = deptQuery
    .LeftJoin(
        empQuery,
        d => d.DeptId,
        e => e.DeptId,
        (d, e) => new 
        {
            d.DeptId,
            d.DeptName,
            EmployeeName = e != null ? e.Name : "NO_EMPLOYEE" 
        })
    .OrderBy(x => x.DeptId)
    .ThenBy(x => x.EmployeeName)
    .Select(x => new JoinResult
    {
        DeptName = x.DeptName,
        EmployeeName = x.EmployeeName
    });

var results = query.ToList();

let queryResult = 
    query {
        for d in deptQuery do
        leftOuterJoin e in empQuery on (d.DeptId = e.DeptId) into empGroup
        for e in empGroup.DefaultIfEmpty() do
        sortBy d.DeptId
        thenBy e.Name
        
        select {|
            DeptName = d.DeptName

            EmployeeName = if box e = null then "NO_EMPLOYEE" else e.Name
        |}
    }
    |> Seq.toList 

Quick Start

C# Example

using Polars.CSharp;
using Pl = Polars.CSharp.Polars;

// 1. Create a DataFrame
var data = new[] {
    new { Name = "Alice", Age = 25, Dept = "IT" },
    new { Name = "Bob", Age = 30, Dept = "HR" },
    new { Name = "Charlie", Age = 35, Dept = "IT" }
};
var df = DataFrame.From(data);

// 2. Filter & Aggregate
var res = df
    .Filter(Pl.Col("Age") > 28)
    .GroupBy("Dept")
    .Agg(
        Pl.Col("Age").Mean().Alias("AvgAge"),
        Pl.Col("Name").Count().Alias("Count")
    )
    .Sort("AvgAge", descending: true);

// 3. Output
res.Show();
// shape: (2, 3)
// ┌──────┬────────┬───────┐
// │ Dept ┆ AvgAge ┆ Count │
// │ ---  ┆ ---    ┆ ---   │
// │ str  ┆ f64    ┆ u32   │
// ╞══════╪════════╪═══════╡
// │ IT   ┆ 35.0   ┆ 1     │
// │ HR   ┆ 30.0   ┆ 1     │
// └──────┴────────┴───────┘

F# Example

open Polars.FSharp

// 1. Scan CSV (Lazy)
let lf = LazyFrame.ScanCsv "users.csv"

// 2. Transform Pipeline
let res = 
    lf
    |> pl.filterLazy (pl.col "age" .> pl.lit 28)
    |> pl.groupByLazy 
        [ pl.col "dept" ]
        [ 
            pl.col("age").Mean().Alias "AvgAge" 
            pl.col("name").Count().Alias "Count"
        ]
    |> pl.sortAscendingLazy [pl.col "AvgAge"]
    |> pl.collect
    |> pl.show

Benchmark

• Benchmark Code

Architecture

3-Layer Architecture ensures API stability.

1. Hand-written Rust C ABI layer bridging .NET and Polars. (native_shim)
2. .NET Core layer for dirty works like unsafe ops, wrappers, LibraryImports. (Polars.NET.Core)
3. High level C# and F# API layer here. No unsafe blocks. (Polars.CSharp & Polars.FSharp)

Roadmap

• Config module

• More code examples, user cases

• Documentation: Docs Here

Contributing

Contributions are welcome. Whether it's adding new expression mappings, improving documentation, or optimizing the FFI layer.

1. Fork the repo.

2. Create your feature branch.

3. Submit a Pull Request.

License

MIT License. See LICENSE for details.