PyDotNet.DataFrames 1.2.0

PyDotNet.DataFrames

A typed DataFrame plugin for PyDotNet — idiomatic .NET access to Pandas and Polars DataFrames with zero-copy Arrow column reads.

Installation

dotnet add package PyDotNet.DataFrames

Pandas and/or Polars must be installed in the active Python environment:

pip install pandas polars pyarrow

Quick start

using PyDotNet.DataFrames;
using PyDotNet.Runtime;

PyRuntime.Initialize(new PyRuntimeOptions { ReleaseGilAfterInit = true });

using var interp = PyRuntime.CreateInterpreter();
using var pd     = PandasModule.Import(interp);

// Create a DataFrame from C# arrays
using var df = pd.FromColumns(new Dictionary<string, Array>
{
    ["product"]  = new string[] { "Apple", "Banana", "Cherry" },
    ["quantity"] = new long[]   { 120L, 85L, 200L },
    ["price"]    = new double[] { 1.20, 0.50, 3.00 },
});

Console.WriteLine($"Rows: {df.RowCount}"); // 3

// Inspect
using var top2 = df.Head(2);                // first 2 rows
using var sorted = df.Sort("price");        // ascending by price
using var cheap  = df.Filter("product", "Banana");

// Column statistics
using var prices = df["price"];
Console.WriteLine($"Mean price: {prices.Mean():F2}");

// Group-by
using var grouped = df.GroupBySum("product", "quantity");

// Export
df.ToCsv("/tmp/products.csv");

// Read a column as managed array
using var qty = df["quantity"];
long total = qty.ToArray<long>().Sum();

// Zero-copy Arrow read
using var reader = df.ToArrowBatches();
foreach (var batch in reader)
{
    ReadOnlySpan<double> p = batch.GetColumn<double>("price");
    // p points directly into Python-owned Arrow buffer — no copy
}

PyRuntime.Shutdown();

API reference

PandasModule

PolarsModule

DataFrame

Series

ArrowBatchReader

Implements IEnumerable<RecordBatch> and IDisposable. Iterates Arrow record batches exported from the DataFrame via __arrow_c_stream__().

RecordBatch

Disposed automatically when the enclosing foreach loop advances or completes.

ColumnInfo

public readonly struct ColumnInfo
{
    public string     Name  { get; }
    public ColumnDType DType { get; }
    public int        Index { get; }
}

ColumnDType

Arrow format codes mapped to .NET names:

Transformations

Row selection, grouping, joining and ordering are backend-neutral: the same call means the same thing whether the frame came from pandas or polars, which is more work than it sounds because the two libraries disagree about more than spelling.

Selecting rows

using var large = sales.Query("amount > 150 and region == 'EU'");

pandas evaluates this with DataFrame.query; polars evaluates it as a SQL WHERE clause. The two dialects agree on ordinary predicates — comparisons, and, or, quoted string literals — which is the range the method is for. Beyond that they diverge: pandas accepts Python expressions and @variable references, polars accepts SQL functions.

For a predicate that must be portable in every detail, build a mask instead. The comparison methods on Series are named identically in both libraries, so there is no divergence to be exposed to:

using var amounts = sales["amount"];
using var mask = amounts.Gt(150.0);
using var large = sales.Filter(mask);

Grouping and aggregating

using var summary = sales
    .GroupBy("region", "quarter")
    .Agg(
        DataFrameAggregation.Sum("revenue", "total"),
        DataFrameAggregation.Mean("margin"),
        DataFrameAggregation.DistinctCount("rep", "reps"));

The result has the key columns first, then one column per aggregation in the order given. Aggregations without an alias are named {column}_{function} — margin_mean above.

That naming is deliberate rather than inherited. Left to themselves, pandas returns the keys as an index and polars returns them as columns, they name aggregated columns differently, and polars does not promise a row order at all. A caller would have to know which library it had. GroupBy pins all three.

DistinctCount is the one aggregate whose method name genuinely differs — nunique in pandas, n_unique in polars.

Joining

using var enriched = sales.Join(regions, "region", DataFrameJoinType.FullOuter);

Naming the join type rather than passing a string is what makes this portable: a full outer join is outer in pandas and full in polars, and polars deprecated the pandas spelling.

Semi and Anti are polars-only. On a pandas frame they raise NotSupportedException naming the alternative, rather than producing a wrong answer or an opaque Python error. CrossJoin is a separate method because a cross join has no key column and both backends reject one being supplied.

Ordering

using var ranked = sales.Sort(
    new DataFrameSortKey("region"),
    new DataFrameSortKey("revenue", Descending: true));

Direction is per column, which a single flag cannot express. pandas asks which columns ascend and polars asks which descend — same information, opposite polarity, inverted here so the caller writes it once.

Writing

ToCsv, ToParquet and ToJson mirror the read paths. ToJson writes an array of row objects on both backends: pandas defaults to_json to a column-oriented layout and polars writes rows, so the orientation is stated explicitly and the output is readable by something that does not know which library wrote it.

Zero-copy usage notes

• GetColumn<T> returns a ReadOnlySpan<T> that points directly into Python-owned memory. The span is valid only until the enclosing RecordBatch is disposed (i.e., until the foreach body exits). Do not store the span.
• The GIL is held during all Python-backed callbacks (schema reads, GetNext, Release).
• Pandas ≥ 3.0 and Polars expose __arrow_c_stream__ natively (SupportsArrow returns true). Pandas 2.x requires pyarrow for the protocol to be available.

Supported frameworks

net8.0 · net9.0 · net10.0