DataFusionSharp 0.1.1

DataFusionSharp

.NET bindings for Apache DataFusion, a fast, extensible query engine built on Apache Arrow for high-performance analytical query processing.

Note: This is an independent community project and is not officially associated with or endorsed by the Apache Software Foundation or the Apache DataFusion project.

Features

✅ Implemented 🟡 Partially implemented ❌ Not yet implemented

Installation

dotnet add package DataFusionSharp

Quick Start

using DataFusionSharp;

await using var runtime = DataFusionRuntime.Create();
using var context = runtime.CreateSessionContext();

// Register a CSV file as a table
await context.RegisterCsvAsync("orders", "path/to/orders.csv");

// Execute SQL query
using var df = await context.SqlAsync(
    "SELECT customer_id, sum(amount) AS total FROM orders GROUP BY customer_id");

// Display results to console
await df.ShowAsync();

// Access schema
var schema = await df.GetSchemaAsync();
foreach (var field in schema.FieldsList)
    Console.WriteLine($"- {field.Name}: {field.DataType}");

// Collect as Arrow batches
var data = await df.CollectAsync();
foreach (var batch in data.Batches)
{
    // Process Arrow RecordBatch...
}

Requirements

• .NET 8.0 or later
• Supported platforms:
  • Linux (x64, ARM64) 🚧 Planned
  • Windows (x64) 🚧 Planned
  • macOS (ARM64) 🚧 Planned

Building from Source

Prerequisites

• .NET 10.0 SDK or later
• Rust toolchain (1.93+) - Install from https://rustup.rs

Build Steps

1. Clone the repository:

   git clone https://github.com/nazarii-piontko/datafusion-sharp.git
   cd datafusion-sharp
   

2. Build the project:

   dotnet build -c Release
   

   This will automatically:

   • Compile the Rust native library (via cargo)
   • Build the .NET library
   • Link the native library into the managed library

3. Run tests:

   dotnet test -c Release
   

4. (Optional) Run benchmarks:

   cd tests/DataFusionSharp.Benchmark
   dotnet run -c Release
   

   For detailed benchmark results, see the Benchmark README.

Documentation

TODO: Link to or provide documentation for the library

Project Structure

• src/DataFusionSharp/ - Core .NET library with managed wrappers
• native/ - Rust FFI layer bridging .NET to Apache DataFusion
• tests/DataFusionSharp.Tests/ - Integration tests
• tests/DataFusionSharp.Benchmark/ - Performance benchmarks with native reference implementation
• examples/ - Example usage and sample data

Contributing

Contributions are welcome! Please read our Contributing Guide for details on our code of conduct and the process for submitting pull requests.

License

DataFusionSharp is licensed under the Apache License 2.0. See LICENSE.txt for details.

This project contains bindings to Apache DataFusion, which is also licensed under Apache License 2.0. See NOTICE.txt for attribution details.

Acknowledgments

• Apache DataFusion - The underlying query engine
• Apache Arrow - Columnar memory format
• The Apache Software Foundation

Related Projects

• Apache DataFusion - Rust implementation
• datafusion-python - Python bindings
• datafusion-java - Java bindings

Apache®, Apache DataFusion™, Apache Arrow™, and the Apache feather logo are trademarks of The Apache Software Foundation.