Noundry.Slurp.Tool 1.0.0

Noundry.Slurp

High-performance CSV to Database ingestion tool built for speed, flexibility, and ease of use. Slurp can ingest millions of rows in seconds while automatically inferring schema, creating tables, and building intelligent indexes.

Features

• Blazing Fast - Built on the SEP library, the world's fastest .NET CSV parser
• Multi-Database Support - SQL Server, PostgreSQL, MySQL, and SQLite out of the box
• Smart Schema Inference - Automatically detects column types, sizes, and nullable constraints
• Intelligent Indexing - Automatically creates indexes on ID columns, foreign keys, and frequently filtered columns
• Beautiful CLI - Interactive prompts with Spectre.Console for a delightful user experience
• Parallel Processing - Multi-threaded ingestion for massive files
• Flexible Configuration - JSON config files or command-line arguments
• Library + CLI - Use as a NuGet package or standalone CLI tool
• .NET 8.0, 9.0 & 10.0 Support - Works with .NET 8.0, 9.0, and 10.0 runtimes

Installation

As a .NET Tool

dotnet tool install --global Noundry.Slurp.Tool

As a NuGet Package

dotnet add package Noundry.Slurp

Quick Start

Interactive Mode

Simply run slurp with a CSV file:

slurp data.csv

The tool will interactively prompt you for database connection details.

Command Line Mode

slurp data.csv -p postgres -s localhost -d mydb -u myuser --password mypass

Configuration File Mode

Create a slurp.json file:

{
  "provider": "postgres",
  "server": "localhost",
  "port": 5432,
  "database": "mydb",
  "username": "myuser",
  "password": "mypass",
  "schema": "public",
  "batchSize": 10000,
  "parallel": true,
  "createIndexes": true
}

Then run:

slurp data.csv -c slurp.json

Command Line Options

Examples

Basic Examples

# Ingest to SQLite (simplest)
slurp sales.csv -p sqlite -d sales.db

# Ingest to PostgreSQL
slurp customers.csv -p postgres -s localhost -d mydb -u myuser --password mypass

# SQL Server with Windows Authentication
slurp products.csv -p sqlserver -s .\\SQLEXPRESS -d ProductDB

# MySQL with custom port
slurp orders.csv -p mysql -s localhost --port 3307 -d orderdb -u root --password root

Advanced Examples

# Custom table name and schema
slurp data.csv -p postgres -d mydb -t my_table --schema staging

# Semicolon-delimited file with no header
slurp data.csv --delimiter ";" --no-header

# Drop existing table and use smaller batches
slurp huge_file.csv --drop --batch-size 5000

# Disable parallel processing for ordered inserts
slurp sequential.csv --no-parallel

# Skip index creation for faster initial load
slurp bulk_data.csv --no-indexes

Library Usage

using Noundry.Slurp;
using Noundry.Slurp.Configuration;

var config = new SlurpConfiguration
{
    Provider = "postgres",
    Server = "localhost",
    Database = "mydb",
    Username = "user",
    Password = "pass",
    TableName = "my_data",
    BatchSize = 10000,
    Parallel = true,
    CreateIndexes = true
};

using var engine = new SlurpEngine(config);

// Subscribe to progress events
engine.ProgressChanged += (s, e) =>
    Console.WriteLine($"Progress: {e.PercentComplete:F1}% ({e.RowsPerSecond:N0} rows/sec)");

engine.StatusChanged += (s, msg) =>
    Console.WriteLine($"Status: {msg}");

// Perform ingestion
var result = await engine.IngestAsync("data.csv");

Console.WriteLine($"Ingested {result.RowsInserted:N0} rows in {result.Duration.TotalSeconds:F2} seconds");

Schema Inference

Slurp automatically infers the most appropriate data types:

Intelligent Indexing

Slurp automatically creates indexes for:

• Primary key candidates: Columns named id with 100% unique values become primary keys
• ID columns: Columns ending in id, Id, ID, or containing _id, _key, _code
• Email columns: Columns with "email" in the name
• Phone columns: Columns with "phone" in the name
• Date/time columns: Columns with "date", "time", "created", or "updated" in the name
• High cardinality columns: Columns with >95% unique values (unique constraint + index)
• Low cardinality columns: Columns with <10% unique values (good for filtering)

Project Structure

Noundry.Slurp/
├── src/
│   ├── lib/                 # Core library
│   │   ├── Core/            # Abstractions and interfaces
│   │   ├── Parsers/         # CSV parsing and schema inference
│   │   ├── Providers/       # Database providers
│   │   └── Configuration/   # Configuration models
│   ├── cli/                 # CLI application
│   └── tests/              # Test suite
├── Noundry.Slurp.sln
└── README.md

Performance

Slurp is optimized for speed:

• Zero-allocation CSV parsing via SEP library
• Bulk insert operations for all supported databases
• Parallel processing for large files (>10MB or >100K rows)
• Optimized batch sizes for network efficiency
• Connection pooling and reuse
• Minimal memory footprint

Typical performance: 100,000+ rows per second on modern hardware.

Testing

# Run all tests
dotnet test

# Run with coverage
dotnet test /p:CollectCoverage=true

# Run specific test category
dotnet test --filter Category=Integration

Building from Source

# Clone the repository
git clone https://github.com/noundry/slurp.git
cd slurp

# Build the solution
dotnet build

# Run tests
dotnet test

# Pack the tool
dotnet pack src/cli/Noundry.Slurp.Cli.csproj

# Install locally
dotnet tool install --global --add-source ./src/cli/nupkg Noundry.Slurp.Tool

Configuration Schema

{
  "provider": "string",           // Database provider
  "connectionString": "string",   // Full connection string (optional)
  "server": "string",             // Server hostname/IP
  "port": 0,                      // Server port
  "database": "string",           // Database name
  "username": "string",           // Username
  "password": "string",           // Password
  "tableName": "string",          // Target table name
  "schema": "string",             // Database schema
  "batchSize": 10000,             // Rows per batch
  "createIndexes": true,          // Create indexes
  "dropTableIfExists": false,     // Drop existing table
  "delimiter": ",",               // CSV delimiter
  "hasHeader": true,              // CSV has header row
  "inferTypes": true,             // Infer column types
  "maxSampleRows": 1000,          // Rows to sample for inference
  "parallel": true,               // Enable parallel processing
  "parallelism": 0,               // Thread count (0 = auto)
  "timeout": 300,                 // Command timeout in seconds
  "columnMappings": {             // Rename columns
    "OldName": "NewName"
  },
  "columnTypes": {                // Override column types
    "ColumnName": "BIGINT"
  }
}

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

License

MIT License - see LICENSE file for details.

Acknowledgments

• SEP - The blazing fast CSV parser that powers Slurp
• Spectre.Console - Beautiful console applications made easy

Troubleshooting

Common Issues

Issue: "Table already exists" Solution: Use the --drop flag to drop and recreate the table

Issue: "Connection timeout" Solution: Increase timeout with --timeout 600 or reduce batch size with --batch-size 5000

Issue: "Out of memory on large files" Solution: Reduce batch size and ensure parallel processing is enabled

Issue: "Type inference incorrect" Solution: Use columnTypes in configuration to override specific columns

Roadmap

• Support for more file formats (TSV, Excel, JSON)
• Cloud database support (Azure SQL, Amazon RDS)
• Data transformation pipelines
• Incremental/upsert operations
• Web UI for monitoring
• Data validation rules
• Export functionality

Made by Noundry