dtpipe 1.2.6

DtPipe

A self-contained CLI for streaming and anonymizing data across databases and file formats.

DtPipe is an Arrow-native, Zero-Copy data pipeline. It reads from a source, applies transformations in columnar batches, and writes to a destination — with no intermediate staging and minimal memory overhead. It targets automation and CI/CD scenarios where you need repeatable, high-performance data movement.

🚀 See the COOKBOOK for Recipes & Examples

Anonymization guides, pipeline examples, and detailed tutorials.

Installation

.NET Global Tool (Recommended)

dotnet tool install -g dtpipe
dtpipe --help

Build from Source

Prerequisite: .NET 10 SDK

# Bash (Mac/Linux/Windows Git Bash)
./build.sh

# PowerShell (Windows/Cross-platform)
./build.ps1

Binary created at: ./dist/release/dtpipe

Note: Pre-compiled binaries in GitHub Releases are self-contained — no .NET runtime required.

Shell Autocompletion

DtPipe supports smart suggestions for bash, zsh, and powershell:

• strategies (Append, Truncate, Upsert…), providers (pg:, ora:, csv:…), and keyring aliases (keyring://…)

dtpipe completion --install

Restart your terminal (or source ~/.zshrc) to activate.

Quick Reference

CLI Usage

dtpipe --input [SOURCE] --query [SQL] --output [DEST] [OPTIONS]

1. Connection Strings (Input & Output)

DtPipe detects providers from file extensions (.csv, .parquet, .duckdb, .sqlite) or explicit prefixes. Explicit prefixes are recommended to avoid ambiguity.

2. Anonymization & Fakers

Use --fake "Col:Generator" to replace sensitive data. See COOKBOOK.md for the full reference.

3. Positional CLI Option Scoping (Reader vs Writer)

Options are scoped based on their position relative to the output flag (-o):

• Before -o: applied to the reader / pipeline globally.
• After -o: applied to the writer, overriding reader defaults.

# Comma separator for input, semicolon for output
dtpipe -i input.csv --csv-separator "," -o output.csv --csv-separator ";"

4. CLI Options Reference

🔵 Core Essentials

The mandatory flags to define your pipeline.

Example: dtpipe -i orders.csv -o orders.parquet

📥 Source (Reader) Configuration

Adjust how DtPipe pulls data from the source.

🧪 Data Transformation (Mutations)

Modify the content of your rows.

📐 Schema & Projection

Modify the structure of the output table.

📤 Target (Writer) Configuration

Control how data is persisted to the destination.

⚙️ Execution & Statistics

Performance tuning and automation.

🔒 Secret Management

DtPipe integrates with the OS keyring (Windows Credential Manager, macOS Keychain, Linux Secret Service) to store connection strings without exposing them in scripts, command history, or ps output.

A secret can store a full connection string including its provider prefix (e.g. pg:Host=...).

Store a Secret

dtpipe secret set prod-db "ora:Data Source=PROD;User Id=scott;Password=tiger"

Use it in a Transfer

dtpipe -i keyring://prod-db -q "SELECT * FROM users" -o users.parquet

Manage Secrets

🔀 Multi-Stream Pipelines (DAG)

You can run multiple branches in a single command using --input flags and --sql processors. Each branch can be named with --alias so that downstream steps can reference it.

A pipeline with multiple inputs or processors is assembled as a DAG and executed concurrently.

Example: In-Memory Join

dtpipe \
  -i customers.parquet --alias customers \
  -i orders.csv --alias orders \
  --from orders --ref customers \
  --sql "SELECT o.*, c.name FROM orders o JOIN customers c ON o.customer_id = c.id" \
  -o result.parquet

Contributing

Adding a new database adapter or transformer? See the Developer Guide.

License

MIT