dtpipe 1.13.0

There is a newer version of this package available.
See the version list below for details.
dotnet tool install --global dtpipe --version 1.13.0
                    
This package contains a .NET tool you can call from the shell/command line.
dotnet new tool-manifest
                    
if you are setting up this repo
dotnet tool install --local dtpipe --version 1.13.0
                    
This package contains a .NET tool you can call from the shell/command line.
#tool dotnet:?package=dtpipe&version=1.13.0
                    
nuke :add-package dtpipe --version 1.13.0
                    

DtPipe

A self-contained CLI for streaming, transforming, and anonymizing data across databases and file formats.

DtPipe reads from a source, applies row and columnar transformations in batches, and writes to a destination with no intermediate staging. It is designed for automation and CI/CD workflows where repeatable, observable data pipelines matter.


📚 Start with the documentation → docs/

Install, a first pipeline, the concepts, and a guide per task — anonymization, joins and fan-out, incremental loading, YAML jobs, secrets, and running in a .NET estate.


Install

dotnet tool install -g dtpipe        # or download a binary from the releases page
dtpipe --help

Six platforms per release — Linux, macOS and Windows, x64 and arm64 — each a single executable with the .NET runtime, the DuckDB engine and every database driver inside it. Nothing else to install. Full instructions, including building from source: docs/install.md.

One example

Anonymize a table on the way out, without ever writing the clear values to disk:

dtpipe \
  -i "pg:Host=localhost;Database=prod;Username=postgres" \
  --query "SELECT * FROM users" \
  --fake "email:internet.email" \
  --fake "name:name.fullName" \
  --mask "phone:###-****" \
  --null "ssn" \
  -o anonymized_users.parquet

More, from a first pipeline to a DAG: docs/quickstart.md.

Studio

dtpipe studio ./jobs opens a local page over the YAML jobs of a folder: each job as cards you edit and sample on the real engine, with the target never written. It is part of the CLI and needs no coordinator; --coordinator only adds the bricks and the pipeline library of a distributed setup. Details in REFERENCE.md.

Not everything in this release is stable. The distributed pipeline (coordinator, pipeline nodes, runner) and dtpipe studio are new: they may change or disappear without notice, and the distributed projects ship in no package. Do not build a workload on them yet. What was there before, reading, transforming and writing through the CLI, a job file or the MCP server, is versioned as it always was. The distributed family has not been run on Windows; see docs/experimentation.md.

Distributed pipelines (experimental)

A coordinator cuts a pipeline into fragments and runs each next to its data, on nodes that offer bricks (reads and writes their owner declared on their own database); a runner is a node with no data that runs the steps between the bricks. The coordinator keeps the pipelines in a git-backed library, decides who may send rows to whom, and gives each run one verdict. The coordinator and the runner are published as container images, and docker compose -f deploy/compose.yaml up -d starts a trial of both on your machine. Start with Your first distributed pipeline; to run them for real, see deploy/README.md; every setting is in REFERENCE.md.

What it connects to

Family Examples Prefix
Databases PostgreSQL, MySQL, SQLite, DuckDB, SQL Server, Oracle pg:, mysql:, sqlite:, duck:, mssql:, ora:
Files CSV, JsonL, Parquet, Arrow, XML csv:, jsonl:, parquet:, arrow:, xml:
Object storage S3-compatible, Azure Blob s3://bucket/key.parquet, azure://container/blob.csv
Special Data generator (source), null / checksum (sink) generate:N, null:, checksum:

dtpipe providers lists what your binary actually carries. The connection catalog, with the string format and the rules that apply to every provider, is at docs/connections/.

Is DuckDB alone enough? Often, yes — and then use it directly. DtPipe embeds it as its SQL engine and adds what a transfer needs around it: anonymization in transit, concurrent fan-out, write strategies (upsert, auto-migrate, bulk), Oracle / SQL Server / XML sources, and repeatable YAML jobs with secret management.

Documentation

Where What it holds
docs/ Start here. Install, quickstart, concepts, a guide per task, a page per connection
REFERENCE.md The exhaustive lists: every CLI flag, the YAML job schema, provider options, the MCP tools
EXTENDING.md Writing a new adapter or transformer

Contributing

See EXTENDING.md for the adapter and transformer patterns.

License

MIT

Product Compatible and additional computed target framework versions.
.NET net10.0 is compatible.  net10.0-android was computed.  net10.0-browser was computed.  net10.0-ios was computed.  net10.0-maccatalyst was computed.  net10.0-macos was computed.  net10.0-tvos was computed.  net10.0-windows was computed. 
Compatible target framework(s)
Included target framework(s) (in package)
Learn more about Target Frameworks and .NET Standard.

This package has no dependencies.

Version Downloads Last Updated
1.17.0 0 11/16/2026
1.16.1 88 11/12/2026
1.16.0 100 11/11/2026
1.15.2 98 11/10/2026
1.15.1 98 11/10/2026
1.15.0 96 11/9/2026
1.14.0 99 11/8/2026
1.13.0 101 11/6/2026
1.12.0 111 11/5/2026
1.11.0 108 11/4/2026
1.10.0 157 10/18/2026
1.9.0 152 10/2/2026
1.8.2 150 9/14/2026
1.8.1 130 9/10/2026
1.8.0 222 9/9/2026
1.7.0 135 9/5/2026
1.6.0 210 8/27/2026
1.5.0 185 8/6/2026
1.4.3 165 7/7/2026
1.4.2 157 6/20/2026
Loading failed