Scrubkit.Tool 1.8.0

Scrubkit.Tool

The scrubkit command-line tool — run Scrubkit from any shell or CI, no code required. Point it at a folder and it extracts text + metadata from PDFs, Office / OpenDocument, email, EPUB, and text / image files, optionally redacts PII and secrets, and writes a CSV / JSON / JSON Lines / Parquet table. Everything runs on your machine — no network calls.

Install

dotnet tool install --global Scrubkit.Tool

Use

scrubkit scan <folder> [options]

# Extract a folder to CSV on stdout
scrubkit scan ./docs

# Redact PII + secrets and write JSON Lines for a RAG / embedding pipeline
scrubkit scan ./docs --redact --format jsonl --out docs.jsonl

# Aggressive redaction, only PDFs and email, with a content hash per file
scrubkit scan ./docs --redact=aggressive --include .pdf,.eml --hash

# Columnar output for analytics
scrubkit scan ./data --format parquet --out data.parquet

# Incremental: only output files changed since the last run, and update the manifest
scrubkit scan ./docs --since state.txt --manifest state.txt --out delta.jsonl

Options

The table goes to stdout (or --out); progress and a summary go to stderr, so you can pipe the output cleanly. Exit code is 0 on success, 1 on a usage or I/O error.

Scrubbing is best-effort, not a compliance tool.

Custom rules (--rules)

Add your own patterns (and allow/deny/disable lists) without code. Custom rules run before the built-ins, so a domain pattern wins an overlap. Using --rules turns redaction on (--redact=aggressive still composes). Example rules.json:

{
  "rules": [
    { "category": "EmployeeId", "pattern": "\\bE\\d{6}\\b", "token": "[EMP]" },
    { "category": "CaseNo",     "pattern": "\\bC-\\d+\\b", "ignoreCase": false }
  ],
  "allow": ["support@ourco.com"],
  "deny":  ["Project Titan"],
  "disable": ["Phone"],
  "stableTokens": true,
  "tokenSalt": "per-deployment-secret",
  "revealLast": { "Card": 4 }
}

• rules — each is { category, pattern (.NET regex), token?, ignoreCase? }; token defaults to [CATEGORY]. Patterns run with a match timeout, so a runaway regex can't hang a scan.
• allow — exact values never redacted; deny — literal terms always redacted; disable — built-in categories to switch off (e.g. Email, Phone).
• stableTokens — give each value a deterministic suffix so identical values share a token (e.g. [EMAIL_3f9a1c8e]) and stay joinable for analytics. tokenSalt mixes a secret into that hash so tokens can't be correlated across corpora or reversed by guessing.
• revealLast — per-category count of trailing characters to keep as a format-preserving mask, e.g. { "Card": 4 } renders **** **** **** 1111. maskChar sets the mask character (default *).

scrubkit scan ./docs --rules rules.json --out clean.jsonl

Links

• Scrubkit on NuGet
• Project site
• Source & changelog