FluxCurator.Core 0.16.0

FluxCurator

Clean, protect, and chunk your text for RAG pipelines — no tokenizer, model, or service required.

Overview

FluxCurator is a text preprocessing library for RAG (Retrieval-Augmented Generation) pipelines. It provides multilingual PII masking, content filtering, and intelligent text chunking with support for 13 languages and 13 countries' national IDs.

Minimal Dependencies Philosophy: Core functionality (FluxCurator.Core) works standalone; its only dependency is the shared contract package Flux.Abstractions. The main package (FluxCurator) adds the Curator entry point, DI support and semantic chunking via an external IEmbedder you inject.

Packages

Features

Installation

# Main package (DI support and semantic chunking)
dotnet add package FluxCurator

# Core package only (no Curator/DI; depends only on Flux.Abstractions)
dotnet add package FluxCurator.Core

Quick Start

The entry point is Curator (namespace FluxCurator). Options, chunks and results live in FluxCurator.Core.Domain, and the extension interfaces (IEmbedder, IChunkerFactory, IPIIMasker, …) in FluxCurator.Core.Core. The examples below assume these three using lines:

Basic Chunking

using FluxCurator;                 // Curator, AddFluxCurator
using FluxCurator.Core.Core;       // IEmbedder, IChunkerFactory, IPIIDetector, ...
using FluxCurator.Core.Domain;     // ChunkOptions, DocumentChunk, PIIMaskingOptions, ...

var text = "FluxCurator splits text into chunks. Each chunk keeps whole sentences. It works without a tokenizer.";

// Create a curator with default options
var curator = new Curator();

// Chunk the text (the default strategy is Auto)
var chunks = await curator.ChunkAsync(text);

foreach (var chunk in chunks)
{
    Console.WriteLine($"Chunk {chunk.ChunkIndex + 1}/{chunk.TotalChunks}:");
    Console.WriteLine(chunk.Content);
    Console.WriteLine($"Tokens: ~{chunk.Metadata.EstimatedTokenCount}");
}

Streaming Chunks

// Memory-efficient streaming for large texts
var curator = new Curator();

await foreach (var chunk in curator.ChunkStreamAsync(largeText))
{
    // Handle each chunk as soon as it is produced
    Console.WriteLine($"Chunk {chunk.ChunkIndex}: {chunk.Content.Length} chars");
}

Dependency Injection

// Program.cs
services.AddFluxCurator(options =>
{
    options.DefaultChunkOptions = ChunkOptions.ForMediumContext;  // ForRAG is Semantic: it needs an IEmbedder (below)
    options.EnablePIIMasking = true;
    options.EnableContentFiltering = true;
});

// With an external IEmbedder for semantic chunking
services.AddSingleton<IEmbedder>(myEmbedder);  // Register your embedder first
services.AddFluxCurator(options =>
{
    options.DefaultChunkOptions = new ChunkOptions
    {
        Strategy = ChunkingStrategy.Semantic,
        TargetChunkSize = 512
    };
});

Using IChunkerFactory

// Inject IChunkerFactory (registered by AddFluxCurator) for flexible chunker creation
public class MyService(IChunkerFactory chunkerFactory)
{
    public Task<IReadOnlyList<DocumentChunk>> ProcessAsync(string text)
    {
        // Create a specific chunker
        var chunker = chunkerFactory.CreateChunker(ChunkingStrategy.Hierarchical);
        return chunker.ChunkAsync(text, ChunkOptions.Default);
    }
}

Text Refinement

// Clean noisy text before processing
var curator = new Curator()
    .WithTextRefinement(TextRefineOptions.Standard);

var result = await curator.PreprocessAsync(rawText);
// Pipeline: Refine → Filter → Mask → Chunk

// Custom patterns
var options = new TextRefineOptions
{
    RemoveBlankLines = true,
    RemoveDuplicateLines = true,
    RemoveEmptyListItems = true,  // Supports Korean markers: ㅇ, ○, ●, □, ■
    TrimLines = true,
    RemovePatterns = [@"^#\s*댓글\s*$", @"^\[광고\].*$"]
};

Presets for specific content types:

PII Masking

// Enable PII masking
var curator = new Curator()
    .WithPIIMasking();

// Mask PII in text
var result = curator.MaskPII("Contact: 010-1234-5678, Email: test@example.com");
Console.WriteLine(result.MaskedText);
// Output: "Contact: [PHONE], Email: [EMAIL]"

Multilingual National ID Detection

// Auto-detect PII for all supported languages
var curator = new Curator()
    .WithPIIMasking(PIIMaskingOptions.Default);

var result = curator.MaskPII("SSN: 536-22-1234, RRN: 901231-1234567");
// Output: "SSN: [NATIONAL_ID], RRN: [NATIONAL_ID]"

// Detect for specific language
var koreanCurator = new Curator()
    .WithPIIMasking(PIIMaskingOptions.ForLanguage("ko"));

var krResult = koreanCurator.MaskPII("주민등록번호: 901231-1234567");
// Output: "주민등록번호: [NATIONAL_ID]"
// Validates using Modulo-11 checksum algorithm

// Detect for multiple languages
var multiCurator = new Curator()
    .WithPIIMasking(PIIMaskingOptions.ForLanguages("en-US", "ko", "pt-BR"));

Hierarchical Chunking

var curator = new Curator()
    .WithChunkingOptions(opt =>
    {
        opt.Strategy = ChunkingStrategy.Hierarchical;
        opt.MaxChunkSize = 1024;
    });

var chunks = await curator.ChunkAsync(markdownText);

foreach (var chunk in chunks)
{
    // Access hierarchy information
    var level = chunk.Metadata.Custom?["HierarchyLevel"];
    var parentId = chunk.Metadata.Custom?["ParentId"];
    var sectionPath = chunk.Location.SectionPath;

    Console.WriteLine($"[Level {level}] {sectionPath}");
    Console.WriteLine(chunk.Content);
}

Full Pipeline Processing

// Complete preprocessing pipeline
var curator = new Curator()
    .WithTextRefinement(TextRefineOptions.Standard)
    .WithContentFiltering()
    .WithPIIMasking(PIIMaskingOptions.ForLanguages("en", "ko", "ja"))
    .WithChunkingOptions(ChunkOptions.ForMediumContext);  // ForRAG needs UseEmbedder(...) first

// Process: Refine → Filter → Mask PII → Chunk
var result = await curator.PreprocessAsync(text);

Console.WriteLine(result.GetSummary());
// e.g. "Produced 5 chunk(s). Text refined. Filtered 2 content item(s). Masked 3 PII item(s)."

Semantic Chunking

// Requires an IEmbedder implementation (e.g., OpenAI, LMSupply, etc.)
var curator = new Curator()
    .UseEmbedder(myEmbedder)  // Inject your IEmbedder implementation
    .WithChunkingOptions(opt =>
    {
        opt.Strategy = ChunkingStrategy.Semantic;
        opt.SemanticSimilarityThreshold = 0.5f;
    });

var chunks = await curator.ChunkAsync(text);
// Chunks at natural semantic boundaries

Chunking Strategies

Large Document Processing

For documents with 50K+ tokens, use hierarchical chunking with the ForLargeDocument preset:

// Use the preset for large documents
var curator = new Curator()
    .WithChunkingOptions(ChunkOptions.ForLargeDocument);

var chunks = await curator.ChunkAsync(largeDocument);

// Access hierarchy metadata
foreach (var chunk in chunks)
{
    var level = chunk.Metadata.Custom?["HierarchyLevel"];
    var sectionPath = chunk.Location.SectionPath;
    Console.WriteLine($"[Level {level}] {sectionPath}: {chunk.Content.Length} chars");
}

See Large Document Chunking Guide for detailed configuration options.

Supported Languages

FluxCurator includes language profiles for accurate sentence detection and token estimation. A tag with a region (zh-TW, pt-BR, en-US) uses the profile of its language:

PII Types Supported

Global PII Types

PIIMaskingOptions.TypesToMask selects which built-in detectors run. Its default, PIIType.Common, covers Email, Phone, CreditCard and NationalId; the other types below are detected only when you include them (for example TypesToMask = PIIType.All, or the PIIMaskingOptions.Strict preset).

DriversLicense, PersonName, Address, TaxId and SocialSecurityNumber are categories for detectors you register: no built-in detector reports them (US SSN, UK NINO and the other national numbers report NationalId). PIIMasker.CoveredTypes returns the types a masker can actually find with its options and registered detectors — assert on it when a masking policy requires a type.

Every built-in detector matches whole values only: a run of digits or ASCII letters is never reported in part, so timestamps, order numbers, hashes and UUIDs are not masked as phone numbers or IDs. A value written directly next to text in a script without spaces (연락처010-1234-5678로) is still detected. A bare run of ten digits is not treated as a US phone number, nor a bare 15xxxxxx as a service number; write them with separators.

A value the text marks as a machine identifier is not reported, whatever its shape: the value of a key in PIIMaskingOptions.NonPiiKeys (uid=, auid=, pid=, port=, "ts": …, journald fields - see DefaultNonPiiKeys; add your own keys or clear the set), and a segment of a compound identifier such as a journald cursor (s=4bf9…;i=43b37;m=0161431588;t=65c9…). The same number under another key (phone=, "user":) is still masked. Other values are read by shape: a run that happens to have a valid date and checksum (a timestamp, an order number) can still be reported as an ID or a card number; raise MinConfidence, add its key to NonPiiKeys, or filter such fields before masking. A custom detector deriving from PIIDetectorBase gets the same behaviour by placing TokenStart/TokenEnd (or, for numeric IDs, NumberStart/NumberEnd) around its Pattern.

National ID Types by Country

Configuration Options

ChunkOptions

var options = new ChunkOptions
{
    Strategy = ChunkingStrategy.Sentence,
    TargetChunkSize = 512,
    MinChunkSize = 100,
    MaxChunkSize = 1024,
    OverlapSize = 50,
    LanguageCode = "ko",  // null = auto-detect
    PreserveSentences = true,
    PreserveParagraphs = true,
    SemanticSimilarityThreshold = 0.5f
};

// Or start from a preset (compared below)
var fixedSize = ChunkOptions.FixedSize(256, 32);  // Fixed token size with overlap

Preset Comparison by Embedding Model

Choose the right preset based on your embedding model's context window:

Note: All sizes are in estimated tokens, not characters. See FAQ for character-to-token conversion ratios.

Masking Strategies

Extensibility

FluxCurator is designed for extensibility. You can add custom PII detectors for your specific needs.

Custom PII Detector

Implement IPIIDetector or extend PIIDetectorBase for pattern-based detection:

using FluxCurator;
using FluxCurator.Core.Domain;
using FluxCurator.Core.Infrastructure.PII;

// Register and use via PIIMasker
var masker = new PIIMasker(PIIMaskingOptions.Default);
masker.RegisterDetector(new EmployeeIdDetector());

var result = masker.Mask("Contact employee EMP-123456 for details.");
// Output: "Contact employee [PII] for details."

// Or register directly on the curator
var curator = new Curator()
    .WithPIIMasking()
    .RegisterPIIDetector(new EmployeeIdDetector());

var curatorResult = curator.MaskPII("Contact employee EMP-123456 for details.");
// Output: "Contact employee [PII] for details."

public class EmployeeIdDetector : PIIDetectorBase
{
    public override PIIType PIIType => PIIType.Custom;
    public override string Name => "Employee ID Detector";

    // Pattern: EMP-123456
    protected override string Pattern => @"EMP-\d{6}";

    protected override bool ValidateMatch(string value, out float confidence)
    {
        confidence = 0.95f;
        return true;
    }
}

Custom National ID Detector

Extend NationalIdDetectorBase to add a country the library does not cover (the 13 in the table above are built in and registered by default):

using FluxCurator.Core.Domain;
using FluxCurator.Core.Infrastructure.PII;
using FluxCurator.Core.Infrastructure.PII.NationalId;

// Register with the national ID registry
var registry = new NationalIdRegistry();
registry.Register(new SingaporeNricDetector());

var masker = new PIIMasker(PIIMaskingOptions.ForLanguage("en-SG"), registry);

public class SingaporeNricDetector : NationalIdDetectorBase
{
    public override string LanguageCode => "en-SG";
    public override string NationalIdType => "NRIC";
    public override string FormatDescription => "Letter, 7 digits, check letter";
    public override string CountryName => "Singapore";
    public override string Name => "Singapore NRIC Detector";

    // Pattern: S1234567D
    protected override string Pattern => @"[STFGM]\d{7}[A-Z]";

    protected override bool ValidateMatch(string value, out float confidence)
    {
        var normalized = NormalizeValue(value);
        if (normalized.Length != 9)
        {
            confidence = 0.0f;
            return false;
        }

        // A real detector verifies the check letter here
        confidence = 0.9f;
        return true;
    }
}

Dependency Injection with Custom Detectors

using FluxCurator.Core.Infrastructure.PII;
using FluxCurator.Core.Infrastructure.PII.NationalId;

// Register a registry that also knows your own detectors
services.AddSingleton<INationalIdRegistry>(sp =>
{
    var registry = new NationalIdRegistry();   // the built-in detectors are already registered
    registry.Register(new SingaporeNricDetector());
    return registry;
});

// Register PIIMasker with custom registry
services.AddScoped<IPIIMasker>(sp =>
{
    var registry = sp.GetRequiredService<INationalIdRegistry>();
    var options = PIIMaskingOptions.ForLanguages("en", "hi");
    return new PIIMasker(options, registry);
});

Extension Points Summary

Integration with Iyulab Ecosystem

FluxCurator is part of the Iyulab open-source RAG ecosystem:

┌─────────────────────────────────────────────────────────────┐
│                    Foundation Layer                          │
├─────────────────────────────────────────────────────────────┤
│  LMSupply          LMSupply          FluxCurator  FluxImprover│
│  (Embeddings)     (Reranking)      (Chunking)   (LLM-based) │
└───────────┬───────────────────────────┬─────────────────────┘
            │                           │
            ▼                           ▼
┌───────────────────────────────────────────────────────────────┐
│                    Processing Layer                           │
├───────────────────────────────────────────────────────────────┤
│        FileFlux (Document Processing)    WebFlux (Web)        │
└───────────────────────────┬───────────────────────────────────┘
                            │
                            ▼
┌───────────────────────────────────────────────────────────────┐
│                    Storage Layer                              │
├───────────────────────────────────────────────────────────────┤
│                    FluxIndex (Vector DB)                      │
└───────────────────────────┬───────────────────────────────────┘
                            │
                            ▼
┌───────────────────────────────────────────────────────────────┐
│                    Application Layer                          │
├───────────────────────────────────────────────────────────────┤
│                      Your application                         │
└───────────────────────────────────────────────────────────────┘

FileFlux Integration

FileFlux delegates its chunking to FluxCurator through IChunkerFactory; the adapter types live in the FileFlux package. See FileFlux Integration.

Project Structure

FluxCurator/
├── src/
│   ├── FluxCurator.Core/   # Core (depends only on Flux.Abstractions): interfaces (Core/), models (Domain/), chunkers, PII, filters, languages
│   └── FluxCurator/        # Main package: Curator, DI registration, semantic chunking (requires IEmbedder)
└── docs/                   # Guides

Documentation

• Getting Started - Installation and basic usage
• Chunking Strategies - Detailed guide for each strategy
• Large Document Chunking - Processing 50K+ token documents
• Dependency Injection - DI configuration and patterns
• FileFlux Integration - Integration with FileFlux

FAQ

Q: How do I process large documents (100K+ tokens)?

Use ChunkingStrategy.Hierarchical with the ForLargeDocument preset. It recognizes document structure (#, ##, ###) and chunks at section boundaries while preserving context.

var curator = new Curator()
    .WithChunkingOptions(ChunkOptions.ForLargeDocument);

Q: My chunks are too small/too large. How do I fix this?

Chunk balancing is on by default (EnableChunkBalancing); adjust the size limits:

var options = new ChunkOptions
{
    MinChunkSize = 200,        // Merge chunks smaller than this
    MaxChunkSize = 1024,       // Split chunks larger than this
    EnableChunkBalancing = true
};

Q: I set MaxChunkSize=512 but my chunks are 1,500+ characters. Is this a bug?

No, this is expected behavior. MaxChunkSize specifies estimated tokens, not characters. FluxCurator estimates tokens using language-specific heuristics without an actual tokenizer:

For mixed-language content (e.g., Korean with English terms), chunk sizes fall between the language-specific ranges. If you need precise token counts, verify with your actual tokenizer after chunking.

Q: How do I preserve context across chunk boundaries?

Increase the overlap size. For technical documents, 15-20% overlap is recommended:

var options = new ChunkOptions
{
    TargetChunkSize = 512,
    OverlapSize = 100  // ~20% overlap
};

Q: Does FluxCurator support document structure from FileFlux?

Yes, through the text itself: FileFlux emits Markdown with headings, and ChunkingStrategy.Hierarchical chunks at those heading boundaries. There is no separate structure-hint API in FluxCurator.

Q: How do I process Korean documents (DOCX, PPTX, HWP)?

FluxCurator processes text, not document files directly. Use FileFlux to extract text first, then chunk with FluxCurator:

// 1. Extract the text with FileFlux (see the FileFlux README); here it is already in `extractedText`

// 2. Chunk the extracted Korean text
var curator = new Curator()
    .WithTextRefinement(TextRefineOptions.ForKorean)
    .WithChunkingOptions(opt =>
    {
        opt.Strategy = ChunkingStrategy.Hierarchical;
        opt.LanguageCode = "ko";  // Use Korean language profile
        opt.EnableChunkBalancing = true;
    });

var chunks = await curator.ChunkAsync(extractedText);

See Large Document Chunking Guide for detailed Korean document processing examples.

Contributing

Contributions are welcome — open an issue or a pull request. Release notes are in CHANGELOG.md.

License

MIT License - see LICENSE for details.

Part of the Iyulab Open Source Ecosystem