FluxCurator 0.7.0

FluxCurator

Clean, protect, and chunk your text for RAG pipelines — no dependencies required.

Overview

FluxCurator is a text preprocessing library for RAG (Retrieval-Augmented Generation) pipelines. It provides multilingual PII masking, content filtering, and intelligent text chunking with support for 14 languages and 13 countries' national IDs.

Zero Dependencies Philosophy: Core functionality (FluxCurator.Core) works standalone with no external dependencies. The main package (FluxCurator) adds DI support and semantic chunking capabilities via external IEmbedder injection.

Features

• Text Refinement - Clean noisy text by removing blank lines, duplicates, empty list markers, and custom patterns
• Multilingual PII Masking - Auto-detect and mask emails, phones, national IDs, credit cards across 14 languages
• Content Filtering - Filter harmful content with customizable rules and blocklists
• Smart Chunking - Rule-based chunking (sentence, paragraph, token)
• Semantic Chunking - Embedding-based chunking for semantic boundaries
• Hierarchical Chunking - Document structure-aware chunking with parent-child relationships
• Multi-Language Support - 14 languages including Korean, English, Japanese, Chinese, Vietnamese, Thai
• National ID Validation - Checksum validation for 13 countries including SSN (US), RRN (Korea), Aadhaar (India), SIN (Canada)
• Streaming Support - Memory-efficient streaming chunk generation via ChunkStreamAsync
• Pipeline Processing - Combine filtering, masking, and chunking in one call
• Dependency Injection - Full DI support with IServiceCollection extensions
• FileFlux Integration - Seamless integration with FileFlux document processing

Installation

# Main package (DI support and semantic chunking)
dotnet add package FluxCurator

# Core package only (zero dependencies)
dotnet add package FluxCurator.Core

Quick Start

Basic Chunking

using FluxCurator;
using FluxCurator.Core.Domain;

// Create curator with default options
var curator = new FluxCurator();

// Chunk text using sentence strategy
var chunks = await curator.ChunkAsync(text);

foreach (var chunk in chunks)
{
    Console.WriteLine($"Chunk {chunk.Index + 1}/{chunk.TotalChunks}:");
    Console.WriteLine(chunk.Content);
    Console.WriteLine($"Tokens: ~{chunk.Metadata.EstimatedTokenCount}");
}

Streaming Chunks

// Memory-efficient streaming for large texts
var curator = new FluxCurator();

await foreach (var chunk in curator.ChunkStreamAsync(largeText))
{
    // Process chunks as they are generated
    Console.WriteLine($"Chunk {chunk.Index}: {chunk.Content.Length} chars");
    await ProcessChunkAsync(chunk);
}

Dependency Injection

// Program.cs or Startup.cs
services.AddFluxCurator(options =>
{
    options.DefaultChunkOptions = ChunkOptions.ForRAG;
    options.EnablePIIMasking = true;
    options.EnableContentFiltering = true;
});

// With external IEmbedder for semantic chunking
services.AddSingleton<IEmbedder>(myEmbedder);  // Register your embedder first
services.AddFluxCurator(options =>
{
    options.DefaultChunkOptions = new ChunkOptions
    {
        Strategy = ChunkingStrategy.Semantic,
        TargetChunkSize = 512
    };
});

Using IChunkerFactory

// Inject IChunkerFactory for flexible chunker creation
public class MyService
{
    private readonly IChunkerFactory _chunkerFactory;

    public MyService(IChunkerFactory chunkerFactory)
    {
        _chunkerFactory = chunkerFactory;
    }

    public async Task<IReadOnlyList<DocumentChunk>> ProcessAsync(string text)
    {
        // Create specific chunker
        var chunker = _chunkerFactory.CreateChunker(ChunkingStrategy.Hierarchical);
        return await chunker.ChunkAsync(text, ChunkOptions.Default);
    }
}

Text Refinement

// Clean noisy text before processing
var curator = new FluxCurator()
    .WithTextRefinement(TextRefineOptions.Standard);

var result = await curator.PreprocessAsync(rawText);
// Pipeline: Refine → Filter → Mask → Chunk

// Use presets for specific content types
TextRefineOptions.Light        // Minimal: empty list markers, trim, collapse blanks
TextRefineOptions.Standard     // Default: + remove duplicates
TextRefineOptions.ForWebContent  // Web-optimized: aggressive cleaning
TextRefineOptions.ForKorean    // Korean: removes 댓글 sections, copyright
TextRefineOptions.ForPdfContent  // PDF: removes page numbers

// Custom patterns
var options = new TextRefineOptions
{
    RemoveBlankLines = true,
    RemoveDuplicateLines = true,
    RemoveEmptyListItems = true,  // Supports Korean markers: ㅇ, ○, ●, □, ■
    TrimLines = true,
    RemovePatterns = [@"^#\s*댓글\s*$", @"^\[광고\].*$"]
};

PII Masking

// Enable PII masking
var curator = new FluxCurator()
    .WithPIIMasking();

// Mask PII in text
var result = curator.MaskPII("Contact: 010-1234-5678, Email: test@example.com");
Console.WriteLine(result.MaskedText);
// Output: "Contact: [PHONE], Email: [EMAIL]"

Multilingual National ID Detection

// Auto-detect PII for all supported languages
var curator = new FluxCurator()
    .WithPIIMasking(PIIMaskingOptions.Default);

var result = curator.MaskPII("SSN: 123-45-6789, RRN: 901231-1234567");
// Output: "SSN: [NATIONAL_ID], RRN: [NATIONAL_ID]"

// Detect for specific language
var koreanCurator = new FluxCurator()
    .WithPIIMasking(PIIMaskingOptions.ForLanguage("ko"));

var krResult = koreanCurator.MaskPII("주민등록번호: 901231-1234567");
// Output: "주민등록번호: [NATIONAL_ID]"
// Validates using Modulo-11 checksum algorithm

// Detect for multiple languages
var multiCurator = new FluxCurator()
    .WithPIIMasking(PIIMaskingOptions.ForLanguages("en-US", "ko", "pt-BR"));

Hierarchical Chunking

var curator = new FluxCurator()
    .WithChunkingOptions(opt =>
    {
        opt.Strategy = ChunkingStrategy.Hierarchical;
        opt.MaxChunkSize = 1024;
    });

var chunks = await curator.ChunkAsync(markdownText);

foreach (var chunk in chunks)
{
    // Access hierarchy information
    var level = chunk.Metadata.Custom?["HierarchyLevel"];
    var parentId = chunk.Metadata.Custom?["ParentId"];
    var sectionPath = chunk.Location.SectionPath;

    Console.WriteLine($"[Level {level}] {sectionPath}");
    Console.WriteLine(chunk.Content);
}

Full Pipeline Processing

// Complete preprocessing pipeline
var curator = new FluxCurator()
    .WithTextRefinement(TextRefineOptions.Standard)
    .WithContentFiltering()
    .WithPIIMasking(PIIMaskingOptions.ForLanguages("en", "ko", "ja"))
    .WithChunkingOptions(ChunkOptions.ForRAG);

// Process: Refine → Filter → Mask PII → Chunk
var result = await curator.PreprocessAsync(text);

Console.WriteLine(result.GetSummary());
// Output: "Produced 5 chunk(s). Filtered 2 content item(s). Masked 3 PII item(s)."

Semantic Chunking

// Requires an IEmbedder implementation (e.g., OpenAI, LMSupply, etc.)
var curator = new FluxCurator()
    .UseEmbedder(myEmbedder)  // Inject your IEmbedder implementation
    .WithChunkingOptions(opt =>
    {
        opt.Strategy = ChunkingStrategy.Semantic;
        opt.SemanticSimilarityThreshold = 0.5f;
    });

var chunks = await curator.ChunkAsync(text);
// Chunks at natural semantic boundaries

Chunking Strategies

Large Document Processing

For documents with 50K+ tokens, use hierarchical chunking with the ForLargeDocument preset:

// Use the preset for large documents
var curator = new FluxCurator()
    .WithChunkingOptions(ChunkOptions.ForLargeDocument);

var chunks = await curator.ChunkAsync(largeDocument);

// Access hierarchy metadata
foreach (var chunk in chunks)
{
    var level = chunk.Metadata.Custom?["HierarchyLevel"];
    var sectionPath = chunk.Location.SectionPath;
    Console.WriteLine($"[Level {level}] {sectionPath}: {chunk.Content.Length} chars");
}

See Large Document Chunking Guide for detailed configuration options.

Supported Languages

FluxCurator includes language profiles for accurate sentence detection and token estimation:

PII Types Supported

Global PII Types

National ID Types by Country

Configuration Options

ChunkOptions

var options = new ChunkOptions
{
    Strategy = ChunkingStrategy.Sentence,
    TargetChunkSize = 512,
    MinChunkSize = 100,
    MaxChunkSize = 1024,
    OverlapSize = 50,
    LanguageCode = "ko",  // null = auto-detect
    PreserveSentences = true,
    PreserveParagraphs = true,
    SemanticSimilarityThreshold = 0.5f
};

// Preset configurations
ChunkOptions.Default           // General purpose (512 target, 1024 max)
ChunkOptions.ForRAG            // Optimized for RAG (512 target, semantic)
ChunkOptions.ForKorean         // Optimized for Korean text
ChunkOptions.ForLargeDocument  // Large docs (50K+ tokens, hierarchical)
ChunkOptions.ForShortContext   // Short-context models (MiniLM, BGE-small)
ChunkOptions.ForMediumContext  // Medium-context models (e5, BGE-base)
ChunkOptions.ForLongContext    // Long-context models (OpenAI, Cohere)
ChunkOptions.FixedSize(256, 32)  // Fixed token size with overlap

Preset Comparison by Embedding Model

Choose the right preset based on your embedding model's context window:

Note: All sizes are in estimated tokens, not characters. See FAQ for character-to-token conversion ratios.

Masking Strategies

Extensibility

FluxCurator is designed for extensibility. You can add custom PII detectors for your specific needs.

Custom PII Detector

Implement IPIIDetector or extend PIIDetectorBase for pattern-based detection:

using FluxCurator.Core.Core;
using FluxCurator.Core.Domain;
using FluxCurator.Core.Infrastructure.PII;

public class EmployeeIdDetector : PIIDetectorBase
{
    public override PIIType PIIType => PIIType.Custom;
    public override string Name => "Employee ID Detector";

    // Pattern: EMP-123456
    protected override string Pattern => @"EMP-\d{6}";

    protected override bool ValidateMatch(string value, out float confidence)
    {
        confidence = 0.95f;
        return true;
    }
}

// Register and use via PIIMasker
var masker = new PIIMasker(PIIMaskingOptions.Default);
masker.RegisterDetector(new EmployeeIdDetector());

var result = masker.Mask("Contact employee EMP-123456 for details.");
// Output: "Contact employee [PII] for details."

// Or register directly via FluxCurator
var curator = new FluxCurator()
    .WithPIIMasking()
    .RegisterPIIDetector(new EmployeeIdDetector());

var curatorResult = curator.MaskPII("Contact employee EMP-123456 for details.");
// Output: "Contact employee [PII] for details."

Custom National ID Detector

Extend NationalIdDetectorBase to add support for additional countries:

using FluxCurator.Core.Core;
using FluxCurator.Core.Infrastructure.PII.NationalId;

public class IndiaAadhaarDetector : NationalIdDetectorBase
{
    public override string LanguageCode => "hi";
    public override string NationalIdType => "Aadhaar";
    public override string FormatDescription => "12 digits with optional spaces";
    public override string CountryName => "India";
    public override string Name => "India Aadhaar Detector";

    // Pattern: 1234 5678 9012 or 123456789012
    protected override string Pattern => @"\d{4}\s?\d{4}\s?\d{4}";

    protected override bool ValidateMatch(string value, out float confidence)
    {
        var normalized = NormalizeValue(value);

        if (normalized.Length != 12 || !normalized.All(char.IsDigit))
        {
            confidence = 0.0f;
            return false;
        }

        // Implement Verhoeff checksum validation
        if (!ValidateVerhoeffChecksum(normalized))
        {
            confidence = 0.6f;
            return true; // Still flag as PII
        }

        confidence = 0.98f;
        return true;
    }

    private static bool ValidateVerhoeffChecksum(string number)
    {
        // Verhoeff algorithm implementation
        // ...
        return true;
    }
}

// Register with the national ID registry
var registry = new NationalIdRegistry();
registry.Register(new IndiaAadhaarDetector());

var masker = new PIIMasker(
    PIIMaskingOptions.ForLanguage("hi"),
    registry);

Dependency Injection with Custom Detectors

// Register custom registry with additional detectors
services.AddSingleton<INationalIdRegistry>(sp =>
{
    var registry = new NationalIdRegistry();
    registry.Register(new IndiaAadhaarDetector());
    registry.Register(new CanadaSINDetector());
    registry.Register(new AustraliaTFNDetector());
    return registry;
});

// Register PIIMasker with custom registry
services.AddScoped<IPIIMasker>(sp =>
{
    var registry = sp.GetRequiredService<INationalIdRegistry>();
    var options = PIIMaskingOptions.ForLanguages("en", "hi");
    return new PIIMasker(options, registry);
});

Extension Points Summary

Integration with Iyulab Ecosystem

FluxCurator is part of the Iyulab open-source RAG ecosystem:

┌─────────────────────────────────────────────────────────────┐
│                    Foundation Layer                          │
├─────────────────────────────────────────────────────────────┤
│  LMSupply          LMSupply          FluxCurator  FluxImprover│
│  (Embeddings)     (Reranking)      (Chunking)   (LLM-based) │
└───────────┬───────────────────────────┬─────────────────────┘
            │                           │
            ▼                           ▼
┌───────────────────────────────────────────────────────────────┐
│                    Processing Layer                           │
├───────────────────────────────────────────────────────────────┤
│        FileFlux (Document Processing)    WebFlux (Web)        │
└───────────────────────────┬───────────────────────────────────┘
                            │
                            ▼
┌───────────────────────────────────────────────────────────────┐
│                    Storage Layer                              │
├───────────────────────────────────────────────────────────────┤
│                    FluxIndex (Vector DB)                      │
└───────────────────────────┬───────────────────────────────────┘
                            │
                            ▼
┌───────────────────────────────────────────────────────────────┐
│                    Application Layer                          │
├───────────────────────────────────────────────────────────────┤
│                        Filer (App)                            │
└───────────────────────────────────────────────────────────────┘

FileFlux Integration

using FileFlux.Infrastructure.Strategies;
using FileFlux.Infrastructure.Adapters;

// Use FluxCurator chunking in FileFlux
var chunkerFactory = new ChunkerFactory(embedder);
var strategy = new FluxCuratorChunkingStrategy(
    chunkerFactory,
    ChunkingStrategy.Hierarchical);

var chunks = await strategy.ChunkAsync(documentContent, options);

// Convert between chunk types
var fileFluxChunks = fluxCuratorChunks.ToFileFluxChunks();
var curatorChunks = fileFluxChunks.ToFluxCuratorChunks();

Project Structure

FluxCurator/
├── src/
│   ├── FluxCurator.Core/              # Zero-dependency core
│   │   ├── Core/                      # Interfaces
│   │   │   ├── IChunker.cs
│   │   │   ├── IChunkerFactory.cs
│   │   │   ├── IEmbedder.cs
│   │   │   └── ILanguageProfile.cs
│   │   ├── Domain/                    # Models
│   │   │   ├── ChunkOptions.cs
│   │   │   ├── DocumentChunk.cs
│   │   │   ├── ChunkingStrategy.cs
│   │   │   └── PIIMaskingOptions.cs
│   │   └── Infrastructure/            # Implementations
│   │       ├── Chunking/
│   │       │   ├── ChunkerBase.cs
│   │       │   ├── SentenceChunker.cs
│   │       │   ├── ParagraphChunker.cs
│   │       │   ├── TokenChunker.cs
│   │       │   └── HierarchicalChunker.cs
│   │       └── Languages/
│   │           ├── LanguageProfileRegistry.cs
│   │           ├── KoreanLanguageProfile.cs
│   │           └── EnglishLanguageProfile.cs
│   │
│   └── FluxCurator/                   # Main package
│       ├── Infrastructure/
│       │   └── Chunking/
│       │       ├── ChunkerFactory.cs  # Factory with all strategies
│       │       └── SemanticChunker.cs # Requires IEmbedder
│       ├── ServiceCollectionExtensions.cs
│       └── FluxCurator.cs             # Main API
│
└── docs/                              # Documentation
    ├── getting-started.md
    ├── chunking-strategies.md
    ├── di-integration.md
    └── fileflux-integration.md

Documentation

• Getting Started - Installation and basic usage
• Chunking Strategies - Detailed guide for each strategy
• Large Document Chunking - Processing 50K+ token documents
• Dependency Injection - DI configuration and patterns
• FileFlux Integration - Integration with FileFlux

Roadmap

• Core chunking strategies (Sentence, Paragraph, Token)
• 11 language profiles for text processing
• Language detection
• Batch processing
• Multilingual PII masking (10 countries)
• Content filtering
• Semantic chunking
• Hierarchical chunking
• Dependency Injection support
• FileFlux integration
• Text refinement with Korean support
• Additional national ID detectors (India Aadhaar, Canada SIN, Australia TFN)
• Additional language profiles (Vietnamese, Thai)
• Custom detector registration via RegisterPIIDetector
• Streaming chunk support via ChunkStreamAsync

FAQ

Q: How do I process large documents (100K+ tokens)?

Use ChunkingStrategy.Hierarchical with the ForLargeDocument preset. It recognizes document structure (#, ##, ###) and chunks at section boundaries while preserving context.

var curator = new FluxCurator()
    .WithChunkingOptions(ChunkOptions.ForLargeDocument);

Q: My chunks are too small/too large. How do I fix this?

Enable chunk balancing and adjust size limits:

var options = new ChunkOptions
{
    MinChunkSize = 200,        // Merge chunks smaller than this
    MaxChunkSize = 1024,       // Split chunks larger than this
    EnableChunkBalancing = true
};

Q: I set MaxChunkSize=512 but my chunks are 1,500+ characters. Is this a bug?

No, this is expected behavior. MaxChunkSize specifies estimated tokens, not characters. FluxCurator estimates tokens using language-specific heuristics without an actual tokenizer:

For mixed-language content (e.g., Korean with English terms), chunk sizes fall between the language-specific ranges. If you need precise token counts, verify with your actual tokenizer after chunking.

Q: How do I preserve context across chunk boundaries?

Increase the overlap size. For technical documents, 15-20% overlap is recommended:

var options = new ChunkOptions
{
    TargetChunkSize = 512,
    OverlapSize = 100  // ~20% overlap
};

Q: Does FluxCurator support document structure from FileFlux?

Yes. When documents are processed through FileFlux, structure hints (headings, sections) are passed to FluxCurator for intelligent boundary detection. Use ChunkingStrategy.Hierarchical for best results.

Q: How do I process Korean documents (DOCX, PPTX, HWP)?

FluxCurator processes text, not document files directly. Use FileFlux to extract text first, then chunk with FluxCurator:

// 1. Extract text from document using FileFlux
var document = await fileFlux.ProcessAsync("보고서.docx");

// 2. Chunk the extracted Korean text
var curator = new FluxCurator()
    .WithTextRefinement(TextRefineOptions.ForKorean)
    .WithChunkingOptions(opt =>
    {
        opt.Strategy = ChunkingStrategy.Hierarchical;
        opt.LanguageCode = "ko";  // Use Korean language profile
        opt.EnableChunkBalancing = true;
    });

var chunks = await curator.ChunkAsync(document.Text);

See Large Document Chunking Guide for detailed Korean document processing examples.

Contributing

Contributions are welcome! Please see CONTRIBUTING.md for guidelines.

License

MIT License - see LICENSE for details.

Part of the Iyulab Open Source Ecosystem