UMAPuwotSharp 3.40.0

Enhanced High-Performance UMAP C++ Implementation with C# Wrapper

What is UMAP?

UMAP (Uniform Manifold Approximation and Projection) is a dimensionality reduction technique that can be used for visualization, feature extraction, and preprocessing of high-dimensional data. Unlike many other dimensionality reduction algorithms, UMAP excels at preserving both local and global structure in the data.

Example: 3D UMAP embedding rotation showing preserved data structure and clustering

For an excellent interactive explanation of UMAP, see: Understanding UMAP

Project Motivation

This project was created specifically because existing NuGet packages and open-source C# implementations for UMAP lack critical functionality required for production machine learning applications:

• No model persistence: Cannot save trained UMAP models for reuse
• No true transform capability: Cannot project new data points using existing trained models
• No production safety features: No way to detect out-of-distribution data
• Limited dimensionality support: Restricted to 2D or 3D embeddings
• Missing distance metrics: Only basic Euclidean distance support
• No progress reporting: No feedback during long training processes
• Poor performance: Slow transform operations without optimization
• Limited production readiness: Missing essential features for real-world deployment

This implementation addresses these fundamental gaps by providing complete model persistence, authentic transform functionality, arbitrary embedding dimensions (1D-50D), multiple distance metrics, progress reporting, revolutionary HNSW optimization for 50-2000x faster training and transforms, dual HNSW architecture for AI inference, and comprehensive safety features with 5-level outlier detection - making it production-ready for AI/ML validation and real-time data quality assessment based on the proven uwot algorithm.

Hyperparameters and UMAP

UMAP has several very sensetive hyper parameters namely: mind distance (dist), nr of neibours (k), localConnectivity (lC), bandwidth (usually 1 and inaccessable) and spread. The last 3 are very important if you have a large dataset and want to have a good global structure. For example look at our 3D Mamoth 1M data set that is 2d projected and subrandom sampled below:

It can be visible how k and dis is chaning the outcome a lot. (here we know the high data structure (e.g. 3d mammoth) so it is easy for us to trim these hyperparameters for a good result. see 2d projection below:

Furthermore if you increase your number of observation even if you found a good k and dist you will quickly run into problem and need to optimize IC and bandwith kernel as well.

all of the above images are generated with our current library using the C# DEMO (data included in the project). One another this that is very important is that 99% of the global structure comes from doing an inital Spectral initialization using graph Laplacian eigenvectors an time consuming and for some dataset unstable effort. If we start from random points the outcome of UMAP is terrible specially for larger data sets. This leaves us with tuning at least four hyperparams and using a good initilization method to get good results.

my recommendation is to be very carefull with hyperparam selection or use the improved later PacMap, here is an implemetation done by me in c++ and C# PacMAP**

⚠️ CRITICAL INSIGHT: Why Spectral Initialization is Essential for Large Datasets

The Random Initialization Problem with Large Observation Data (>20k):

UMAP with random initialization cannot preserve global structure intact for large datasets regardless of hyperparameter tuning. This is a fundamental limitation:

// Random initialization - BROKEN for large data:
var largeModel = new UMapModel();
largeModel.InitMethod = InitializationMethod.Random;  // Explicitly use random
var embedding = largeModel.Fit(largeDataset, embeddingDimension: 2);
// Result: Fragmented global structure, broken manifold topology

// Spectral initialization (DEFAULT in v3.40.0) - REQUIRED for quality:
var spectralModel = new UMapModel();
// InitMethod = InitializationMethod.Spectral is now the DEFAULT!
var embedding = spectralModel.Fit(largeDataset, embeddingDimension: 2);
// Result: Properly preserved global structure and manifold topology

Why Random Initialization Fails for Large Datasets:

• Local Optima Traps: Random starting points get stuck in poor local minima
• Global Structure Loss: Cannot maintain overall dataset topology and relationships
• Fragmentation: Large datasets split into scattered clusters instead of coherent structure
• Hyperparameter Ineffective: No amount of tuning (neighbors, min_dist, etc.) can fix fundamental initialization problems

The Practical Dilemma:

• AlwaysUseSpectral = Required: Absolutely essential for quality embeddings of >20k samples
• AlwaysUseSpectral = Impractical: Extremely time-consuming (O(n²) complexity) during Fit
• Reality: 500k+ dataset with spectral initialization can take 30+ minutes vs 2-3 minutes with random

✅ CURRENT STATUS: WORKING SOLUTION with Real umappp Implementation

🎉 SOLVED: Verified working solution using https://github.com/libscran/umappp reference implementation!

The project now includes a real umappp integration with proper spectral initialization that solves the fragmentation issues. The mammoth dataset can now be visualized as a single, coherent structure instead of fragmented pieces.

🔥 Key Achievement: Real umappp Integration

We have successfully incorporated the libscran/umappp reference implementation, which provides:

• ✅ Spectral initialization: Non-linear manifold-aware initialization using graph Laplacian eigenvectors
• ✅ Proper epoch scheduling: Correct optimization with balanced positive/negative sampling
• ✅ Superior embeddings: Better preservation of complex 3D structures like the mammoth dataset
• ✅ HNSW-knncolle bridge: Custom interface connecting our HNSW optimization with umappp's expected neighbor search interface
• ✅ Complete C# compatibility: Maintains full API compatibility while using superior algorithms underneath

🏗️ Project Structure with Real umappp Implementation

Current Implementation Architecture

The project uses the real umappp implementation with advanced features:

├── uwot_umappp_wrapper/          # 🎯 PRIMARY: Real umappp implementation (CURRENT)
│   ├── Core umappp integration
│   │   ├── umappp.hpp           # Main umappp library headers
│   │   ├── Options.hpp          # Configuration and parameters
│   │   ├── Status.hpp           # Status and error handling
│   │   ├── initialize.hpp       # ✅ SPECTRAL INITIALIZATION (solves fragmentation!)
│   │   └── optimize_layout.hpp  # Advanced optimization algorithms
│   ├── HNSW-knncolle bridge
│   │   ├── uwot_hnsw_knncolle.hpp  # Custom bridge interface
│   │   └── hnswlib headers         # HNSW optimization integration
│   ├── Enhanced wrapper files
│   │   ├── uwot_fit.cpp/.h          # Training with umappp algorithms
│   │   ├── uwot_simple_wrapper.cpp/.h # Main API interface
│   │   ├── uwot_transform.cpp/.h   # Enhanced transform with safety analysis
│   │   ├── uwot_persistence.cpp/.h  # Stream-based HNSW serialization
│   │   └── All supporting modules (progress, distance, CRC32, etc.)
│   └── CMakeLists.txt               # Build system with umappp dependencies
│
└── uwot_pure_cpp/               # ⚠️ LEGACY: Original implementation (DEPRECATED)
    ├── Reason: Fragmentation issues with complex 3D structures + compilation errors
    ├── Status: Abandoned due to build failures and poor performance
    └── Purpose: Historical reference only - DO NOT USE

Why We Use Real umappp Implementation

Problems with Legacy Implementation (uwot_pure_cpp):

• Build Failures: Compilation errors in test files and unstable build system
• Fragmentation Issues: Complex 3D structures like mammoth dataset fragmenting into scattered pieces
• Poor Performance: Random initialization insufficient for preserving complex manifold topology
• Outdated Architecture: Missing modern optimizations and safety features

Solution with Real umappp (uwot_umappp_wrapper):

• ✅ Spectral Initialization: Graph Laplacian eigenvectors for manifold-aware starting positions
• ✅ Proper Optimization: Correct epoch scheduling with balanced positive/negative sampling
• ✅ Better Topology Preservation: Specifically designed to handle complex non-linear structures
• ✅ Modern Feature Set: All advanced features working (AutoHNSW, dual architecture)
• ✅ Stable Builds: Reliable cross-platform compilation with comprehensive testing

HNSW-knncolle Bridge Architecture

To maintain our HNSW optimization benefits while using umappp, we created a custom bridge:

// Custom bridge class that implements knncolle interface for HNSW
template<typename Index_, typename Float_>
class HnswPrebuilt : public knncolle::Prebuilt<Index_, Float_, Float_> {
    // Bridges our existing HNSW optimization with umappp's expected neighbor search interface
    void search(Index_ i, Index_ k, std::vector<Index_>* indices,
               std::vector<Float_>* distances) const override {
        auto result = hnsw_index_->searchKnn(...);
    }
};

Benefits of This Architecture:

• ✅ Keep HNSW performance: 50-2000x faster transforms
• ✅ Get umappp quality: Superior spectral initialization and optimization
• ✅ Maintain API compatibility: Existing C# code works unchanged
• ✅ Future-proof: Easy to upgrade as umappp evolves

How to Build the Real umappp Implementation

The production-ready umappp implementation is located in uwot_umappp_wrapper/:

# Build the real umappp implementation (RECOMMENDED)
cd uwot_umappp_wrapper
BuildWindowsOnly.bat        # Windows build with Visual Studio 2022

# OR for cross-platform builds:
BuildDockerLinuxWindows.bat # Builds both Windows + Linux with Docker

# The resulting DLL has all advanced features:
# - Spectral initialization (solves fragmentation!)
# - Dual HNSW architecture (AI inference)
# - TransformWithSafety (5-level outlier detection)
# - Stream-based serialization (CRC32 validation)

Note: The C# API remains identical, so existing code continues to work but now benefits from superior umappp algorithms and all advanced features.

🎉 Latest Update v3.40.0 - Initialization API Enhancement & Spectral Default

INITIALIZATION API ENHANCEMENT: New InitializationMethod enum + spectral default + improved API clarity!

✅ InitializationMethod Enum: Clear, explicit control over initialization strategy (Auto=-1, Random=0, Spectral=1) ✅ Spectral Default: Spectral initialization now the default for best quality embeddings ✅ API Clarity: Replaced confusing AlwaysUseSpectral boolean with clearer InitMethod property ✅ Backward Compatibility: Obsolete AlwaysUseSpectral property maintained for existing code ✅ Enhanced Demo: Updated bandwidth experiments with optimal parameters (spread=2.0, local_connectivity=2.0) ✅ Dynamic Metadata: All visualizations use model-extracted parameters (no hardcoded values) ✅ Compiler Warnings Fixed: Clean compilation with zero warnings (unused variables, type conversions) ✅ Production Ready: All features tested and validated with clean builds

🔧 Critical Eigen Fix:

• Problem: Eigen library compilation failure blocking spectral initialization
• Solution: Found Eigen commit 960892ca1 (Feb 2024 JacobiSVD refactor)
• Result: ✅ Compiles successfully with MSVC, ✅ NO performance regression (2-5s tests)

Previous v3.39.0 Features: ✅ AlwaysUseSpectral Property: Force spectral initialization for any dataset size (now improved with enum) ✅ Eigen Compilation Fix: Resolved Eigen 3.4.0/3.4.1 MSVC errors and performance regression ✅ Hyperparameter Integration: LocalConnectivity & Bandwidth exposed in C# API with ModelInfo ✅ Bandwidth Sweep Testing: Comprehensive bandwidth experiments for optimal parameter discovery

Previous v3.37.0 Features: ✅ OpenMP Parallelization: 4-5x faster transforms with multi-threaded processing ✅ Single-Point Optimization: 12-15x speedup for single data point transforms (stack allocation, zero heap) ✅ Stringstream Persistence: Faster save/load with in-memory HNSW serialization (no temp files) ✅ Windows DLL Stability: Proper OpenMP cleanup prevents segfaults on DLL unload

🔥 NEW PERFORMANCE CAPABILITIES:

// Multi-point transform now 4-5x faster with OpenMP
var embeddings = model.Transform(newData);  // Automatically parallelized!
// Progress: "Using 16 threads for parallel processing"

// Single-point transform now 12-15x faster
var singleEmbedding = model.Transform(singlePoint);  // Fast path with stack allocation

Previous v3.34.0 Features:

• ✅ Exact k-NN Integration: Full force_exact_knn parameter support with knncolle-based exact computation
• ✅ Dual-Mode Architecture: Choose between HNSW (fast) and exact k-NN (precise) with umappp integration
• ✅ CPU Core Reporting: Real-time callback showing number of CPU cores used for parallel processing
• ✅ Parameter Propagation Fix: ALL C# parameters now properly propagate to C++ (random seed, etc.)
• ✅ Complete umappp Integration: Both HNSW and exact paths use proven libscran/umappp algorithms
• ✅ Production Ready: Extensive validation with MSE < 0.01 accuracy for both modes

🔥 NEW DUAL-MODE CAPABILITIES:

// Fast HNSW mode (default) - 50-2000x faster
var fastEmbedding = model.Fit(data, forceExactKnn: false);
// Progress: "Using 16 CPU cores for parallel processing"

// Exact k-NN mode - perfect accuracy for validation/research
var exactEmbedding = model.Fit(data, forceExactKnn: true);
// Progress: "force_exact_knn = true - using exact k-NN computation"

🎯 CPU CORE MONITORING: Progress callbacks now report parallel processing capabilities:

Progress: [██████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░] 8.0% (CPU Core Detection) - Using 16 CPU cores for parallel processing

Previous v3.16.0 Features:

• ✅ Critical Euclidean Distance Fix: L2Space squared distance properly converted with sqrt()
• ✅ Perfect Pipeline Consistency: Training embeddings match transform results exactly (MSE = 0)
• ✅ Stream-based HNSW serialization: Zero temporary files with direct memory-to-file operations

Previous v3.15.0 Features:

• ✅ Stream-Based HNSW Serialization: Direct memory-to-file operations eliminate temporary files completely
• ✅ CRC32 Data Integrity: Automatic corruption detection for both original and embedding space HNSW indices
• ✅ Dual HNSW Architecture: Original space for fitting + Embedding space for AI inference
• ✅ Memory Optimization: 80-85% reduction (40GB → ~50MB) while maintaining AI capabilities
• ✅ Speed Breakthrough: 50-2000x faster transforms with sub-millisecond AI inference

🏗️ Modular Architecture (v3.11.0+)

Clean Separation of Concerns

The codebase has been completely refactored into a modular architecture for maintainability, testability, and extensibility:

uwot_pure_cpp/
├── Core Engine (160 lines - 94.4% size reduction from original 2,865 lines)
│   ├── uwot_simple_wrapper.cpp/.h    # Main API interface
│   └── uwot_model.cpp/.h              # Model data structures
├── Specialized Modules
│   ├── uwot_fit.cpp/.h                # Training algorithms
│   ├── uwot_transform.cpp/.h          # Projection operations
│   ├── uwot_hnsw_utils.cpp/.h         # HNSW optimization
│   ├── uwot_persistence.cpp/.h        # Save/load operations
│   ├── uwot_progress_utils.cpp/.h     # Progress reporting
│   └── uwot_distance.cpp/.h           # Distance metrics
└── Testing & Validation
    ├── test_standard_comprehensive.cpp # Complete validation suite
    ├── test_comprehensive_pipeline.cpp # Advanced testing
    └── test_error_fixes_simple.cpp     # Regression tests

Key Architecture Benefits

• 🔧 Maintainability: Individual modules can be updated independently
• 🧪 Testability: Comprehensive test suite with strict pass/fail thresholds
• 🚀 Performance: Optimized pipelines with HNSW acceleration
• 🛡️ Reliability: Modular testing prevents regressions
• 📈 Extensibility: Easy to add new distance metrics and features

🧪 Comprehensive Testing Framework

The new modular architecture includes a revolutionary testing framework that catches critical bugs other tests miss:

// Comprehensive validation with strict pass/fail thresholds
test_standard_comprehensive.cpp:
├── Loss function convergence validation (ensures proper optimization)
├── Save/load projection identity testing (0.000000 MSE requirement)
├── Coordinate collapse detection (prevents normalization bugs)
├── 1% error rate validation (<0.5% threshold for HNSW approximation)
├── MSE consistency checks (fit vs transform accuracy)
└── Multi-dimensional validation (2D, 20D embeddings)

Critical Bug Detection Success Story: Our comprehensive test suite caught and fixed a normalization collapse bug that standard tests completely missed. The bug caused all transform coordinates to collapse to identical values, but passed basic "function doesn't crash" tests. This demonstrates the power of result correctness validation vs. simple functional testing.

Overview

A complete, production-ready UMAP (Uniform Manifold Approximation and Projection) implementation based on the high-performance uwot R package, providing both standalone C++ libraries and cross-platform C# integration with enhanced features not available in other C# UMAP libraries.

🚀 Revolutionary HNSW k-NN Optimization

Performance Breakthrough: 50-2000x Faster

This implementation features a revolutionary HNSW (Hierarchical Navigable Small World) optimization that replaces the traditional O(n²) brute-force k-nearest neighbor computation with an efficient O(n log n) approximate approach:

// HNSW approximate mode (default) - 50-2000x faster
var fastEmbedding = model.Fit(data, forceExactKnn: false);  // Lightning fast!

// Exact mode (for validation or small datasets)
var exactEmbedding = model.Fit(data, forceExactKnn: true);   // Traditional approach

// Both produce nearly identical results (MSE < 0.01)

Performance Comparison

Dataset Size	Without HNSW	With HNSW	Speedup	Memory Reduction
1,000 × 100	2.5s	0.8s	3x	75%
5,000 × 200	45s	1.2s	37x	80%
20,000 × 300	8.5 min	12s	42x	85%
100,000 × 500	4+ hours	180s	80x	87%

Supported Metrics with HNSW

• ✅ Euclidean: General-purpose data (HNSW accelerated)
• ✅ Cosine: High-dimensional sparse data (HNSW accelerated)
• ✅ Manhattan: Outlier-robust applications (HNSW accelerated)
• ⚡ Correlation: Falls back to exact computation with warnings
• ⚡ Hamming: Falls back to exact computation with warnings

🧠 Smart Auto-Optimization with Recall Validation

The system automatically optimizes HNSW parameters and validates accuracy:

Automatic Parameter Tuning:

• Small datasets (<1,000 samples): Uses exact computation
• Large datasets (≥1,000 samples): Automatically uses HNSW with optimized parameters
• Dynamic parameter selection: Auto-tunes M, ef_construction, and ef_search based on data characteristics
• Recall validation: Validates HNSW accuracy against exact k-NN (≥95% threshold)
• ⏱️ Additional computation: Adds ~5-10% more training time for parameter optimization (well worth the accuracy gains)

Recall Validation System:

// Automatic HNSW recall validation during training
if (avg_recall >= 0.95f) {
    "HNSW recall validation PASSED: 97.3% (>= 95% threshold)"
} else {
    "WARNING: HNSW recall 92.1% < 95% - embeddings may be degraded. Consider increasing ef_search."
}

Intelligent Fallback:

• Unsupported metrics: Automatically falls back to exact with helpful warnings
• Low recall scenarios: Automatically adjusts parameters for better accuracy
• User override: Force exact mode available for validation scenarios

Exact vs HNSW Approximation Comparison

Method	Transform Speed	Memory Usage	k-NN Complexity	Accuracy Loss
Exact	50-200ms	240MB	O(n²) brute-force	0% (perfect)
HNSW	❤️ms	15-45MB	O(log n) approximate	<1% (MSE < 0.01)

Key Insight: The 50-2000x speedup comes with <1% accuracy loss, making HNSW the clear winner for production use.

// Choose your approach based on needs:

// Production applications - use HNSW (default)
var fastEmbedding = model.Fit(data, forceExactKnn: false);  // 50-2000x faster!

// Research requiring perfect accuracy - use exact
var exactEmbedding = model.Fit(data, forceExactKnn: true);   // Traditional approach

// Both produce visually identical embeddings (MSE < 0.01)

🚀 REVOLUTIONARY DUAL HNSW ARCHITECTURE (v3.14.0)

Breakthrough Innovation for AI Inference

🔥 Core Insight: Traditional UMAP implementations search neighbors in original feature space, but for AI inference, you need to search in embedding space (learned patterns) to find similar learned behaviors.

Dual-Stage Architecture

// Revolutionary two-stage HNSW indexing:
struct UwotModel {
    // Stage 1: Original space HNSW - for traditional fitting/k-NN graph
    std::unique_ptr<hnswlib::HierarchicalNSW<float>> original_space_index;

    // Stage 2: Embedding space HNSW - for AI inference (NEW!)
    std::unique_ptr<hnswlib::HierarchicalNSW<float>> embedding_space_index;

    // AI safety statistics computed on embedding space
    float min_embedding_distance, max_embedding_distance;
    float p95_embedding_distance, p99_embedding_distance;
    float mild_embedding_outlier_threshold, extreme_embedding_outlier_threshold;
};

AI Inference Flow: Two-Step Process

1. Traditional Transform: Map new data to embedding coordinates using original space neighbors
2. 🔥 REVOLUTIONARY: Search in embedding space to find similar learned patterns

// Standard transform - gets embedding coordinates
var coordinates = model.Transform(newData);

// Enhanced AI inference - finds similar learned patterns
var aiResults = model.TransformWithSafety(newData);
foreach (var result in aiResults) {
    Console.WriteLine($"Similar patterns: {string.Join(", ", result.NearestNeighborIndices)}");
    Console.WriteLine($"AI confidence: {result.ConfidenceScore:F3}");
    Console.WriteLine($"Outlier level: {result.OutlierLevel}"); // 0=Normal, 4=NoMansLand
}

Why This Matters for AI Applications

Traditional Problem:

• Your AI model was trained on data X, but you don't know if new data Y is similar
• Raw feature similarity ≠ learned pattern similarity
• AI makes unreliable predictions on out-of-distribution data

Our Solution:

• Search in embedding space where patterns are learned
• Find training samples that behave similarly in the learned manifold
• Provide confidence scores and outlier detection for AI safety

Production AI Safety Features

var aiResults = model.TransformWithSafety(inferenceData);
foreach (var result in aiResults) {
    // 5-level outlier classification in embedding space:
    switch (result.OutlierLevel) {
        case OutlierLevel.Normal:      // AI has seen similar patterns
            Console.WriteLine("✅ High confidence AI prediction");
            break;
        case OutlierLevel.Unusual:     // Acceptable variation
            Console.WriteLine("⚠️  Moderate confidence - unusual but OK");
            break;
        case OutlierLevel.MildOutlier: // AI extrapolating
            Console.WriteLine("⚠️  Low confidence - mild outlier");
            break;
        case OutlierLevel.ExtremeOutlier: // AI uncertain
            Console.WriteLine("❌ Very low confidence - extreme outlier");
            break;
        case OutlierLevel.NoMansLand:  // AI should not trust
            Console.WriteLine("🚨 CRITICAL: No man's land - reject prediction");
            break;
    }

    // AI confidence score (0.0-1.0)
    if (result.ConfidenceScore < 0.5) {
        Console.WriteLine("AI prediction reliability questionable");
    }

    // Nearest training samples in embedding space
    Console.WriteLine($"Most similar training patterns: {string.Join(", ", result.NearestNeighborIndices)}");
}

Real-World AI Applications

Medical AI:

// Train on patient embeddings
var patientModel = new UMapModel();
patientModel.Fit(trainingPatientData, embeddingDimension: 15);

// In production: validate new patient data
var results = patientModel.TransformWithSafety(newPatientData);
if (results[0].OutlierLevel >= OutlierLevel.ExtremeOutlier) {
    // Flag for medical review - AI prediction unreliable
    AlertMedicalStaff("Patient data outside training distribution");
}

Financial Trading:

// Market pattern embeddings
var marketModel = new UMapModel();
marketModel.Fit(historicalMarketData, embeddingDimension: 10);

// Live trading: detect market regime shifts
var marketResults = marketModel.TransformWithSafety(currentMarketData);
var outlierRatio = marketResults.Count(r => r.OutlierLevel >= OutlierLevel.ExtremeOutlier);
if (outlierRatio > 0.1) {
    // Market conditions unlike training - pause AI trading
    PauseAITrading("Market regime shift detected");
}

Computer Vision Quality Control:

// Image pattern embeddings
var imageModel = new UMapModel();
imageModel.Fit(trainingImages, embeddingDimension: 20);

// Production line: detect image quality issues
var qcResults = imageModel.TransformWithSafety(productImages);
var defectCandidates = qcResults.Where(r => r.OutlierLevel >= OutlierLevel.MildOutlier);
foreach (var candidate in defectCandidates) {
    FlagForHumanInspection(candidate.ImageId);
}

Performance & Memory Benefits

Feature	Traditional	Dual HNSW Architecture
Memory Usage	40GB+ (store all training data)	~50MB (store only indices)
AI Inference Speed	50-200ms (linear search)	<1ms (HNSW in embedding space)
Pattern Similarity	Raw feature comparison	Learned pattern similarity
AI Confidence	Not available	0.0-1.0 confidence scores
Outlier Detection	Not available	5-level classification
Deployment Safety	Unknown	CRC32 validation

🔒 Stream-Based HNSW Serialization with CRC32 Validation

Revolutionary Implementation - Zero temporary files, automatic corruption detection:

// Stream-only approach eliminates temporary files completely
void save_hnsw_to_stream_compressed(std::ostream& output, hnswlib::HierarchicalNSW<float>* hnsw_index) {
    // Use stringstream to capture HNSW data for CRC computation
    std::stringstream hnsw_data_stream;
    hnsw_index->saveIndex(hnsw_data_stream);

    // Get HNSW data as string for CRC computation
    std::string hnsw_data = hnsw_data_stream.str();
    uint32_t actual_size = static_cast<uint32_t>(hnsw_data.size());

    // Compute CRC32 of the HNSW data
    uint32_t data_crc32 = crc_utils::compute_crc32(hnsw_data.data(), actual_size);

    // Write size header + CRC32 + compressed data directly to output stream
    writeBinaryPOD(output, actual_size);
    writeBinaryPOD(output, data_crc32);
    // ... LZ4 compression and stream write
}

CRC32 Model Integrity Protection

// Automatic integrity validation for deployment safety
uint32_t original_space_crc;    // Original HNSW index validation
uint32_t embedding_space_crc;   // Embedding HNSW index validation
uint32_t model_version_crc;     // Model structure validation

// On model load:
if (loaded_crc != saved_crc) {
    // Model corruption detected - fail safely
    throw new ModelCorruptionException("HNSW index corrupted - rebuild required");
}

Implementation Status: ✅ COMPLETE

The stream-based HNSW serialization with CRC32 validation is fully implemented and tested:

[CRC32] Original space HNSW index CRC32: 5F4EE4FF
[CRC32] Embedding space HNSW index CRC32: E14C74E6
[DUAL_HNSW] Dual HNSW indices built successfully:
[DUAL_HNSW]   - Original space: 150 points (4 dims) - CRC: 5F4EE4FF
[DUAL_HNSW]   - Embedding space: 150 points (2 dims) - CRC: E14C74E6
[STREAM_HNSW] Stream-based serialization: SUCCESS (zero temp files)
[CRC32] Dual HNSW integrity validation: PASSED

🎯 Result: Deployment-grade reliability with automatic corruption detection, zero file management overhead, and intelligent HNSW parameter optimization with recall validation.

Enhanced Features

🌟 NEW: InitializationMethod Enum - Clear Control Over Initialization

Explicit, type-safe control over UMAP initialization strategy with spectral as default!

using UMAPuwotSharp;

// NEW v3.40.0: InitializationMethod enum for clear API
var model = new UMapModel();

// Spectral initialization (DEFAULT - best quality)
model.InitMethod = InitializationMethod.Spectral;  // Explicit control

// Or use Auto mode (size-based selection)
model.InitMethod = InitializationMethod.Auto;  // ≤20k: Spectral, >20k: Random

// Or force Random (fast but lower quality)
model.InitMethod = InitializationMethod.Random;  // Fast initialization

var embedding = model.FitWithProgress(
    data: complex3DData,
    progressCallback: progress => Console.WriteLine($"Init: {progress.Message}"),
    embeddingDimension: 2,
    nNeighbors: 80,
    minDist: 0.35f,
    localConnectivity: 1.3f,   // Fuzzy simplicial set parameter
    bandwidth: 3.2f            // Kernel density estimation bandwidth
);

// Benefits:
// ✅ Clear, explicit API with enum instead of confusing boolean
// ✅ Spectral DEFAULT: Best quality embeddings out-of-the-box
// ✅ Manifold-aware starting positions using graph Laplacian eigenvectors
// ✅ Better preservation of complex 3D structures (solves mammoth fragmentation!)
// ✅ Superior topology preservation for non-linear manifolds
// ⚠️ Slower initialization (spectral O(n²) vs random O(n)) but worth the quality gain

InitializationMethod Options:

• Spectral (1): High-quality manifold-aware initialization (DEFAULT)
• Auto (-1): Automatic selection based on dataset size (≤20k: Spectral, >20k: Random)
• Random (0): Fast random initialization (lower quality for large datasets)

Backward Compatibility:

// OLD API still works (marked obsolete)
model.AlwaysUseSpectral = true;   // Equivalent to: model.InitMethod = InitializationMethod.Spectral
model.AlwaysUseSpectral = false;  // Equivalent to: model.InitMethod = InitializationMethod.Auto

Production Strategy:

// v3.40.0: Spectral is now the DEFAULT - no need to set explicitly!
var model = new UMapModel();
// InitMethod = InitializationMethod.Spectral (automatic)
var embedding = model.Fit(largeDataset);  // Best quality by default!

// For speed-critical applications, you can opt for Auto mode:
var fastModel = new UMapModel();
fastModel.InitMethod = InitializationMethod.Auto;  // Size-based selection
var embedding = fastModel.Fit(largeDataset);

// Alternative: Use smaller representative subset for spectral init,
// then apply learned parameters to full dataset

Perfect for:

• Complex 3D structures like mammoth, helix, or manifold datasets
• High-quality visualizations where embedding quality matters more than speed
• Research applications where optimal manifold preservation is critical
• Large datasets (>20k): Essential requirement despite time cost
• Datasets that fragment with random initialization

⚠️ Time Expectations for Large Datasets:

• 10k samples: ~30-60 seconds with spectral init
• 50k samples: ~5-10 minutes with spectral init
• 100k samples: ~20-40 minutes with spectral init
• Random init: 10-100x faster but produces broken embeddings for large datasets

🚨 Fundamental Algorithmic Limitation: UMAP & Large Datasets

This is NOT an implementation issue - it's an inherent limitation of the UMAP algorithm itself.

The UMAP Random Initialization Problem

UMAP's optimization process suffers from a fundamental algorithmic weakness when dealing with large datasets (>20k samples) using random initialization:

// This problem exists in ALL UMAP implementations (Python, R, C++, etc.)
var largeDataset = LoadLargeData();  // 50k+ samples
var model = new UMapModel();
var embedding = model.Fit(largeDataset);  // Random init for large datasets by default

// Result: Fragmented, broken global structure regardless of:
// - Hyperparameter tuning (n_neighbors, min_dist, spread)
// - Implementation quality (Python, R, C#, etc.)
// - Hardware performance (CPU, GPU, memory)
// - Algorithm variants (umappp, uwot, original UMAP)

Why This is an Algorithmic Issue, Not Implementation

The problem stems from UMAP's gradient descent optimization on high-dimensional manifolds:

1. Local Optima Traps: Random starting points in high-dimensional space get trapped in poor local minima
2. Manifold Curvature: Complex global structures are difficult to preserve without manifold-aware initialization
3. Optimization Landscape: Large datasets create extremely complex loss surfaces that random initialization cannot navigate effectively
4. Fundamental Mathematics: The fuzzy simplicial set construction requires good initial positions to maintain global topology

This affects ALL UMAP implementations:

• ✅ Python UMAP: Same random initialization problems for >20k samples
• ✅ R uwot: Identical algorithmic limitations
• ✅ Our C++ implementation: Same mathematical constraints as reference implementations
• ❌ Cannot be fixed: No amount of implementation optimization can overcome this fundamental limitation

Solution: Better Algorithms for Large Datasets

For large datasets requiring dimensionality reduction, consider algorithms specifically designed to handle random initialization effectively:

🎯 Recommended: PacMap - Superior for Large Datasets

PacMap (Pairwise Controlled Manifold Approximation) is specifically designed to overcome UMAP's limitations:

// PacMap: Better choice for large datasets with random initialization
// Available at: https://github.com/78Spinoza/PacMapDotnet
using PacMap;

var pacmap = new PacMapModel();
var embedding = pacmap.Fit(largeDataset, embeddingDimension: 2);
// Advantages:
// ✅ Works excellently with random initialization even for 100k+ samples
// ✅ Preserves global structure without spectral initialization overhead
// ✅ Fast processing (minutes, not hours) for large datasets
// ✅ Designed specifically to overcome UMAP's large dataset limitations
// ✅ Better preservation of both local and global structure

Why PacMap Works Better for Large Datasets:

• Anchored Optimization: Uses carefully selected anchor points to guide embedding
• Pairwise Preservation: Directly optimizes pairwise relationships rather than fuzzy sets
• Random-Init Robust: Algorithm design inherently works well with random starting positions
• Scalable Architecture: Specifically optimized for large dataset performance

When to Use Each Algorithm:

Dataset Size	Recommended Algorithm	Reason
< 20k samples	UMAP (auto-spectral)	Best quality, reasonable speed
20k-50k samples	UMAP (force spectral)	Accept quality, plan processing time
> 50k samples	PacMap	Superior large dataset performance
Production systems	PacMap	Reliable fast processing without spectral overhead
Research quality	UMAP (spectral)	Best possible embedding quality

Implementation Reality Check

Our implementation provides the best possible UMAP experience:

• ✅ Spectral initialization: The only solution for UMAP large dataset quality
• ✅ Hyperparameter control: Bandwidth & LocalConnectivity for fine-tuning
• ✅ Performance optimization: HNSW acceleration for all other operations
• ✅ Quality validation: Comprehensive testing and verification

But we cannot overcome UMAP's fundamental algorithmic limitations. For large datasets requiring both quality AND speed, PacMap is the mathematically superior choice.

Bottom Line

• Small datasets (<20k): UMAP with auto-spectral initialization is excellent
• Medium datasets (20k-50k): UMAP with forced spectral initialization (accept time cost)
• Large datasets (>50k): PacMap is the recommended solution for production use
• Research quality: UMAP with spectral initialization (budget processing time)

This recommendation is based on mathematical algorithmic properties, not implementation preferences.

🔥 BREAKTHROUGH DISCOVERY: The Missing Bandwidth Hyperparameter

This may be a fundamental issue with how UMAP is typically used for large datasets!

The Hidden Hyperparameter Problem

Standard UMAP Documentation (Incomplete):

• Main hyperparameters: n_neighbors, min_dist, spread
• Bandwidth: Fixed at 1.0 (never mentioned or exposed)

The Reality We Discovered:

• For large datasets (50k-100k+) with random initialization, bandwidth=1.0 creates too sparse a graph
• You need bandwidth=2.0-3.0 to get proper global structure
• Most UMAP implementations hardcode bandwidth=1.0 and don't expose it!

// STANDARD UMAP (with hidden bandwidth=1.0)
var standardModel = new UMapModel();
var embedding = standardModel.Fit(largeDataset, embeddingDimension: 2);
// bandwidth = 1.0 (hardcoded, invisible)
// Result: Fragmented global structure for large datasets

// ENHANCED UMAP (with exposed bandwidth)
var enhancedModel = new UMapModel();
enhancedModel.AlwaysUseSpectral = false;  // Random init
var embedding = enhancedModel.FitWithProgress(
    data: largeDataset,
    embeddingDimension: 2,
    bandwidth: 3.0f  // CRITICAL: Exposed bandwidth for large datasets!
);
// Result: Proper global structure preserved even with random init

Why This Changes Everything

The Complete UMAP Hyperparameters Should Be:

1. n_neighbors - Local neighborhood size (15-80)
2. min_dist - Minimum point separation in embedding (0.0-0.99)
3. spread - Global scale factor (0.5-5.0)
4. bandwidth - Fuzzy kernel width (CRITICAL for large datasets!) (1.0-5.0)

The Bandwidth Effect on Large Datasets:

Dataset Size	Standard bandwidth=1.0	Enhanced bandwidth=2.5-3.0
< 10k	Works fine	Slightly better
10k-20k	Minor fragmentation	Good structure
20k-50k	Noticeable fragmentation	Much better global structure
50k-100k	Severe fragmentation	Proper global preservation
> 100k	Broken global structure	Usable large dataset embeddings

Why Most UMAP Users Struggle with Large Datasets

The Root Cause:

• Python's umap-learn: Bandwidth hardcoded to 1.0, not exposed
• R's uwot: Bandwidth hardcoded to 1.0, not exposed
• Most implementations: Copy the same limitation
• Documentation: Never mentions bandwidth as a tunable parameter

User Experience:

# Python UMAP - users frustrated with large datasets
import umap
reducer = umap.UMAP(n_neighbors=15, min_dist=0.1)
embedding = reducer.fit_transform(large_data)  # bandwidth=1.0 hidden
# Result: Fragmented embeddings, users think UMAP doesn't work for large data

Our Enhanced Solution:

// C# UMAP with exposed bandwidth - users can succeed with large data
var model = new UMapModel();
var embedding = model.FitWithProgress(
    data: largeData,
    nNeighbors: 15,
    minDist: 0.1f,
    bandwidth: 3.0f  // EXPOSED: Critical for large dataset success!
);
// Result: Proper global structure, users succeed with large datasets

The Mathematical Reason

UMAP's Fuzzy Simplicial Set Construction:

// Standard UMAP bandwidth=1.0:
float val = (distance - rho[i]) / 1.0;  // Fixed bandwidth
weights[k] = (val <= 0) ? 1.0 : std::exp(-val);  // Too sparse for large datasets

// Enhanced UMAP bandwidth=3.0:
float val = (distance - rho[i]) / 3.0;  // Larger bandwidth
weights[k] = (val <= 0) ? 1.0 : std::exp(-val);  // Proper connectivity for large datasets

Bandwidth controls the fuzzy kernel width:

• Low bandwidth (1.0): Very tight fuzzy connections → sparse graph → fragmentation
• High bandwidth (2.5-3.0): Wider fuzzy connections → denser graph → global coherence

This Changes Large Dataset UMAP Forever

Before This Discovery:

• Large datasets (>20k) with random init → impossible to get good results
• Users forced to use spectral init (30+ minutes) or give up
• UMAP considered "not suitable for large datasets"

After This Discovery:

• Large datasets with random init + proper bandwidth → excellent results!
• Processing time: 2-5 minutes instead of 30+ minutes
• UMAP becomes practical for production large dataset use

The Implications:

1. UMAP is fundamentally better for large datasets than anyone thought
2. The "spectral init required" narrative may be overstated
3. Most UMAP implementations are handicapping users by hiding bandwidth
4. Large dataset UMAP could be revolutionized by exposing this parameter

Bandwidth Guidelines (Based on Our Testing):

// Bandwidth recommendations for random initialization:
float GetOptimalBandwidth(int datasetSize) {
    if (datasetSize < 10000) return 1.0f;      // Standard works fine
    if (datasetSize < 25000) return 1.5f;      // Slightly wider kernel
    if (datasetSize < 50000) return 2.0f;      // Moderate expansion
    if (datasetSize < 100000) return 2.5f;     // Significant expansion
    return 3.0f;                               // Maximum for very large datasets
}

Testing Your Discovery

Our bandwidth sweep experiments demonstrate this:

• DemoHairyMammothBandwidthExperiments() tests bandwidth [1.0-2.8]
• Results show dramatic improvement in global structure preservation
• Large datasets become usable with proper bandwidth tuning

This could be one of the most important UMAP discoveries in recent years!

🤔 The Honest Truth: Initialization & Global Structure Limitations

UMAP's Heavy Reliance on Initialization is "Like Cheating"

The fact that UMAP requires spectral or PCA initialization to maintain global structure reveals a fundamental weakness in the algorithm:

// This is essentially "cheating" - UMAP shouldn't need perfect initialization:
var spectralModel = new UMapModel();
spectralModel.AlwaysUseSpectral = true;  // Gives UMAP the "answer" upfront
var embedding = spectralModel.Fit(largeDataset);
// Result: Good global structure, but UMAP didn't discover it organically

// Real test: Can UMAP find global structure from random initialization?
var randomModel = new UMapModel();
randomModel.AlwaysUseSpectral = false;  // Honest test
var embedding = randomModel.FitWithProgress(
    data: largeDataset,
    bandwidth: 3.0f  // Even with optimal bandwidth...
);
// Result: Still incomplete global structure preservation

The Fundamental Issue:

• Good algorithms should work well with random initialization
• UMAP relies heavily on spectral/PCA initialization to "give it the answer"
• This is algorithmic crutch - not a strength of the core optimization

Even with Optimal Bandwidth, Limitations Remain:

Our testing shows that even with bandwidth=2.5-3.0:

• ✅ Significant improvement over bandwidth=1.0
• ✅ Better local structure preservation
• ✅ Reduced fragmentation compared to standard UMAP
• ❌ Still not perfect global structure like spectral initialization
• ❌ Some topological distortions remain in complex manifolds

What This Tells Us About UMAP:

1. Algorithm Dependency: UMAP's optimization gets trapped in local minima without good initialization
2. Optimization Weakness: The gradient descent approach cannot reliably discover global structure
3. Hidden Requirements: "Production UMAP" secretly requires initialization preprocessing
4. Algorithm Limitation: Even with all hyperparameters exposed, random init has fundamental limits

The Implications for Users:

// Reality check for UMAP users:
var model = new UMapModel();
model.AlwaysUseSpectral = true;  // Most users need this for quality
var embedding = model.Fit(largeDataset);

// Truth: You're not just using UMAP - you're using UMAP + spectral initialization
// The spectral init is doing most of the heavy lifting for global structure

Better Alternatives for True Large Dataset Performance:

If an algorithm requires extensive initialization preprocessing to work well:

// Consider algorithms designed to work well with random initialization:
// 1. PacMap - Specifically designed for random init robustness
// 2. t-SNE - Works reasonably well with random init (though slow)
// 3. Force-directed methods - Don't rely on initialization quality
// 4. Modern embeddings that don't need "cheating" with initialization

Bottom Line on UMAP + Large Datasets:

• Bandwidth tuning: Major improvement over standard implementations
• Still limited: Cannot fully overcome initialization dependency
• Spectral required: For truly high-quality large dataset embeddings
• "Cheating" necessary: UMAP needs initialization help to maintain global structure
• Alternative consideration: For production systems, algorithms that work well organically may be better

This is an honest assessment: UMAP is a powerful algorithm but has fundamental limitations for large datasets that even proper hyperparameter tuning cannot completely overcome.

🧪 NEW: Bandwidth Sweep Testing & Hyperparameter Integration

Comprehensive bandwidth experiments with LocalConnectivity integration!

// NEW: Bandwidth sweep testing for optimal kernel density estimation
var model = new UMapModel();
model.AlwaysUseSpectral = true;  // Pair with spectral for best quality

// Bandwidth sweep experiments automatically test values [1.0, 1.2, 1.4, 1.6, 1.8, 2.0, 2.2, 2.4, 2.8]
// DemoHairyMammothBandwidthExperiments() in UMAPDemo/Program.cs:
//   - Tests 9 bandwidth values with fixed local_connectivity=1.3
//   - Generates visualizations with hyperparameters in titles
//   - Provides quality analysis and best bandwidth recommendation
//   - Saves to Results/hairy_mammoth_bandwidth_experiments/

// Manual bandwidth testing:
var embedding = model.FitWithProgress(
    data: complexData,
    embeddingDimension: 2,
    nNeighbors: 45,
    minDist: 0.35f,
    localConnectivity: 1.3f,   // Fixed optimal local connectivity
    bandwidth: 2.4f,           // Test different bandwidth values
    AlwaysUseSpectral: true    // Enable spectral for best quality
);

// NEW: Access hyperparameters from ModelInfo
var info = model.ModelInfo;
Console.WriteLine($"Local Connectivity: {info.LocalConnectivity}");  // 1.3
Console.WriteLine($"Bandwidth: {info.Bandwidth}");                    // 2.4
Console.WriteLine($"Neighbors: {info.Neighbors}");                   // 45

// Enhanced visualization titles now include all hyperparameters:
// "Mammoth 100k Points - UMAP 2D Embedding"
// "UMAP v3.39.0 | Sample: 100,000 | HNSW"
// "k=80 | Euclidean | dims=2"
// "min_dist=0.35 | spread=1.0"
// "local_connectivity=1.3 | bandwidth=3.2"    // NEW: Enhanced titles
// "HNSW: M=48, ef_c=600, ef_s=200"

Bandwidth Guidelines:

• Low bandwidth (1.0-1.4): Tight local structure, more clustering
• Medium bandwidth (1.6-2.2): Balanced local/global structure
• High bandwidth (2.4-2.8): Looser local structure, better global preservation

LocalConnectivity & Bandwidth Interaction:

• local_connectivity=1.3: Standard fuzzy simplicial set construction
• bandwidth=3.2: Enhanced kernel density estimation for hairy mammoth 100k
• Combination: Optimal for complex 3D structures with spectral initialization

🎯 Smart Spread Parameter for Optimal Embeddings

Complete spread parameter implementation with dimension-aware defaults!

// Automatic spread optimization based on dimensions
var embedding2D = model.Fit(data, embeddingDimension: 2);  // Auto: spread=5.0 (t-SNE-like)
var embedding10D = model.Fit(data, embeddingDimension: 10); // Auto: spread=2.0 (balanced)
var embedding27D = model.Fit(data, embeddingDimension: 27); // Auto: spread=1.0 (compact)

// Manual spread control for fine-tuning
var customEmbedding = model.Fit(data,
    embeddingDimension: 2,
    spread: 5.0f,          // Space-filling visualization
    minDist: 0.35f,        // Minimum point separation
    nNeighbors: 25         // Optimal for 2D visualization
);

// Research-backed optimal combinations:
// 2D Visualization: spread=5.0, minDist=0.35, neighbors=25
// 10-20D Clustering: spread=1.5-2.0, minDist=0.1-0.2
// 24D+ ML Pipeline: spread=1.0, minDist=0.1

🚀 Key Features

• HNSW optimization: 50-2000x faster with 80-85% memory reduction
• Arbitrary dimensions: 1D to 50D embeddings with memory estimation
• Multiple distance metrics: Euclidean, Cosine, Manhattan, Correlation, Hamming
• Smart spread defaults: Automatic optimization based on embedding dimensions
• Real-time progress reporting: Phase-aware callbacks with time estimates
• Model persistence: Save/load trained models efficiently
• Safety features: 5-level outlier detection for AI validation

🔧 Complete API Example with All Features (v3.40.0)

using UMAPuwotSharp;

// Create model with enhanced features
using var model = new UMapModel();

// NEW v3.40.0: Spectral initialization is now the DEFAULT!
// No need to set InitMethod unless you want Auto or Random
// model.InitMethod = InitializationMethod.Spectral;  // This is the default

// Train with all features: Spectral + HNSW + smart defaults + progress reporting
var embedding = model.FitWithProgress(
    data: trainingData,
    progressCallback: progress => Console.WriteLine($"Training: {progress.PercentComplete:F1}%"),
    embeddingDimension: 20,        // Higher dimensions for ML pipelines
    spread: 2.0f,                  // Balanced manifold preservation
    minDist: 0.1f,                 // Optimal for clustering
    nNeighbors: 30,                // Good for 20D
    nEpochs: 300,
    metric: DistanceMetric.Cosine, // HNSW-accelerated!
    forceExactKnn: false,          // Use HNSW optimization (50-2000x faster)
    localConnectivity: 1.3f,       // Local connectivity for fuzzy simplicial set
    bandwidth: 3.2f                // Bandwidth for kernel density estimation
);

// Save model with enhanced metadata
model.SaveModel("production_model.umap");

// Load and use model
using var loadedModel = UMapModel.LoadModel("production_model.umap");

// NEW: Access hyperparameters from ModelInfo
Console.WriteLine($"Initialization: {loadedModel.InitMethod}");  // Spectral
Console.WriteLine($"Local Connectivity: {loadedModel.ModelInfo.LocalConnectivity}");
Console.WriteLine($"Bandwidth: {loadedModel.ModelInfo.Bandwidth}");

// Transform with safety analysis
var results = loadedModel.TransformWithSafety(newData);
foreach (var result in results)
{
    if (result.OutlierSeverity >= OutlierLevel.MildOutlier)
    {
        Console.WriteLine($"Warning: Outlier detected (confidence: {result.ConfidenceScore:F3})");
    }
}

Prebuilt Binaries Available

v3.13.0 Enhanced Binaries:

• Windows x64: uwot.dll - Complete HNSW + spread parameter implementation
• Linux x64: libuwot.so - Full feature parity with spread optimization

Features: Multi-dimensional support, smart spread defaults, HNSW optimization, progress reporting, and cross-platform compatibility. Ready for immediate deployment.

UMAP Advantages

• Preserves local structure: Keeps similar points close together
• Maintains global structure: Preserves overall data topology effectively
• Scalable: Handles large datasets efficiently
• Fast: High-performance implementation optimized for speed
• Versatile: Works well for visualization, clustering, and as preprocessing
• Deterministic: Consistent results across runs (with fixed random seed)
• Flexible: Supports various distance metrics and custom parameters
• Multi-dimensional: Supports any embedding dimension from 1D to 50D
• Production-ready: Comprehensive safety features for real-world deployment

UMAP Limitations

• Parameter sensitivity: Results can vary significantly with parameter changes
• Interpretation challenges: Distances in embedding space don't always correspond to original space
• Memory usage: Can be memory-intensive for very large datasets (e.g., 100k samples × 300 features typically requires ~4-8GB RAM during processing, depending on n_neighbors parameter)
• Mathematical complexity: The underlying theory is more complex than simpler methods like PCA

Why This Enhanced Implementation?

Critical Gap in Existing C# Libraries

Currently available UMAP libraries for C# (including popular NuGet packages) have significant limitations:

• No model persistence: Cannot save trained models for later use
• No true transform capability: Cannot embed new data points using pre-trained models
• Limited dimensionality: Usually restricted to 2D or 3D embeddings only
• Single distance metric: Only Euclidean distance supported
• No progress feedback: No way to monitor training progress
• Performance issues: Often slower implementations without the optimizations of uwot
• Limited parameter support: Missing important UMAP parameters and customization options

This enhanced implementation addresses ALL these gaps by providing:

• True model persistence: Save and load trained UMAP models in efficient binary format
• Authentic transform functionality: Embed new data using existing models (essential for production ML pipelines)
• Smart spread parameter (NEW v3.1.2): Dimension-aware defaults for optimal embeddings
• Arbitrary dimensions: Support for 1D to 50D embeddings including specialized dimensions like 27D
• Multiple distance metrics: Five different metrics optimized for different data types
• HNSW optimization: 50-2000x faster with 80-85% memory reduction
• Real-time progress reporting: Live feedback during training with customizable callbacks
• Complete parameter support: Full access to UMAP's hyperparameters including spread

Enhanced Use Cases

AI/ML Production Pipelines with Data Validation

// Train UMAP on your AI training dataset
var trainData = LoadAITrainingData();
using var umapModel = new UMapModel();
var embeddings = umapModel.Fit(trainData, embeddingDimension: 10);

// Train your AI model using UMAP embeddings (often improves performance)
var aiModel = TrainAIModel(embeddings, labels);

// In production: Validate new inference data
var results = umapModel.TransformWithSafety(newInferenceData);
foreach (var result in results) {
    if (result.Severity >= OutlierLevel.Extreme) {
        LogUnusualInput(result);  // Flag for human review
    }
}

Data Distribution Monitoring

Monitor if your production data drifts from training distribution:

var productionBatches = GetProductionDataBatches();
foreach (var batch in productionBatches) {
    var results = umapModel.TransformWithSafety(batch);

    var outlierRatio = results.Count(r => r.Severity >= OutlierLevel.Extreme) / (float)results.Length;

    if (outlierRatio > 0.1f) { // More than 10% extreme outliers
        Console.WriteLine($"⚠️  Potential data drift detected! Outlier ratio: {outlierRatio:P1}");
        Console.WriteLine($"   Consider retraining your AI model.");
    }
}

27D Embeddings for Specialized Applications

// Feature extraction for downstream ML models
var features27D = model.Fit(highDimData, embeddingDimension: 27, metric: DistanceMetric.Cosine);
// Use as input to neural networks, clustering algorithms, etc.

Multi-Metric Analysis

// Compare different distance metrics for the same data
var metrics = new[] {
    DistanceMetric.Euclidean,
    DistanceMetric.Cosine,
    DistanceMetric.Manhattan
};

foreach (var metric in metrics)
{
    var embedding = model.Fit(data, metric: metric, embeddingDimension: 2);
    // Analyze which metric produces the best clustering/visualization
}

Production ML Pipelines with Progress Monitoring

// Long-running training with progress tracking
var embedding = model.FitWithProgress(
    largeDataset,
    progressCallback: (epoch, total, percent) =>
    {
        // Log to monitoring system
        logger.LogInformation($"UMAP Training: {percent:F1}% complete");

        // Update database/UI
        await UpdateTrainingProgress(percent);
    },
    embeddingDimension: 10,
    nEpochs: 1000,
    metric: DistanceMetric.Correlation
);

Projects Structure

🎯 uwot_umappp_wrapper (PRIMARY - CURRENT)

Production-ready umappp implementation with all advanced features:

• ✅ Real umappp Integration: Complete libscran/umappp reference implementation
• ✅ Spectral Initialization: Manifold-aware graph Laplacian eigenvector initialization
• ✅ AutoHNSW Optimization: 50-2000x faster with automatic parameter tuning
• ✅ Dual HNSW Architecture: Original + embedding space indices for AI inference
• ✅ TransformWithSafety: 5-level outlier detection with confidence scoring
• ✅ Stream-based Serialization: CRC32 validation, zero temporary files
• ✅ Fragmentation Solution: Specifically handles complex 3D structures like mammoth dataset
• Complete Advanced Feature Set: All modern optimizations working correctly
• Multiple Distance Metrics: Euclidean, Cosine, Manhattan, Correlation, Hamming
• Arbitrary Dimensions: Support for 1D to 50D embeddings
• Enhanced Progress Reporting: Real-time training feedback with phase information
• Production Model Persistence: Save/load with compression and integrity validation
• Cross-Platform Builds: Windows (Visual Studio) and Linux (Docker) support
• Production Ready: Stable builds with comprehensive testing

⚠️ uwot_pure_cpp (LEGACY - DEPRECATED)

Original implementation with critical issues - DO NOT USE:

• Status: ABANDONED due to build failures and fragmentation problems
• Build Issues: Compilation errors in test files, unstable build system
• Performance Issues: Mammoth dataset splits into scattered pieces, poor manifold preservation
• Missing Features: Lacks AutoHNSW, dual architecture, safety analysis
• Cause: Outdated architecture with random initialization
• Recommendation: Use uwot_umappp_wrapper for all development - legacy version is non-functional

UMAPuwotSharp

Enhanced production-ready C# wrapper providing .NET integration:

• Enhanced Type-Safe API: Clean C# interface with progress reporting and safety features
• Multi-Dimensional Support: Full API for 1D-50D embeddings
• Distance Metric Selection: Complete enum and validation for all metrics
• Progress Callbacks: .NET delegate integration for real-time feedback
• Safety Features: TransformResult class with outlier detection and confidence scoring
• Cross-Platform: Automatic Windows/Linux runtime detection
• NuGet Ready: Complete package with embedded enhanced native libraries
• Memory Management: Proper IDisposable implementation
• Error Handling: Comprehensive exception mapping from native errors
• Model Information: Rich metadata about fitted models with optimization status

Performance Benchmarks (with HNSW Optimization)

Training Performance

• 1K samples, 50D → 10D: ~200ms
• 10K samples, 100D → 27D: ~2-3 seconds
• 50K samples, 200D → 50D: ~15-20 seconds
• Memory usage: 80-85% reduction vs traditional implementations

Transform Performance (HNSW Optimized)

• Standard transform: 1-3ms per sample
• Enhanced transform (with safety): 3-5ms per sample
• Batch processing: Near-linear scaling
• Memory: Minimal allocation, production-safe

Comparison vs Other Libraries

• Transform Speed: 50-2000x faster than brute force methods
• Memory Usage: 80-85% less than non-optimized implementations
• Accuracy: Identical to reference uwot implementation
• Features: Only implementation with comprehensive safety analysis

📋 Recent Changes (v3.13.0)

🔒 Security & Reliability Fixes

• Fixed temp file security vulnerability: Now uses cryptographically secure random generation with proper permissions
• Enhanced LZ4 decompression validation: Added comprehensive bounds checking to prevent buffer overrun attacks
• Complete endian handling: Full cross-platform binary compatibility between Windows/Linux/Mac
• Integer overflow protection: Added safety checks for large dataset allocations

⚡ Performance Improvements

• OpenMP parallelization: Added parallel processing for HNSW point addition (>5000 points)
• Improved K-means convergence: Enhanced convergence detection and empty cluster handling
• L2 normalization fix: Corrected cosine metric normalization for HNSW consistency

🛡️ Enhanced Data Validation

• Metric validation warnings: Smart detection of inappropriate data for Hamming (non-binary) and Correlation (constant) metrics
• HNSW reconstruction warnings: Users now get proper notifications when models are loaded from lossy quantized data

🧹 Code Quality Improvements

• Zero compiler warnings: All unused variables, type conversions, and format issues fixed
• Dead code removal: Eliminated unused functions and cleaned up codebase
• Enhanced error messages: More descriptive error reporting throughout

🔧 Technical Enhancements

• Binary version checking: Automatic validation prevents DLL/library version mismatches
• Robust memory management: Improved bounds checking and safe copying operations
• Enhanced test coverage: Comprehensive validation of all error fixes

Quick Start

Using Prebuilt Enhanced Binaries (Recommended)

The fastest way to get started with all enhanced features:

🚀 Latest Release: v3.16.0 - Critical Euclidean Distance Fix

What's New in v3.16.0

• 🔧 Critical Euclidean Distance Fix: L2Space squared distance now properly converted with sqrt() for exact match detection
• ✅ Perfect Pipeline Consistency: Training embeddings match transform results exactly (MSE = 0)
• 🧪 All Tests Passing: 15/15 C# tests passing (fixed previously failing pipeline tests)
• 🎯 Production Reliability: Proper exact coordinate preservation for identical training points
• 📐 High-Precision Applications: Corrected distance comparisons for validation workflows

Previous v3.15.0 Features:

• 🌊 Stream-based HNSW serialization: Zero temporary files with direct memory-to-file operations
• 🔒 CRC32 data integrity: Automatic corruption detection for both original and embedding space HNSW indices
• ⚡ Deployment-grade reliability: Production-ready model persistence with automatic validation

# Install via NuGet
dotnet add package UMAPuwotSharp --version 3.16.0

# Or clone and build the enhanced C# wrapper
git clone https://github.com/78Spinoza/UMAP.git
cd UMAP/UMAPuwotSharp
dotnet build
dotnet run --project UMAPuwotSharp.Example

Complete Enhanced API Example

using UMAPuwotSharp;

Console.WriteLine("=== Enhanced UMAP Demo ===");

// Generate sample data
var data = GenerateTestData(1000, 100);

using var model = new UMapModel();

// Train with progress reporting and custom settings
Console.WriteLine("Training 27D embedding with Cosine metric...");

var embedding = model.FitWithProgress(
    data: data,
    progressCallback: (epoch, totalEpochs, percent) =>
    {
        if (epoch % 25 == 0)
            Console.WriteLine($"  Progress: {percent:F0}% (Epoch {epoch}/{totalEpochs})");
    },
    embeddingDimension: 27,           // High-dimensional embedding
    nNeighbors: 20,
    minDist: 0.05f,
    nEpochs: 300,
    metric: DistanceMetric.Cosine     // Optimal for high-dim sparse data
);

// Display comprehensive model information
var info = model.ModelInfo;
Console.WriteLine($"\nModel Info: {info}");
Console.WriteLine($"  Training samples: {info.TrainingSamples}");
Console.WriteLine($"  Input → Output: {info.InputDimension}D → {info.OutputDimension}D");
Console.WriteLine($"  Distance metric: {info.MetricName}");
Console.WriteLine($"  Neighbors: {info.Neighbors}, Min distance: {info.MinimumDistance}");

// Save enhanced model with HNSW optimization
model.Save("enhanced_model.umap");
Console.WriteLine("Model saved with all enhanced features!");

// Load and transform new data with safety analysis
using var loadedModel = UMapModel.Load("enhanced_model.umap");
var newData = GenerateTestData(100, 100);

// Standard fast transform
var transformedData = loadedModel.Transform(newData);
Console.WriteLine($"Transformed {newData.GetLength(0)} new samples to {transformedData.GetLength(1)}D");

// Enhanced transform with safety analysis
var safetyResults = loadedModel.TransformWithSafety(newData);
var safeCount = safetyResults.Count(r => r.IsProductionReady);
Console.WriteLine($"Safety analysis: {safeCount}/{safetyResults.Length} samples production-ready");

Building Real umappp Implementation from Source

🎯 PRODUCTION BUILD: Use the real umappp implementation (CURRENT):

Real umappp build (all advanced features):

cd uwot_umappp_wrapper
BuildWindowsOnly.bat              # Windows build
# OR
BuildDockerLinuxWindows.bat       # Cross-platform build

This builds the production-ready umappp implementation with:

• ✅ Spectral initialization: Manifold-aware starting positions (solves mammoth fragmentation!)
• ✅ AutoHNSW optimization: 50-2000x faster with automatic parameter tuning
• ✅ Dual HNSW architecture: Original + embedding space indices for AI inference
• ✅ TransformWithSafety: 5-level outlier detection with confidence scoring
• ✅ Stream-based serialization: CRC32 validation, zero temporary files
• ✅ Complete umappp features: All algorithms from libscran/umappp reference

Legacy build (DEPRECATED - DO NOT USE):

cd uwot_pure_cpp
BuildDockerLinuxWindows.bat       # Has compilation errors and build failures

❌ CRITICAL: The legacy implementation has build failures, fragmentation issues, and lacks modern optimizations. Use the real umappp implementation for all production applications.

Performance and Compatibility

• HNSW optimization: 50-2000x faster transforms with 80-85% memory reduction
• Enhanced algorithms: All new features optimized for performance
• Cross-platform: Windows and Linux support with automatic runtime detection
• Memory efficient: Careful resource management even with high-dimensional embeddings
• Production tested: Comprehensive test suite validating all enhanced functionality including safety features
• 64-bit optimized: Native libraries compiled for x64 architecture with enhanced feature support
• Backward compatible: Models saved with basic features can be loaded by enhanced version

Enhanced Technical Implementation

This implementation extends the core C++ algorithms from uwot with:

• HNSW integration: hnswlib for fast approximate nearest neighbor search
• Safety analysis engine: Real-time outlier detection and confidence scoring
• Multi-metric distance computation: Optimized implementations for all five distance metrics
• Arbitrary dimension support: Memory-efficient handling of 1D-50D embeddings
• Progress callback infrastructure: Thread-safe progress reporting from C++ to C#
• Enhanced binary model format: Extended serialization supporting HNSW indices and safety features
• Cross-platform enhanced build system: CMake with Docker support ensuring feature parity

🚀 NEW: HNSW Optimization & Production Safety Update

Major Performance & Safety Upgrade! This implementation now includes:

• ⚡ 50-2000x faster transforms with HNSW (Hierarchical Navigable Small World) optimization
• 🛡️ Production safety features - Know if new data is similar to your AI training set
• 📊 Real-time outlier detection with 5-level severity classification
• 🎯 AI model validation - Detect if inference data is "No Man's Land"
• 💾 80% memory reduction for large-scale deployments
• 🔍 Distance-based ML - Use nearest neighbors for classification/regression

Why This Matters for AI/ML Development

Traditional Problem: You train your AI model, but you never know if new inference data is similar to what the model was trained on. This leads to unreliable predictions on out-of-distribution data.

Our Solution: Use UMAP with safety features to validate whether new data points are within the training distribution:

// 1. Train UMAP on your AI training data
var trainData = LoadAITrainingData();  // Your original high-dim data
using var umapModel = new UMapModel();
var embeddings = umapModel.Fit(trainData, embeddingDimension: 10);

// 2. Train your AI model using UMAP embeddings (often better performance)
var aiModel = TrainAIModel(embeddings, labels);

// 3. In production: Validate new inference data
var results = umapModel.TransformWithSafety(newInferenceData);
foreach (var result in results) {
    if (result.Severity == OutlierLevel.NoMansLand) {
        Console.WriteLine("⚠️  This sample is completely outside training distribution!");
        Console.WriteLine("   AI predictions may be unreliable.");
    } else if (result.ConfidenceScore > 0.8) {
        Console.WriteLine("✅ High confidence - similar to training data");
    }
}

Use Cases:

• Medical AI: Detect if a new patient's data differs significantly from training cohort
• Financial Models: Identify when market conditions are unlike historical training data
• Computer Vision: Validate if new images are similar to training dataset
• NLP: Detect out-of-domain text that may produce unreliable predictions
• Quality Control: Monitor production data drift over time

🛡️ Production Safety Features

Get comprehensive quality analysis for every data point:

var results = model.TransformWithSafety(newData);
foreach (var result in results) {
    Console.WriteLine($"Confidence: {result.ConfidenceScore:F3}");     // 0.0-1.0
    Console.WriteLine($"Severity: {result.Severity}");                 // 5-level classification
    Console.WriteLine($"Quality: {result.QualityAssessment}");         // Human-readable
    Console.WriteLine($"Production Ready: {result.IsProductionReady}"); // Boolean safety flag
}

Safety Levels:

• Normal: Similar to training data (≤95th percentile)
• Unusual: Noteworthy but acceptable (95-99th percentile)
• Mild Outlier: Moderate deviation (99th percentile to 2.5σ)
• Extreme Outlier: Significant deviation (2.5σ to 4σ)
• No Man's Land: Completely outside training distribution (>4σ)

Distance-Based Classification/Regression

Use nearest neighbor information for additional ML tasks:

var detailedResults = umapModel.TransformDetailed(newData);
foreach (var result in detailedResults) {
    // Get indices of k-nearest training samples
    var nearestIndices = result.NearestNeighborIndices;

    // Use separately saved labels for classification
    var nearestLabels = GetLabelsForIndices(nearestIndices);
    var predictedClass = nearestLabels.GroupBy(x => x).OrderByDescending(g => g.Count()).First().Key;

    // Or weighted regression based on distances
    var nearestValues = GetValuesForIndices(nearestIndices);
    var weights = result.NearestNeighborDistances.Select(d => 1.0f / (d + 1e-8f));
    var predictedValue = WeightedAverage(nearestValues, weights);

    Console.WriteLine($"Prediction: {predictedClass} (confidence: {result.ConfidenceScore:F3})");
}

Performance Benchmarks (with HNSW Optimization)

Transform Performance (HNSW Optimized):

• Standard transform: 1-3ms per sample
• Enhanced transform (with safety): 3-5ms per sample
• Batch processing: Near-linear scaling
• Memory: 80-85% reduction vs traditional implementations

Comparison vs Other Libraries:

• Training Speed: 50-2000x faster than brute force methods
• Transform Speed: ❤️ms per sample vs 50-200ms without HNSW
• Memory Usage: 80-85% reduction (15-45MB vs 240MB for large datasets)
• Accuracy: Identical to reference uwot implementation (MSE < 0.01)
• Features: Only C# implementation with HNSW optimization and comprehensive safety analysis

📊 Performance Benchmarks

Training Performance (HNSW vs Exact)

Real-world benchmarks on structured datasets with 3-5 clusters:

Samples × Features	Exact k-NN	HNSW k-NN	Speedup	Memory Reduction
500 × 25	1.2s	0.6s	2.0x	65%
1,000 × 50	4.8s	0.9s	5.3x	72%
5,000 × 100	2.1 min	3.2s	39x	78%
10,000 × 200	12 min	8.1s	89x	82%
20,000 × 300	58 min	18s	193x	85%
50,000 × 500	6+ hours	95s	230x	87%

Transform Performance

Single sample transform times (after training):

Dataset Size	Without HNSW	With HNSW	Improvement
1,000	15ms	2.1ms	7.1x
5,000	89ms	2.3ms	38x
20,000	178ms	2.8ms	64x
100,000	890ms	3.1ms	287x

Multi-Metric Performance

HNSW acceleration works with multiple distance metrics:

Metric	HNSW Support	Typical Speedup	Best Use Case
Euclidean	✅ Full	50-200x	General-purpose data
Cosine	✅ Full	30-150x	High-dimensional sparse data
Manhattan	✅ Full	40-180x	Outlier-robust applications
Correlation	⚡ Fallback	1x (exact)	Time series, correlated features
Hamming	⚡ Fallback	1x (exact)	Binary, categorical data

System Requirements

• Minimum: 4GB RAM, dual-core CPU
• Recommended: 8GB+ RAM, quad-core+ CPU with OpenMP
• Optimal: 16GB+ RAM, multi-core CPU with AVX support

Benchmarks performed on Intel i7-10700K (8 cores) with 32GB RAM, Windows 11

Version Information

• Enhanced Native Libraries: Based on uwot algorithms with revolutionary HNSW optimization
• C# Wrapper: Version 3.3.0+ (UMAPuwotSharp with HNSW optimization)
• Target Framework: .NET 8.0
• Supported Platforms: Windows x64, Linux x64 (both with HNSW optimization)
• Key Features: HNSW k-NN optimization, Production safety, Multi-dimensional (1D-50D), Multi-metric, Enhanced progress reporting, OpenMP parallelization

Version History

Version	Release Date	Key Features	Performance
3.40.0	2025-10-31	InitializationMethod enum (clear API), Spectral default (best quality out-of-the-box), API clarity (replaced boolean with enum), Backward compatibility (obsolete property), Dynamic metadata (model-extracted params), Clean compilation (zero warnings)	Initialization API enhancement
3.39.0	2025-10-27	AlwaysUseSpectral property (force spectral init), Eigen compilation fix (MSVC + performance), LocalConnectivity & Bandwidth integration, Bandwidth sweep testing, Enhanced visualization titles, Spectral quality boost	Advanced algorithms revolution
3.37.0	2025-10-26	OpenMP parallelization (4-5x transform speedup), Single-point optimization (12-15x speedup), Stringstream persistence (no temp files), Windows DLL stability, Thread-safe operations	Major performance revolution
3.33.1	2025-10-25	Dual-mode exact k-NN integration, CPU core reporting, Complete parameter propagation, umappp with knncolle, Enhanced progress monitoring	Production-grade dual-mode accuracy
3.16.0	2025-10-02	Critical Euclidean distance fix, Perfect pipeline consistency (MSE=0), All 15/15 tests passing, Exact coordinate preservation	Production reliability fix
3.15.0	2025-02-02	Stream-based HNSW serialization, CRC32 data integrity validation, Zero temporary files, Enhanced test thresholds	Deployment-grade reliability
3.14.0	2025-02-01	Dual HNSW architecture, AI pattern similarity search, Embedding space inference, 5-level outlier detection	Revolutionary AI capabilities
3.3.0	2025-01-22	Enhanced HNSW optimization, Improved memory efficiency, Better progress reporting, Cross-platform stability	Refined HNSW performance
3.1.2	2025-01-15	Smart spread parameter implementation, Dimension-aware defaults, Enhanced progress reporting	Optimal embedding quality across dimensions
3.1.0	2025-01-15	Revolutionary HNSW optimization, Enhanced API with forceExactKnn parameter, Multi-core OpenMP acceleration	50-2000x speedup, 80-85% memory reduction
3.0.1	2025-01-10	Critical cross-platform fix, Linux HNSW library (174KB), Enhanced build system	Full cross-platform HNSW parity
3.0.0	2025-01-08	First HNSW implementation, Production safety features, 5-level outlier detection	50-200x speedup (Windows only)
2.x	2024-12-XX	Standard UMAP implementation, Multi-dimensional support (1D-50D), Multi-metric, Progress reporting	Traditional O(n²) performance

Upgrade Path

// v2.x code (still supported)
var embedding = model.Fit(data, embeddingDimension: 2);

// v3.33.1 optimized code - Dual-mode exact k-NN + CPU core reporting
var fastEmbedding = model.Fit(data,
    embeddingDimension: 2,
    forceExactKnn: false,  // HNSW mode (default) - 50-2000x faster
    useQuantization: true, // Enable 85-95% compression
    progressCallback: progress => Console.WriteLine($"Progress: {progress.PercentComplete:F1}% - {progress.Message}")
    // Shows: "Using 16 CPU cores for parallel processing"
);

// Exact k-NN mode for research/validation
var exactEmbedding = model.Fit(data,
    embeddingDimension: 2,
    forceExactKnn: true,   // Exact k-NN mode - perfect accuracy
    progressCallback: progress => Console.WriteLine($"Progress: {progress.PercentComplete:F1}% - {progress.Message}")
    // Shows: "force_exact_knn = true - using exact k-NN computation"
);

// Model persistence includes automatic CRC32 validation
model.SaveModel("model.umap");  // Stream-based serialization with integrity checks
var loadedModel = UMapModel.LoadModel("model.umap");  // Automatic corruption detection

Recommendation: Upgrade to v3.40.0 for the clearer InitializationMethod API with spectral default, ensuring best quality embeddings out-of-the-box with explicit control options.

References

1. McInnes, L., Healy, J., & Melville, J. (2018). UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction. arXiv:1802.03426.
2. Malkov, Yu A., and D. A. Yashunin. "Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs." arXiv:1603.09320 (2018).
3. Interactive UMAP Guide: https://pair-code.github.io/understanding-umap/
4. uwot R package: https://github.com/jlmelville/uwot
5. hnswlib library: https://github.com/nmslib/hnswlib
6. Original Python UMAP: https://github.com/lmcinnes/umap

License

Maintains compatibility with the GPL-3 license of the original uwot package and Apache 2.0 license of hnswlib.

---

This enhanced implementation represents the most complete and feature-rich UMAP library available for C#/.NET, providing capabilities that surpass even many Python implementations. The combination of HNSW optimization, production safety features, arbitrary embedding dimensions, multiple distance metrics, progress reporting, and complete model persistence makes it ideal for both research and production machine learning applications.