Vorcyc.Quiver 3.2.1

Vorcyc Quiver 3.2.1

A pure .NET embedded vector database — zero native dependencies, runs in-process, no standalone database server deployment required.

📖 Github Repo (full documention)

Quiver draws on EF Core's DbContext design pattern, allowing developers to define entities and indexing strategies through declarative attributes such as [QuiverKey], [QuiverVector], and [QuiverIndex], with the framework automatically completing model discovery, index construction, and persistence management at runtime.

🐛 What's Fixed in 3.2.1

v3.2.1 is fully backward-compatible with all previous data files.

🆕 What's New in 3.2.0

v3.2.0 is fully backward-compatible with all previous data files.

Both methods are no-op in FullMemory mode. Vector index structures always remain in memory.

🆕 What's New in 3.1.0

v3.1.0 is fully backward-compatible with v1.x, v2.x, and v3.0.0 data files — no migration needed.

Breaking Changes

Migrating from v3.0.0: Remove VectorStorage = VectorStorageMode.MemoryMapped from your QuiverDbOptions if present. No other changes required.

What's New in 3.0.0

v3.0.0 is fully backward-compatible with v1.x and v2.x data files.

What's New in 2.0.0

v2.0.0 is fully backward-compatible with v1.x data files — no migration needed.

• Code-First Declarative Modeling — Annotate entity classes with attributes; the framework auto-discovers and registers QuiverSet<T> collections via reflection — zero configuration.
• Multiple ANN Index Algorithms — Built-in Flat (brute-force), HNSW, IVF, and KDTree indexes, covering small-scale exact search to million-scale approximate search.
• 9 Distance Metrics + Custom Similarity — Cosine, Euclidean, DotProduct, Manhattan, Chebyshev, Pearson, Hamming, Jaccard, Canberra. Or plug in your own ISimilarity<float>.
• Binary-First Persistence — Primary storage is always high-performance binary. JSON and XML are available as export/import side channels (ExportAsync / ImportAsync) for readable backups and interoperability. WAL incremental persistence reduces complexity from O(N) to O(Δ).
• Concurrency Safe — QuiverSet<T> uses ReaderWriterLockSlim internally; concurrent reads and writes are safe out-of-the-box.
• SIMD Accelerated — All similarity implementations use TensorPrimitives + Vector<float> SIMD, auto-adapting to SSE4/AVX2/AVX-512.
• Lazy-loading Page Cache — Optional EntityCache = EntityCacheMode.LazyPaging with LRU eviction and MaxCachedPages / PageSize controls. Entity objects load on demand; vector indexes remain resident for full search performance.
• CompactMemory — CompactMemory() / CompactMemoryAsync() on QuiverSet<T> and CompactAllMemoryAsync() on QuiverDbContext flush dirty pages and release all cached pages on demand.
• Schema Migration — Property rename and value transform via ConfigureMigration<T>(). Adding/removing fields requires no configuration.

Typical Use Cases: Semantic search · RAG · Face recognition · Image-to-image search · Recommendation systems · Multimodal retrieval

🚀 Quick Start

1. Define Entity

using Vorcyc.Quiver;

public class Document
{
    [QuiverKey]
    public string Id { get; set; } = string.Empty;

    public string Title { get; set; } = string.Empty;

    public string Category { get; set; } = string.Empty;

    [QuiverVector(384, DistanceMetric.Cosine)]
    public float[] Embedding { get; set; } = [];
}

2. Define Database Context

public class MyDocumentDb : QuiverDbContext
{
    public QuiverSet<Document> Documents { get; set; } = null!;

    public MyDocumentDb() : base(new QuiverDbOptions
    {
        DatabasePath = "documents.vdb",
        DefaultMetric = DistanceMetric.Cosine
    })
    { }
}

3. Use It

await using var db = new MyDocumentDb();
await db.LoadAsync();

// Add
db.Documents.Add(new Document
{
    Id = "doc-001",
    Title = "Introduction to Vector Databases",
    Category = "Tutorial",
    Embedding = new float[384] // embedding vector from your model
});

// Search Top-5
float[] queryVector = new float[384];
var results = db.Documents.Search(e => e.Embedding, queryVector, topK: 5);

foreach (var r in results)
    Console.WriteLine($"{r.Entity.Title} — {r.Similarity:F4}");

// Auto-saved on DisposeAsync

📐 Distance Metrics

Or define your own: [QuiverVector(128, CustomSimilarity = typeof(MyMetric))]

🗂️ Index Types

// HNSW example
[QuiverVector(768, DistanceMetric.Cosine)]
[QuiverIndex(VectorIndexType.HNSW, M = 32, EfConstruction = 300, EfSearch = 100)]
public float[] Embedding { get; set; } = [];

// IVF example
[QuiverVector(128, DistanceMetric.Cosine)]
[QuiverIndex(VectorIndexType.IVF, NumClusters = 100, NumProbes = 15)]
public float[] Feature { get; set; } = [];

🔧 CRUD Operations

// Add
db.Documents.Add(entity);
db.Documents.AddRange(batch);          // Atomic two-phase commit
await db.Documents.AddRangeAsync(batch);

// Upsert (insert or update, single write lock)
db.Documents.Upsert(entity);

// Remove
db.Documents.Remove(entity);           // By entity (matched by key)
db.Documents.RemoveByKey("doc-001");   // By key directly

// Find
Document? doc = db.Documents.Find("doc-001"); // O(1)

// Exists
bool exists = db.Documents.Exists("doc-001");              // By key, O(1)
bool hasTutorial = db.Documents.Exists(e => e.Category == "Tutorial"); // By predicate, O(n) short-circuit

// Clear
db.Documents.Clear();

// Count
int count = db.Documents.Count;

🔍 Vector Search

// Top-K
var results = db.Documents.Search(e => e.Embedding, queryVector, topK: 10);

// Threshold
var results = db.Documents.SearchByThreshold(e => e.Embedding, queryVector, threshold: 0.85f);

// Filtered
var results = db.Documents.Search(
    e => e.Embedding, queryVector, topK: 10,
    filter: e => e.Category == "Tutorial",
    overFetchMultiplier: 4);

// Top-1
var best = db.Documents.SearchTop1(e => e.Embedding, queryVector);

// Async (all methods have Async variants)
var results = await db.Documents.SearchAsync(e => e.Embedding, queryVector, topK: 10, ct);

// Default field shorthand (single vector field entities)
var results = db.Documents.Search(queryVector, topK: 5);

💾 Persistence

await db.SaveAsync();           // Full binary snapshot
await db.SaveChangesAsync();    // WAL incremental, O(Δ)
await db.CompactAsync();        // Full snapshot + clear WAL
await db.LoadAsync();           // Load snapshot + replay WAL

// Export / Import side channels
await db.ExportAsync("backup.json", ExportFormat.Json);
await db.ImportAsync("backup.json", ExportFormat.Json);

WAL Mode

var options = new QuiverDbOptions
{
    DatabasePath = "mydata.vdb",
    EnableWal = true,
    WalCompactionThreshold = 10_000,
    WalFlushToDisk = true
};

🔄 Schema Migration

When entity structures evolve, Quiver handles differences transparently during LoadAsync.

Automatic (zero config):

• New field added → gets CLR default value
• Old field removed → silently skipped

Property Renaming & Value Transform:

public class MyDb : QuiverDbContext
{
    public QuiverSet<Document> Documents { get; set; } = null!;

    public MyDb() : base(new QuiverDbOptions { DatabasePath = "my.db" })
    {
        ConfigureMigration<Document>(m => m
            .RenameProperty("OldTitle", "Title")
            .TransformValue("Score", v => v is int i ? (double)i : v));
    }
}

⚙️ Configuration

🔒 Concurrency

QuiverSet<T> uses ReaderWriterLockSlim:

• Read operations (Search / Find / Exists / Count) — shared lock, parallel execution ✅
• Write operations (Add / Upsert / Remove / Clear) — exclusive lock 🔒

No external locking needed.

♻️ Lifecycle

// Recommended: await using (auto-save on dispose)
await using var db = new MyDocumentDb();
await db.LoadAsync();
// ... use db ...
// Scope ends → DisposeAsync → auto-save → release resources