ByTech.LsmTree 1.1.0

ByTech.LsmTree

A disk-based, persistent Log-Structured Merge-tree for .NET 10 — a write-optimized sorted key-value store with WAL-based durability, leveled compaction, bloom-filter accelerated point lookups, and concurrent reads/writes.

v1.1.0 — see the CHANGELOG. Originally exported one-time from the ByTech.Bedrock durable-primitives framework.

Features

• Sorted key-value store — point lookups, range scans, prefix scans, batch mutations
• Write-optimized — in-memory memtable, write-ahead log (WAL), leveled disk segments, background flush and compaction
• WAL durability — write-ahead log with replay-on-open crash recovery
• Configurable durability — MemoryOnly / WriteBuffered / PeriodicSync / FlushOnCommit / ImmediateSync
• Bloom filters — accelerated negative point lookups, configurable false-positive rate
• Block cache — hot-block caching for read-mostly workloads
• Online compaction — single-flight tiered/leveled strategy
• Generic keys and values — bring your own ICodec<T>; built-in Utf8StringCodec, JsonCodec<T>, BytePassthroughCodec, plus a CodecPipeline<T> for layering ICodecMiddleware (compression, encryption, …)

Installation

dotnet add package ByTech.LsmTree

Quick start

using ByTech.LsmTree;

var options = new LsmTreeOptions
{
    BaseDirectory   = "./data/mystore",
    DurabilityLevel = DurabilityLevel.PeriodicSync,
};

await using var tree = new LsmTree<string, string?>(
    name:           "mystore",
    keyCodec:       new Utf8StringCodec(),
    valueCodec:     new Utf8StringCodec()!,
    options:        options,
    tombstoneValue: null,
    isTombstone:    v => v is null);

await tree.PutAsync("hello", "world");
var value = await tree.GetAsync("hello");          // "world"

await tree.DeleteAsync("hello");
value = await tree.GetAsync("hello");              // null

Range and prefix scans

// Range scan, inclusive bounds
await foreach (var kv in tree.RangeScanAsync("sku-00100", "sku-00200"))
    Console.WriteLine($"{kv.Key} -> {kv.Value}");

// Prefix scan (requires a prefixMatcher at construction)
await using var tree = new LsmTree<string, string?>(
    "skus", new Utf8StringCodec(), new Utf8StringCodec()!, options,
    tombstoneValue: null, isTombstone: v => v is null,
    prefixMatcher: (k, p) => k.StartsWith(p, StringComparison.Ordinal));

await foreach (var kv in tree.PrefixScanAsync("sku-09"))
    Console.WriteLine($"{kv.Key} -> {kv.Value}");

Batch mutations

await tree.BatchMutateAsync(
[
    new(MutationType.Put,    "sku-01", "alpha"),
    new(MutationType.Put,    "sku-02", "beta"),
    new(MutationType.Delete, "sku-03", null),
]);

Maintenance & introspection

await tree.CheckpointAsync();                 // flush memtable to disk
await tree.CompactAsync();                    // single-flight compaction pass

var integrity = await tree.ValidateIntegrityAsync();
Console.WriteLine($"CRC failures: {integrity.ChecksumFailures}");

var metrics = tree.GetMetrics();
Console.WriteLine($"ops={metrics.TotalOperations}  items={metrics.ItemCount}  compactions={metrics.CompactionCount}");
Console.WriteLine($"block cache: {metrics.BlockCacheHits} hits / {metrics.BlockCacheMisses} misses");
Console.WriteLine($"bloom: {metrics.BloomFilterNegatives}/{metrics.BloomFilterQueries} probes skipped a segment");

Diagnostics

The tree emits System.Diagnostics.ActivitySource spans for read, write-throttle, and compaction events. Attach a listener filtered to source ByTech.LsmTree:

using System.Diagnostics;
using ByTech.LsmTree.Diagnostics;

ActivitySource.AddActivityListener(new ActivityListener
{
    ShouldListenTo = src => src.Name == LsmTelemetry.SourceName,
    Sample = (ref ActivityCreationOptions<ActivityContext> _) => ActivitySamplingResult.AllData,
    ActivityStarted = a => Console.WriteLine($"start {a.OperationName}"),
});

Available span names are listed as constants on LsmTelemetry (ReadGet, ReadRwLockWait, ReadPathWalk, CompactionSwap, ThrottleWait).

Options reference

Dependencies

• ByTech.BloomFilter — bloom-filter primitive used by disk-segment readers
• System.IO.Hashing — checksums on disk segments and WAL records

License

MIT — see LICENSE.MD.