SquidStd.Persistence 0.40.0

<h1 align="center">SquidStd.Persistence</h1>

Embeddable in-memory entity store with durable binary snapshot + journal (write-ahead log) persistence. Full state lives in memory (synchronous reads), every mutation is appended to a length+checksum-framed binary journal, and a periodic snapshot captures all state and trims the journal. On startup the engine loads the snapshot and replays the journal tail. The engine is serializer-agnostic (via SquidStd's IDataSerializer/IDataDeserializer) and has no MessagePack or domain dependency.

Install

dotnet add package SquidStd.Persistence
dotnet add package SquidStd.Persistence.MessagePack   # recommended binary serializer

Usage (standalone, no bootstrap)

Wire the stack by hand when you are not using SquidStdBootstrap - construct the registry, journal, snapshot service, and PersistenceService yourself:

using SquidStd.Persistence.Abstractions.Data;
using SquidStd.Persistence.Data;
using SquidStd.Persistence.MessagePack;
using SquidStd.Persistence.Services;

public sealed class Player
{
    public int Id { get; set; }
    public string Name { get; set; } = string.Empty;
}

var serializer = new MessagePackDataSerializer();

var registry = new PersistenceEntityRegistry();
registry.Register(new PersistenceEntityDescriptor<Player, int>(
    serializer, serializer, typeId: 1, typeName: "Player", schemaVersion: 1, keySelector: p => p.Id));

var config = new PersistenceConfig { SaveDirectory = "./save" };
var journal = new BinaryJournalService(Path.Combine(config.SaveDirectory, config.JournalFileName));
var snapshot = new SnapshotService(config.SaveDirectory, config.SnapshotFileSuffix);

var persistence = new PersistenceService(registry, journal, snapshot, config);
await persistence.InitializeAsync();                       // load snapshot + replay journal

var players = persistence.GetStore<Player, int>();
await players.UpsertAsync(new Player { Id = 1, Name = "Bob" });   // appended to the journal
await persistence.SaveSnapshotAsync();                            // snapshot + trim journal

var bob = await players.GetByIdAsync(1);                          // detached clone

Manual DI registration (without RegisterPersistence)

container.RegisterPersistedEntity<Player, int>(typeName: "Player", schemaVersion: 1, p => p.Id);
container.ApplyPersistedEntityRegistrations();   // builds descriptors into IPersistenceEntityRegistry

Only needed when the rest of the stack (registry, journal, snapshot, lifecycle service) is assembled by hand instead of through RegisterPersistence(), which applies these registrations itself - do not call ApplyPersistedEntityRegistrations() when RegisterPersistence() is in use.

Type ids

Every persisted entity has a ushort type id. It is written into each journal record and into the snapshot file name, so it must be stable for the life of a save.

You do not pick it. It is derived from the store name you pass to RegisterPersistedEntity:

c.RegisterPersistedEntity<Player, int>("Player", 1, p => p.Id);   // id derived from "Player"

The store name is the right source because it is already permanent - it is part of the snapshot file name, so renaming it already loses data. Deriving from it means nothing has to know which ids are taken, which is the only way an author of a plugin or a downstream library can register an entity safely. It also survives class and namespace renames.

Because the id is 16 bits, two store names can collide. That is detected at startup, never in production, and the message names both stores:

Store 'news' and store 'notes' both use type id 41207. Rename one store, or pin one with the
explicit-id overload of RegisterPersistedEntity.

The explicit-id overload remains for exactly that case:

c.RegisterPersistedEntity<Player, int>(typeId: 4200, typeName: "Player", schemaVersion: 1, p => p.Id);

Entities that already have saved data

An entity whose id used to be picked by hand declares where it came from:

c.RegisterPersistedEntity<Player, int>("Player", 1, p => p.Id, legacyTypeId: 1);

On the next start its snapshot is renamed to the derived id and its journal entries are translated. The declaration is idempotent - once migrated it does nothing - and it is worth leaving in place, because an operator may upgrade from an old build at any time.

Unknown ids in the journal

A journal entry whose type id matches no registration and no legacyTypeId fails startup. Such an entry is a write that would otherwise be discarded in silence, and the usual cause is an entity that was renamed or removed. Set SkipUnknownJournalEntries in PersistenceConfig to discard them deliberately.

Bootstrap registration

The one-call path for SquidStdBootstrap apps: register a serializer, register the persistence stack, then declare the persisted entities.

using SquidStd.Persistence.Abstractions.Interfaces.Persistence;
using SquidStd.Persistence.Extensions;
using SquidStd.Persistence.MessagePack.Extensions;

bootstrap.ConfigureServices(c =>
{
    c.RegisterMessagePackSerializer();   // or RegisterDataSerializer() for JSON
    c.RegisterPersistence();             // or RegisterPersistence(new PersistenceConfig { ... })
    c.RegisterPersistedEntity<Player, int>("Player", 1, p => p.Id);
    return c;
});

// after StartAsync: snapshot loaded, journal replayed, autosave running
var players = bootstrap.Resolve<IPersistenceService>().GetStore<Player, int>();

• Serializer prerequisite: register a serializer (RegisterMessagePackSerializer(), RegisterDataSerializer() for JSON, or RegisterYamlDataSerializer() for human-readable saves) before RegisterPersistence() - it throws InvalidOperationException otherwise, so a missing serializer fails fast instead of at first use.
• Config source: RegisterPersistence() binds the persistence YAML section by default; pass an explicit PersistenceConfig instance to skip the file entirely for that section (it is then ignored). Either way, SaveDirectory defaults to the managed save directory under the bootstrap root when left blank.
• Lifecycle: IPersistenceService is registered as a lifecycle service - the snapshot loads and the journal replays at start, autosave runs while the bootstrap is up, and a final snapshot is written at stop.

Seeding a fresh store

Seeders populate initial data into a brand-new save (one that has no snapshot and no journal). They run after snapshot load and journal replay, in registration order, and their writes go through the normal entity stores - so subsequent boots are no longer fresh and seeders never run again. If a seeder exception occurs, startup fails immediately (fail-fast). A seeder that performs no writes leaves the save fresh, so it runs again at every boot.

Delegate seeder

Register an inline seeding callback:

bootstrap.ConfigureServices(c =>
{
    c.RegisterPersistence();
    c.RegisterPersistedEntity<Player, int>(1, "Player", 1, p => p.Id);
    
    // Inline delegate seeder
    c.RegisterPersistenceSeeder(async (persistence, ct) =>
    {
        var players = persistence.GetStore<Player, int>();
        await players.UpsertAsync(new Player { Id = 1, Name = "Admin" }, ct);
    });
    
    return c;
});

Class seeder

Implement IPersistenceSeeder and register it by type:

public sealed class AdminPlayerSeeder : IPersistenceSeeder
{
    public async ValueTask SeedAsync(IPersistenceService persistence, CancellationToken cancellationToken = default)
    {
        var players = persistence.GetStore<Player, int>();
        await players.UpsertAsync(new Player { Id = 1, Name = "Admin" }, cancellationToken);
    }
}

bootstrap.ConfigureServices(c =>
{
    c.RegisterPersistence();
    c.RegisterPersistedEntity<Player, int>(1, "Player", 1, p => p.Id);
    c.RegisterPersistenceSeeder<AdminPlayerSeeder>();
    
    return c;
});

Key semantics

• Fresh-save detection: Seeders run only when the save is brand-new (neither snapshot nor journal existed before). An emptied-but-old save (entities removed through the normal store API) is not fresh at the immediately following boot - the journal still records the removals. Once a snapshot captures the fully-emptied state (autosave or clean stop), the save becomes indistinguishable from a brand-new one and seeders run again at the next boot. Deleting the save files from disk also makes the next boot fresh.
• No re-runs: Since writes go through the normal stores, subsequent boots record the seeded state in the snapshot and journal. The save is no longer fresh.
• Constructor constraints: Class-form seeders must not constructor-inject IPersistenceService (it causes circular resolution). Receive the service as the SeedAsync parameter instead.
• Execution order: Seeders run in registration order. Multiple seeders can be registered via chained RegisterPersistenceSeeder() calls; plugins interleave naturally.
• Fail-fast behavior: If a seeder exception occurs, startup fails immediately and no remaining seeders run. If an earlier seeder's writes reach the journal before a later seeder fails, the save is no longer fresh at the next boot and the remaining seeders never run - prefer a single seeder, or make the set safe to lose a tail.

Key types

Durability

PersistenceConfig.DurabilityMode selects how writes reach disk. Buffered (default) flushes to the OS cache - fast, and safe across a process crash. Durable fsyncs each journal append and the snapshot temp file before its atomic rename, so committed data survives power loss. Pass it through when constructing the services: new BinaryJournalService(path, config.DurabilityMode) and new SnapshotService(dir, suffix, config.DurabilityMode). (.NET has no portable directory fsync, so the guarantee is per-file content durability plus atomic rename.)

Related

• Tutorial: Persistence

License

MIT - part of SquidStd.