MintPlayer.Spark.SubscriptionWorker 10.0.0-preview.77

MintPlayer.Spark.SubscriptionWorker

A RavenDB subscription worker framework with built-in retry logic, incremental backoff, categorized exception handling, and ASP.NET Core lifecycle management. Fully independent -- no dependency on the core Spark CRUD framework. Any project with a RavenDB IDocumentStore can use it.

Installation

dotnet add package MintPlayer.Spark.SubscriptionWorker

If you also use the Spark source generators (for auto-registration), ensure the MintPlayer.Spark.SourceGenerators package is referenced.

Overview

A subscription worker continuously listens for document changes in RavenDB via the Data Subscriptions mechanism. When documents match the subscription's RQL query, RavenDB delivers them in batches to the worker for processing.

SparkSubscriptionWorker<T> wraps this into an ASP.NET Core BackgroundService with:

• Automatic subscription creation/update on startup
• A connection loop that reconnects after errors or normal completion
• Categorized exception handling (retryable vs. fatal)
• Per-document retry tracking via RetryNumerator
• Lifecycle hooks for startup, shutdown, and batch completion

Quick Start

1. Create a Subscription Worker

Extend SparkSubscriptionWorker<T> and implement two abstract methods:

• ConfigureSubscription() -- returns the RQL query that filters which documents are delivered
• ProcessBatchAsync() -- handles each batch of documents

using MintPlayer.Spark.SubscriptionWorker;
using Raven.Client.Documents;
using Raven.Client.Documents.Subscriptions;

public partial class OrderProcessingWorker : SparkSubscriptionWorker<Order>
{
    private readonly RetryNumerator _retryNumerator = new();

    // No constructor: the base class takes its IDocumentStore and ILoggerFactory through
    // [Inject], and the source generator emits the constructor. Declare your own dependencies
    // the same way — which is why the class must be `partial`.
    [Inject] private readonly IOrderService _orders;

    protected override SubscriptionCreationOptions ConfigureSubscription()
        => new() { Query = "from Orders where Status = 'Pending'" };

    protected override async Task ProcessBatchAsync(
        SubscriptionBatch<Order> batch, CancellationToken cancellationToken)
    {
        using var session = batch.OpenAsyncSession();

        foreach (var item in batch.Items)
        {
            try
            {
                var order = item.Result;
                // Process the order...
                order.Status = "Processed";
                await _retryNumerator.ClearRetryAsync(session, order);
            }
            catch (Exception ex)
            {
                var retry = await _retryNumerator.TrackRetryAsync(
                    session, item.Result, ex, Logger);
                if (!retry.WillRetry)
                {
                    Logger.LogError(ex, "Permanently failed processing order {Id} after {Attempts} attempts",
                        item.Id, retry.AttemptCount);
                }
            }
        }

        await session.SaveChangesAsync(cancellationToken);
    }
}

Subscription Naming

By default, the subscription name in RavenDB is derived from the class name by stripping common suffixes:

• OrderProcessingWorker becomes "OrderProcessing"
• OrderProcessingSubscriptionWorker becomes "OrderProcessing"

Override SubscriptionName to set a custom name:

protected override string SubscriptionName => "MyCustomSubscription";

2. Register the Worker

Option A: Source-Generated Registration (Recommended)

If your project references MintPlayer.Spark.SourceGenerators, a source generator discovers all SparkSubscriptionWorker<T> subclasses in your project and generates an AddSubscriptionWorkers() extension on ISparkBuilder:

// Program.cs
builder.Services.AddSparkSubscriptions();
builder.Services.AddSpark(builder.Configuration, spark =>
{
    spark.AddSubscriptionWorkers();   // source-generated
});

The generated code calls AddSubscriptionWorker<T>() for each worker class found, registering each as a hosted service. It is emitted by SubscriptionWorkerRegistrationGenerator into your own root namespace, so it is only visible from your assembly.

Option B: Manual Registration

Register workers individually:

builder.Services.AddSparkSubscriptions();
builder.Services.AddSubscriptionWorker<OrderProcessingWorker>();

How It Works

Subscription Lifecycle

Each worker runs as a BackgroundService:

1. Startup: EnsureSubscriptionExistsAsync creates or updates the RavenDB subscription (idempotent -- if it already exists, the query is updated).
2. OnWorkerStartedAsync(): Lifecycle hook called after the subscription is ready, before the first batch.
3. Connection loop: Opens a subscription worker connection and starts receiving document batches.
4. Batch processing: Calls ProcessBatchAsync() for each batch, then OnBatchCompletedAsync(itemCount).
5. Error recovery: Catches and categorizes exceptions with automatic reconnection (see table below).
6. Shutdown: Triggered by CancellationToken cancellation (e.g., app shutdown). Calls OnWorkerStoppedAsync().

Categorized Exception Handling

The connection loop classifies exceptions into three categories.

Retryable Errors

These errors cause the worker to wait and then reconnect:

Non-Recoverable Errors

These errors cause the worker to stop permanently and call OnNonRecoverableErrorAsync():

Cancellation

OperationCanceledException when the CancellationToken is cancelled triggers a graceful shutdown.

Configuration

Override virtual properties on your worker class to tune behavior:

public class OrderProcessingWorker : SparkSubscriptionWorker<Order>
{
    // Subscription name in RavenDB (default: class name minus "Worker"/"SubscriptionWorker")
    protected override string SubscriptionName => "MyCustomSubscription";

    // Target database (default: null = store default)
    protected override string? Database => null;

    // Max documents per batch (default: 256)
    protected override int MaxDocsPerBatch => 100;

    // Whether to reconnect after normal subscription completion (default: true)
    protected override bool KeepRunning => true;

    // Wait time before connection retry (default: 30 seconds)
    protected override TimeSpan RetryDelay => TimeSpan.FromSeconds(30);

    // Max erroneous period before giving up on connection (default: 5 minutes)
    protected override TimeSpan MaxDownTime => TimeSpan.FromMinutes(5);
}

Global Options

AddSparkSubscriptions() accepts an optional configuration callback, but SparkSubscriptionOptions currently carries no options:

builder.Services.AddSparkSubscriptions();

Earlier versions documented WaitForNonStaleIndexes and NonStaleIndexTimeout here. Nothing ever read them — the workers never waited on indexes — so they were removed rather than left looking like a configured guarantee. If a worker depends on an index being current, wait for the specific query it needs rather than gating startup on every index in the database.

Lifecycle Hooks

Override these virtual methods to react to worker events:

// Called after startup, before the first batch
protected override Task OnWorkerStartedAsync() => Task.CompletedTask;

// Called when the worker stops (graceful or error)
protected override Task OnWorkerStoppedAsync() => Task.CompletedTask;

// Called after each batch is successfully processed
protected override Task OnBatchCompletedAsync(int itemCount) => Task.CompletedTask;

// Called when a non-recoverable error occurs (before stopping)
protected override Task OnNonRecoverableErrorAsync(Exception exception) => Task.CompletedTask;

RetryNumerator: Per-Document Retry Tracking

RetryNumerator tracks failed processing attempts for individual documents using RavenDB counters and the @refresh metadata mechanism.

How It Works

1. When TrackRetryAsync() is called for a failed document, it increments a RavenDB counter on the document.
2. It sets the @refresh metadata to a future timestamp, which causes RavenDB to redeliver the document to the subscription at that time.
3. If the maximum number of attempts is exhausted, the counter is cleared and the document is "parked" for a longer delay (default: 1 day).

Configuration

var retryNumerator = new RetryNumerator
{
    MaxAttempts = 5,                                // default: 5
    BaseDelay = TimeSpan.FromSeconds(30),           // default: 30s
    CounterName = "SparkRetryAttempts",             // default
    ExhaustedDelay = TimeSpan.FromDays(1),          // default: 1 day
};

Backoff Schedule

RetryNumerator uses linear incremental backoff (BaseDelay * attempt):

When an attempt fails, TrackRetryAsync increments the counter and sets @refresh metadata to schedule redelivery. After max attempts, the counter is deleted and the document is parked for ExhaustedDelay.

Usage in ProcessBatchAsync

protected override async Task ProcessBatchAsync(
    SubscriptionBatch<Order> batch, CancellationToken cancellationToken)
{
    using var session = batch.OpenAsyncSession();

    foreach (var item in batch.Items)
    {
        try
        {
            // Process the document...
            await _retryNumerator.ClearRetryAsync(session, item.Result);
        }
        catch (Exception ex)
        {
            var retry = await _retryNumerator.TrackRetryAsync(
                session, item.Result, ex, Logger);
            // retry.WillRetry is false when max attempts are exhausted;
            // retry.AttemptCount and retry.NextAttemptAtUtc describe this failure.
        }
    }

    await session.SaveChangesAsync(cancellationToken);
}

Call ClearRetryAsync() after successful processing to remove any leftover retry counters from previous failures.

Revision Subscriptions

For change detection (comparing previous vs. current document state), subscribe to Revision<T>:

public partial class CompanyChangeWorker : SparkSubscriptionWorker<Revision<Company>>
{
    protected override SubscriptionCreationOptions ConfigureSubscription()
        => new() { Query = "from Companies (Revisions = true)" };

    protected override async Task ProcessBatchAsync(
        SubscriptionBatch<Revision<Company>> batch, CancellationToken cancellationToken)
    {
        using var session = batch.OpenAsyncSession();
        foreach (var item in batch.Items)
        {
            var previous = item.Result.Previous;
            var current = item.Result.Current;
            // React to changes between previous and current...
        }
        await session.SaveChangesAsync(cancellationToken);
    }
}

This requires RavenDB document revisions to be enabled on the collection.

Real-World Example: Spark Messaging

The MintPlayer.Spark.Messaging package uses SparkSubscriptionWorker<T> internally. MessageFeeder subscribes to SparkMessage documents for every queue at once, claims each one, and routes it to an in-process per-queue lane:

internal sealed class MessageFeeder : SparkSubscriptionWorker<SparkMessage>
{
    // No trailing hyphen: the startup cleanup deletes "SparkMessaging-*", and this name
    // escapes that prefix by exactly one character.
    protected override string SubscriptionName => "SparkMessaging";
    protected override int MaxDocsPerBatch => 1;

    protected override SubscriptionCreationOptions ConfigureSubscription()
    {
        // No QueueName predicate — one subscription serves all queues, which also means no
        // queue name is interpolated into RQL.
        return new SubscriptionCreationOptions
        {
            Query = @"from SparkMessages
                      where (Status = 'Pending' and (NextAttemptAtUtc = null or WakeUp = true))
                         or (Status = 'Failed' and WakeUp = true)"
        };
    }

    protected override async Task ProcessBatchAsync(
        SubscriptionBatch<SparkMessage> batch, CancellationToken cancellationToken)
    {
        foreach (var item in batch.Items)
        {
            // Claim under optimistic concurrency and save BEFORE acknowledging the batch,
            // then hand the id to the queue's lane. No handler runs on this path.
        }
    }
}

Two things in that query are worth copying, because both were learned the hard way:

• WakeUp = true, never NextAttemptAtUtc <= now(). A subscription where-clause cannot evaluate time — a now() comparison silently never matches, so a message parked for a retry would never be redelivered. "The backoff has elapsed" must be materialized as plain field state by a component that can evaluate time; MessageRetrySweeper sets the boolean, and the patch that sets it is also what bumps the change vector to trigger re-evaluation.
• The claim is saved before the batch is acknowledged. Subscriptions re-deliver an unacknowledged batch on reconnect but never a acknowledged one, so a status written after the acknowledgement — with nothing that reads it — leaves a crashed host's message stranded for ever.

Extension Methods

Source-Generated

Source Generator Details

The SubscriptionWorkerRegistrationGenerator source generator scans your project for all non-abstract classes that inherit from SparkSubscriptionWorker<T> (at any depth in the inheritance chain). It generates a static extension method:

// Auto-generated: SparkSubscriptionWorkerRegistrations.g.cs
namespace YourProject
{
    internal static class SparkSubscriptionWorkersBuilderExtensions
    {
        internal static ISparkBuilder AddSubscriptionWorkers(this ISparkBuilder builder)
        {
            SparkSubscriptionExtensions.AddSubscriptionWorker<OrderProcessingWorker>(builder.Services);
            SparkSubscriptionExtensions.AddSubscriptionWorker<CompanyChangeWorker>(builder.Services);
            return builder;
        }
    }
}

This eliminates the need to manually register each worker in Program.cs.

Requirements

• .NET 10.0+
• RavenDB 6.2+
• An IDocumentStore registered in the DI container (provided by AddSpark() or registered manually)

Complete Example

See the following files for working implementations:

• SparkSubscriptionWorker.cs -- abstract base class with connection loop and error handling
• RetryNumerator.cs -- per-document retry tracking
• SparkSubscriptionExtensions.cs -- DI registration helpers
• ../MintPlayer.Spark.Messaging/Services/MessageSubscriptionWorker.cs -- real-world usage in the messaging package
• ../MintPlayer.Spark.SourceGenerators/Generators/SubscriptionWorkerRegistrationGenerator.cs -- source generator for auto-registration

License

MIT License