ChannelExtensions.Durability.S3 2026.6.23.2

ChannelExtensions.Durability.S3

A durable, drop-in System.Threading.Channels.Channel<T> backed by Amazon S3. You write through Writer and read through Reader exactly as with an in-memory channel; overflow is buffered in memory, uploaded to S3 as ordered chunks, and replayed back in order. Nothing is written to the local filesystem.

Part of the ChannelExtensions.Durability.* family. See also ChannelExtensions.Durability.FileSystem for the local-disk variant.

Install

dotnet add package ChannelExtensions.Durability.S3

Depends on AWSSDK.S3 — you supply a configured IAmazonS3 client (see Usage).

How it works

• Writes normally go straight to an in-memory bounded channel (direct mode).
• When the channel fills, it switches to spill mode: every write is buffered in memory instead. This preserves order — new items can never jump ahead of items already queued.
• A background writer batches buffered items into a newline-delimited JSON (.ndjson) chunk in memory. When the chunk reaches MaxChunkSize items or CommitInterval elapses, it is uploaded to S3 in a single PutObject.
• A background reader replays committed chunks oldest-first back into the channel, waiting for space so the consumer is never overwhelmed, then deletes each object once fully replayed.
• On startup the bucket is listed once under the prefix to discover pending chunk objects. From then on the set of pending object keys is tracked in memory (the writer appends a key after each upload; the reader consumes them). S3 is never polled to ask "is there more?" — the reader blocks on the in-memory key queue.
• Once the entire backlog is drained, the channel reverts to direct mode.

Guarantees

• Ordering — preserved across the spill boundary and across restarts. Chunk objects are keyed by a time-ordered v7 GUID, so an ordinal sort of the keys is chronological.
• Durability begins at the upload. Because there is no local staging, items that have been buffered but not yet uploaded live only in memory and are lost on a hard process kill. The upload to S3 is the durability boundary — once a chunk is PutObject-ed it survives a restart and is replayed.
• At-least-once replay. A chunk object is deleted only after it has been fully replayed. A crash mid-replay leaves the object in S3, so on restart it is rediscovered by the startup listing and replayed again — consumers may therefore see a chunk's records more than once across a crash. Make your consumer idempotent if that matters.

Usage

using System.Threading.Channels;
using Amazon.S3;
using ChannelExtensions.Durability.S3;
using ChannelExtensions.Durability.S3.S3BackedChannel;

IAmazonS3 s3 = new AmazonS3Client(); // your configured region/credentials

Channel<MyEvent> channel = Channel.CreateS3BackedChannel<MyEvent>(
    new S3BackedChannelOptions(capacity: 10_000, bucket: "my-bucket", client: s3)
    {
        Prefix = "events/durable-channel", // optional sub-key the chunks are stored under
        MaxChunkSize = 1_000,              // upload once this many overflow items have accumulated
    });

// Producer and consumer are identical to any Channel<T>.
await channel.Writer.WriteAsync(new MyEvent(...));
await foreach (var item in channel.Reader.ReadAllAsync())
    Handle(item);

Like the other durable channels, this is not an IHostedService — its background loops start in the constructor. The constructor also performs the one-time S3 listing synchronously, so it makes a blocking S3 call; construct it off the hot path. Logging is configured purely by the Logger option.

Dependency injection

Register the IAmazonS3 client and the channel as singletons. The factory runs eagerly, so use a factory delegate to inject the client and a configured ILogger:

builder.Services.AddSingleton<IAmazonS3>(_ => new AmazonS3Client());

builder.Services.AddSingleton<Channel<MyEvent>>(sp =>
    Channel.CreateS3BackedChannel<MyEvent>(
        new S3BackedChannelOptions(10_000, "my-bucket", sp.GetRequiredService<IAmazonS3>())
        {
            Prefix = "events/durable-channel",
            Logger = sp.GetRequiredService<ILoggerFactory>().CreateLogger("S3BackedChannel"),
        }));

MinIO / non-AWS endpoints: configure the client with ForcePathStyle = true, and because AWSSDK v4 enables request/response checksums by default (which MinIO rejects with an x-amz-content-sha256 mismatch) set RequestChecksumCalculation and ResponseChecksumValidation to WHEN_REQUIRED on the AmazonS3Config.

Options

S3BackedChannelOptions extends ChannelOptions.

On-S3 layout