Sa.Outbox.PostgreSql
0.20.0
dotnet add package Sa.Outbox.PostgreSql --version 0.20.0
NuGet\Install-Package Sa.Outbox.PostgreSql -Version 0.20.0
<PackageReference Include="Sa.Outbox.PostgreSql" Version="0.20.0" />
<PackageVersion Include="Sa.Outbox.PostgreSql" Version="0.20.0" />
<PackageReference Include="Sa.Outbox.PostgreSql" />
paket add Sa.Outbox.PostgreSql --version 0.20.0
#r "nuget: Sa.Outbox.PostgreSql, 0.20.0"
#:package Sa.Outbox.PostgreSql@0.20.0
#addin nuget:?package=Sa.Outbox.PostgreSql&version=0.20.0
#tool nuget:?package=Sa.Outbox.PostgreSql&version=0.20.0
Sa.Outbox.PostgreSql
English | Русский
An AOT-compatible library for implementing the Transactional Outbox pattern using PostgreSQL in .NET applications. Provides guaranteed message delivery with support for multi-tenancy, concurrent processing, partitioning, and advanced consumer management.
Contents
- Installation
- What it actually looks like — the model compared with Redis Stream and Kafka
- Quick Start
- Publishing inside a Caller Transaction
- Configuration Guide
- Per-group settings from configuration
- Consumer group settings (
OutboxConsumerSettingsBuilder) - Latency budget for every group at once —
WithPollingBudget/WithPollingProfile - Runtime settings management
- Message Lifecycle
- Result Statuses
- Database Architecture
- Bulk Publishing at Scale
- Requirements
- License
Detailed configuration (tables, migrations, cleanup, PgPollingSettings) lives in Configuration/Readme.md.
The provider's internals each have their own readme: Commands/Readme.md (data-access
commands and batching), SqlBuilder/Readme.md (the SQL templates),
TypeResolve/Readme.md (the type ↔ hash cache).
What it actually looks like
By its runtime model this is closer to Redis Stream and Kafka than to the classic "outbox → broker":
| Redis Stream | Kafka | here | |
|---|---|---|---|
| Log | XADD into a stream |
append-only partition | __msg$ + msg_seq (BIGSERIAL) |
| Groups | XGROUP / XREADGROUP |
consumer group | consumer_group in the group settings |
| Cursor | last-delivered-id per group | __consumer_offsets |
__offset$.group_offset_seq per (group, tenant) |
| Partitioning | — | partition | (tenant, part) → daily leaves |
| Delivery | PEL + XACK, claim by timeout |
poll + commit offset | task_lock_expires_on + msg.Ok()/Warn()/… |
| Ordering | within a stream | within a partition | within (tenant, part) |
| Retention | XTRIM / MAXLEN |
time or size | DROP PARTITION per day |
What it is not: the classic "put it in the outbox, a separate relay publishes it to Kafka/RabbitMQ" pipeline. There is no broker, no CDC/Debezium and no relay process — the PostgreSQL tables themselves are the log and the queue, and the consumer reads straight from the database. If you need a bus, this schema can play that role: consumer groups give the same fan-out.
What a real broker has and this does not:
- no wire protocol or client libraries — the "broker" is a PostgreSQL schema;
- no dynamic rebalancing: groups are static (registered in DI), and parallelism inside a group comes
from
SKIP LOCKED, not from partition assignment; - no global ordering: it exists only within
(tenant, part); - no retention by offset: retention drops whole daily partitions;
- as a bus it works only inside one database — it is shared schema, not network transport.
Installation
dotnet add package Sa.Outbox.PostgreSql
Quick Start (5 minutes)
One complete example — copy, paste, run:
using Microsoft.Extensions.Hosting;
using Sa.Outbox;
using Sa.Outbox.Delivery;
using Sa.Outbox.PostgreSql;
using Sa.Outbox.Publication;
using Sa.Outbox.Serialization;
IHost host = Host.CreateDefaultBuilder()
.ConfigureServices(services => services
// ① Register outbox core + tenants + consumers
.AddSaOutbox(builder => builder
.WithTenants((_, t) => t.WithTenantIds(1, 2, 3))
.WithDeliveries(b => b
.AddDeliveryScoped<OrderConsumer, OrderCreated>((_, s) =>
{
s
.WithInterval(TimeSpan.FromSeconds(5))
.StartImmediately()
.WithMaxBatchSize(16)
.WithMaxDeliveryAttempts(3)
.WithLockDuration(TimeSpan.FromSeconds(10))
.WithLookbackInterval(TimeSpan.FromDays(7));
})
)
)
// ② Wire up PostgreSQL
.AddSaOutboxUsingPostgreSql(cfg => cfg
.WithDataSource(ds => ds
.Options(ob => ob.Configure(o => o.ConnectionString =
"Host=localhost;Database=outbox_db;Username=postgres;Password=postgres")))
.WithOutboxSettings((_, settings) =>
{
settings.TableSettings.WithSchema("outbox");
settings.CleanupSettings.WithRetentionPeriod(TimeSpan.FromDays(30));
})
.WithMessageSerializer(JsonOutboxMessageSerializer.Instance) // the core default; replace with yours
)
)
.Build();
// ③ Publish a message
var publisher = host.Services.GetRequiredService<IOutboxMessagePublisher>();
await publisher.Publish([
new OrderCreated("order-42", "Premium Widget"),
new OrderCreated("order-43", "Standard Gadget")
], tenantId: 1);
// ④ Run (consumers will pick up messages automatically)
await host.RunAsync();
// ─── Your types ────────────────────────────────────────────────
public sealed record OrderCreated(string PayloadId, string ProductName);
public sealed class OrderConsumer : IConsumer<OrderCreated>
{
public async ValueTask Consume(
OutboxConsumerSettings settings,
OutboxMessageFilter filter,
ReadOnlyMemory<IOutboxContextOperations<OrderCreated>> messages,
CancellationToken ct)
{
foreach (var msg in messages.Span)
{
Console.WriteLine($"[{filter.ConsumerGroupId}] {msg.Payload.ProductName}");
msg.Ok(); // ← mark success
}
}
}
That's it. The library handles:
- Table creation & partitioning
- BINARY COPY bulk insertion
- Concurrent consumption via
SKIP LOCKED - Retry / postpone / error workflows
- Automatic cleanup of old partitions
- Self-bootstrapping of consumer settings into
IOutboxConsumerManager
Publishing inside a Caller Transaction
Normally every Publish(...) writes on its own connection and commits immediately (auto-commit). When the outbox write must commit atomically with your own database changes (the classic outbox transactionality), publish inside an NpgsqlTransaction you control:
using Sa.Outbox.PostgreSql.Publication; // PgOutboxPublisherExtensions
using Sa.Outbox.Publication;
using Sa.Data.PostgreSql; // IPgDataSource / PgTransaction
// Open the transaction through the base PostgreSQL data source:
IPgDataSource dataSource = services.GetRequiredService<IPgDataSource>();
await using PgTransaction sqlTx = await dataSource.BeginTransaction(ct);
// Outbox rows go into the caller's transaction:
await publisher.PublishInTransaction(events, tenantId: 1, sqlTx.Transaction, ct);
// ... your own inserts on sqlTx.Connection / sqlTx.Transaction ...
await sqlTx.CommitAsync(ct); // outbox rows become visible atomically with your changes
// -- dispose without commit (or RollbackAsync) discards everything, outbox rows included
PgOutboxPublisherExtensions adds typed PublishInTransaction(...) / PublishSingleInTransaction(...) helpers that accept an NpgsqlTransaction (for a PgTransaction, pass sqlTx.Transaction; a wrapper overload would duplicate the whole matrix). The name differs from Publish because an instance method always shadows a same-named extension — the typed call should be explicit in the code. The binary COPY then runs on that transaction's connection, inside that transaction, and bypasses the automatic retry layer — retrying over a foreign or aborted transaction is invalid, so failures bubble to the caller.
Passing no context keeps the default auto-commit behavior. There is deliberately no ambient/implicit scope — the transaction is always an explicit argument.
Guards:
- The transaction must belong to the same database (host / port / database) as the outbox data source — a foreign transaction throws
InvalidOperationExceptionbefore anything is written. - The connection must be open and the transaction must be its own active transaction; otherwise
InvalidOperationException.
Partition bookkeeping (
EnsureMsgParts) always runs on the outbox's own connection, outside your transaction, so a rolled-back publish can leave an empty partition (or a type registry row) behind — harmless leftovers, cleaned up by the normal retention jobs.Locking note: while your transaction is open and has written to a partition, partition DDL from another publish on the same data source (e.g., a first message of a new tenant) can block until your transaction finishes. Commit (or roll back) before publishing more outside the transaction.
Configuration Guide
One registration call or two:
| Call | Purpose |
|---|---|
AddSaOutbox(...) |
Core: tenants, consumers, schedules |
AddSaOutboxUsingPostgreSql(cfg) |
PostgreSQL: connection, tables, migration, cleanup |
AddSaOutboxUsingPostgreSql(cfg, outbox) |
One call: core + provider, scopeKey written once |
One registration instead of two:
services.AddSaOutboxUsingPostgreSql(
pg => pg.WithDataSource(...).WithPollingBudget(TimeSpan.FromMinutes(1)),
outbox => outbox
.WithTenants((_, t) => t.WithTenantIds(1, 2, 3))
.WithDeliveries(d => d.AddDelivery<OrderConsumer, OrderCreated>("orders")));
Isolation. The Outbox registers its internal
Sa.Scheduleschedule and itsSa.Partitional.PostgreSqlmodule under the scope key"outbox"(Sa.Outbox.OutboxScope.DefaultKey), and never touches the application's default schedule or partitional module. Both entry points take an optional trailingscopeKeyparameter. In the one-call formAddSaOutboxUsingPostgreSql(cfg, outbox, scopeKey)it is written once; when the core is registered separately, pass the same value to both calls. The reserved"default"key is rejected.
Turning the provider off
WithEnabled(false) is a registration switch, not a runtime flag:
services.AddSaOutboxUsingPostgreSql(pg => pg
.WithEnabled(configuration.GetValue("Outbox:PostgreSql:Enabled", true))
.WithDataSource(...));
The call still runs the configuration callback, but nothing is applied to the container: no settings,
no keyed data source (so ValidateOnStart cannot fail on an absent connection string), no serializer
or poison override, no consumer groups and no Sa.Schedule jobs. Enabled is true by default; a
disabled provider leaves no trace in DI, so the registration can stay in code for every environment
and be switched off by configuration. Providers registered under other keys are unaffected. In the
split form (AddSaOutbox + this call) the core still registers the consumer groups, so a disabled
provider makes the startup scope-key check fail and names those groups — skip AddSaOutbox too, or
use the one-call form.
Registering the same key twice
A repeated AddSaOutboxUsingPostgreSql(..., scopeKey) for one key is tolerated: the provider
contracts, the startup validator, the migration guard, the consumer-group registrations and the
schedule deduplicate. The outbox-settings and polling callbacks of the last registration replace the
ones recorded before, so they run once — not once per registration. The first data-source
registration owns the connection — a second outbox call's data-source callback is ignored instead
of throwing, so register the connection once.
A data source another module registered under the same key is shared, not duplicated: a bare
WithDataSource() borrows it. WithDataSource(configure) is an explicit connection intent and is
applied to that key; if the other module already configured it, AddSaPostgreSqlDataSource reports
the duplicate instead of the outbox silently keeping the other connection.
1. Tenant Configuration
Where inside AddSaOutbox:
.AddSaOutbox(builder => builder
.WithTenants((_, ts) => ts
.WithTenantIds(1, 2, 3) // Fixed list
// .WithAutoDetect() // Discover from messages at runtime
// .WithTenantSource(myTenantSource) // Custom ITenantSource
// .WithTenantProvider(ct => ...) // Func<CancellationToken, ValueTask<int[]>>
)
// ...
)
The detector behind WithAutoDetect() is supplied by the provider package; to replace it, register
your own at the provider level (AddOutboxProvider(..., p => p.WithTenantDetector<MyDetector>())).
Per-tenant parallelism is a consumer-group setting: WithPerTenantMaxDegreeOfParallelism(4) on the
group's OutboxConsumerSettingsBuilder (see below).
2. Consumer Registration
Where inside AddSaOutbox:
.AddSaOutbox(builder => builder
.WithDeliveries(b => b
// Simplest form — auto-named consumer group from type name
.AddDeliveryScoped<GreetingConsumer, EmailSent>()
// Custom consumer group name (allows multiple consumers for one message type)
.AddDeliveryScoped<EmailAnalyticsConsumer, EmailSent>("analytics")
// With inline settings
.AddDeliveryScoped<OrderConsumer, OrderCreated>((_, settings) =>
{
settings
.WithInterval(TimeSpan.FromSeconds(5))
.StartImmediately() // start immediately, don't wait first interval
.WithMaxBatchSize(16) // max messages per batch
.WithMaxDeliveryAttempts(3) // stop retrying after N attempts
.WithLockDuration(TimeSpan.FromSeconds(10))
.WithLookbackInterval(TimeSpan.FromDays(7));
})
)
// ...
)
Per-group Settings from Configuration (configurationKey)
The explicit-consumerGroupId overloads of AddDelivery / AddDeliveryScoped take an optional configurationKey — the path of an IConfiguration section holding that group's settings (bound to PgOutboxDeliveryOptions; see Configuration/Readme.md):
.AddDeliveryScoped<OrderConsumer, OrderCreated>(
"cg_order_consumer",
(_, settings) => settings
.StartImmediately()
.WithMaxBatchSize(16),
configurationKey: "DeliveryGroups:cg_order_consumer")
The group's settings are composed in three layers, lowest → highest:
OutboxDefaults → configuration section → configure callback
- Which section — by convention
Outbox:DeliveryGroups:{groupId}; an explicitconfigurationKeywins over it, andconfigurationKey: ""opts the group out. The convention prefix is changed byIOutboxBuilder.WithDeliveryConfigurationSection(...);nulldisables it. - Key given — the section's non-null values land between the defaults and the
configurecallback; a partially-filled section overrides only what it names. - No key — the group reads the conventional section when it exists; sections under other prefixes are not read.
- An absent or empty section is a no-op.
Pausedis runtime state (IOutboxConsumerManager),AsSingletonis fixed by which method you call, and the group identity is pinned to the explicitconsumerGroupIdargument — none of them comes from the section.
The section just has to be part of the IConfiguration registered in DI (e.g. appsettings.json under Host.CreateDefaultBuilder); the provider reads it lazily at startup, so editing the section and restarting is the configuration workflow.
Consumer group settings (OutboxConsumerSettingsBuilder)
OutboxConsumerSettingsBuilder — the single fluent builder for all consumer group settings:
| Method | Default | Description |
|---|---|---|
WithInterval(span) |
1 min | Polling interval |
WithInitialDelay(span) |
10 s | Delay before the first run after startup |
StartImmediately() |
— | Start without waiting for the first interval |
WithConcurrencyLimit(n) |
1 | Concurrent jobs in the group |
WithMaxConcurrency(n) |
48 | Max simultaneously running jobs within the group (high — for cloud deployments with connection pooling; override for local/small deployments) |
WithRetryCountOnError(n) |
0 | Job retries on error; WithInfiniteRetries() sets int.MaxValue, negative values are rejected |
WithNoRetries() |
— | Disable retries |
WithInfiniteRetries() |
— | Infinite retries |
WithMaxBatchSize(n) |
128 | Max messages per batch |
WithMaxProcessingIterations(n) |
10 | Iterations per cycle (-1 = drain until empty) |
WithSingleIteration() |
— | One iteration (testing) |
WithUnlimitedIterations() |
— | Drain until empty |
WithIterationDelay(span) |
0 s | Delay between iterations within a cycle |
WithIdleBackoff(span) |
— | Enable adaptive empty-queue back-off up to maxInterval (must be >= Interval) |
WithFixedInterval() |
— | Explicitly keep a fixed poll interval (no back-off) |
WithLockDuration(span) |
10 s | Lock TTL for a rented message |
WithLockRenewal(span) |
3 s | Lock renewal interval |
WithLookbackInterval(span) |
7 days | How far back to scan for unprocessed messages |
WithMaxDeliveryAttempts(n) |
3 | Stop retrying a message after N attempts |
WithBatchingWindow(span) |
3 s | Window for accumulating messages into a batch |
WithNoBatchingWindow() |
— | Take everything available now |
WithPerTenantTimeout(span) |
0 (none) | Per-tenant processing timeout |
WithPerTenantMaxDegreeOfParallelism(n) |
1 (sequential) | Per-tenant parallelism (-1 = one per CPU) |
WithTenantSequentialProcessing() |
1 | Sequential processing per tenant |
WithTenantMaxParallelism() |
-1 | Max parallelism per tenant |
Paused(bool) |
false | Pause the consumer group |
Resumed() |
— | Unpause |
AsSingleton(bool) |
true | Singleton (one per cluster) vs scoped consumer |
WithTag(object) |
— | Free-form tag for the group (diagnostics) |
Latency budget for every group at once
Instead of repeating Interval / IdleMaxInterval / InitialDelay on each AddDelivery, set them once on the provider. These are defaults — a group's configuration section and its callback still win:
services.AddSaOutboxUsingPostgreSql(cfg => cfg
.WithDataSource(...)
.WithPollingBudget(TimeSpan.FromMinutes(1))); // idle group: ~48 s worst case
// or one of: .WithPollingProfile(PgPollingProfile.Quiet / Balanced / Responsive)
WithPollingBudget splits the budget as BatchingWindow (3 s) + reserve (max(5 s, 20%)) + IdleMaxInterval and sets MaxProcessingIterations = -1 so a burst is drained in the same cycle. See Configuration/Readme.md → PgPollingSettings for the value tables. Nothing is registered unless one of these methods is called.
Runtime settings management
Settings are automatically registered in IOutboxConsumerManager on first job startup. For runtime changes:
var manager = host.Services.GetRequiredService<IOutboxConsumerManager>();
// Atomic swap — the new snapshot is applied on the next iteration
manager.Apply("cg_order_consumer", s => s with { MaxBatchSize = 64 });
// Pause / Resume
manager.Pause("cg_order_consumer");
manager.Unpause("cg_order_consumer");
// Subscribe to changes
using var sub = manager.Subscribe("cg_order_consumer", updated =>
{
// react to settings changes
});
// Check state
bool paused = manager.IsPaused("cg_order_consumer");
3. PostgreSQL Connection
Where inside AddSaOutboxUsingPostgreSql:
.AddSaOutboxUsingPostgreSql(cfg => cfg
.WithDataSource(ds => ds
.Options(ob => ob.Configure(o => o.ConnectionString =
"Host=...;Database=...;Username=...;Password=...")
// Pooling, timeout and search path live inside the connection string itself, e.g.:
// "Host=...;Minimum Pool Size=5;Maximum Pool Size=100;Search Path=outbox"
))
// ...
)
4. Message Serialization
IOutboxMessageSerializer lives in the core (Sa.Outbox.Serialization); the core registers
JsonOutboxMessageSerializer (UTF-8 JSON) as the fallback, so this section is optional. A serializer
set here is registered for this provider key: another provider keeps its own one (or the
container-level one), so a container with several providers can hold different codecs side by side.
Configure it at the core level (AddSaOutbox(b => b.WithMessageSerializer(...))) when every provider
should share one.
Where inside AddSaOutboxUsingPostgreSql:
.AddSaOutboxUsingPostgreSql(cfg => cfg
.WithMessageSerializer(OrderSerializer.Instance) // singleton instance
// .WithMessageSerializer<MySerializer>() // DI-resolved singleton
// .WithMessageSerializer(sp => sp.GetRequiredService<MySerializer>()) // factory
)
AOT-friendly serializer
For Native AOT, avoid reflection-based serialization:
[JsonSourceGenerationOptions(GenerationMode = JsonSourceGenerationMode.Metadata)]
[JsonSerializable(typeof(OrderCreated))]
public partial class OrderJsonContext : JsonSerializerContext { }
public class OrderSerializer : IOutboxMessageSerializer
{
public T? Deserialize<T>(Stream stream) => typeof(T) switch
{
Type t when t == typeof(OrderCreated) =>
(T?)(object?)JsonSerializer.Deserialize(stream, OrderJsonContext.Default.OrderCreated),
_ => default
};
public void Serialize<T>(Stream stream, T value)
{
if (typeof(T) == typeof(OrderCreated))
JsonSerializer.Serialize(stream, value!, OrderJsonContext.Default.OrderCreated);
}
}
5. Poison Handler (permanent failures)
__error$ and the task row already persist every terminal failure, so the provider does not need a
dead-letter queue. If the application needs its own sink — forward the payload to another queue,
object storage, or raise an alert — register IPgOutboxPoisonHandler:
.AddSaOutboxUsingPostgreSql(cfg => cfg
.WithPoisonHandler<MyPoisonHandler>() // DI-resolved
// .WithPoisonHandler(new MyPoisonHandler()) // instance
// .WithPoisonHandler(sp => new MyPoisonHandler(sp.GetRequiredService<MyStore>()))
)
The handler receives PgOutboxPoisonMessage: OutboxId, TaskId, PayloadId, PayloadType,
ConsumerGroup, TenantId, Part, Attempt, StatusCode, StatusMessage, Exception, and the
payload serialized with the core IOutboxMessageSerializer.
It is called for every Error/Error5xx and for MaximumAttemptsError, before the task is
finalized, and its return value selects the strategy:
| Return | Result |
|---|---|
PgOutboxPoisonAction.Default |
keep the built-in persistence: __error$ row + task finalized with error_id |
PgOutboxPoisonAction.Handled |
the handler owns the error record: __error$ is skipped; the task is still finalized as Error/MaximumAttemptsError with error_id = NULL |
| throw | the batch is not finalized, the task stays Processing, its lock expires and it is re-rented, so the handler runs again (at-least-once) |
The default NoOpPgOutboxPoisonHandler returns Default, which keeps the previous behaviour.
Handled with an empty handler
Returning Handled and doing nothing is a drop that leaves only the skeleton behind:
| Item | Result |
|---|---|
__error$ |
not written |
__log$ (delivery log) |
a row is still written: status Error/508, status_message, payload_id, group, tenant, task_id, transact_id, error_id = NULL |
task$ |
terminal Error/Error5xx/508, error_id = NULL, delivery_attempt + 1, lock_expires_on updated |
msg$ |
the payload row stays unchanged |
| re-delivery | none — Error is not in the rent status set |
The payload is therefore not lost physically: it can still be found through task$/__log$ by
msg_payload_id and then through msg$ by msg_id, until cleanup drops the partition. What is lost
is the error record: no exception type or stack in __error$, and status_message holds only the text
the consumer passed to Error(...).
| Handler result | __error$ |
task$ |
Retry | Data |
|---|---|---|---|---|
Default |
written (de-duplicated) | Error, error_id |
no | payload in msg$, error in __error$ |
Handled + own store |
no | Error, error_id = NULL |
no | payload owned by the handler |
Handled + nothing |
no | Error, error_id = NULL |
no | payload only in msg$/__log$ until cleanup |
| throw | no | stays Processing |
yes, after the lock expires | payload in msg$ |
Return Handled only after the sink succeeded. If the write failed, either throw (the task stays
claimable and the handler runs again) or return Default (the provider writes __error$).
6. Table Settings
Where inside AddSaOutboxUsingPostgreSql:
.AddSaOutboxUsingPostgreSql(cfg => cfg
.WithOutboxSettings((_, settings) =>
{
// ── Schema ─────────────────────────────────────────
settings.TableSettings.WithSchema("my_outbox");
// ── Base table name (all tables derive from it) ────
settings.TableSettings.UseBaseTableName("outbox");
// Results: outbox, outbox__msg$, outbox__log$, outbox__error$, etc.
// ── FillFactor per table ───────────────────────────
settings.TableSettings.Message.FillFactor = 100; // append-only, never updates
settings.TableSettings.TaskQueue.FillFactor = 65; // read-write, needs free space
// ── Custom field names ─────────────────────────────
settings.TableSettings.TaskQueue.Fields.TaskId = "id";
settings.TableSettings.TaskQueue.Fields.TenantId = "client_id";
settings.TableSettings.TaskQueue.Fields.ConsumerGroup = "grp";
// ── Individual table names ─────────────────────────
settings.TableSettings.UseBaseTableName("events");
// or individually:
settings.TableSettings.WithMsgTableName("inbox_messages");
settings.TableSettings.WithDeliveryTableName("inbox_log");
// ── Migration (partition creation) ─────────────────
settings.MigrationSettings.AsBackgroundJob = true; // default
settings.MigrationSettings.ForwardDays = 2; // create partitions N days ahead
settings.MigrationSettings.ExecutionInterval = TimeSpan.FromHours(6);
// ── Cleanup (old partition removal) ────────────────
settings.CleanupSettings.AsBackgroundJob = true; // default
settings.CleanupSettings.DropPartsAfterRetention = TimeSpan.FromDays(30);
settings.CleanupSettings.ExecutionInterval = TimeSpan.FromHours(4);
// ── Min offset (prevent reprocessing) ──────────────
settings.ConsumeSettings.WithMinOffset("cg_order_consumer", DateTimeOffset.Now);
})
)
Table overview
| Setting property | Default table name | Role | FillFactor |
|---|---|---|---|
Message |
outbox__msg$ |
Source messages (BINARY COPY target) | 100 |
TaskQueue |
outbox |
Active task queue (SKIP LOCKED) | 65 |
Delivery |
outbox__log$ |
Delivery history | 100 |
Error |
outbox__error$ |
Permanent errors | 100 |
Type |
outbox__type$ |
Type → hash cache | 100 |
Offset |
outbox__offset$ |
Consumer group offsets | 100 |
Partitioning strategy
| Table | Partition key | Sort key |
|---|---|---|
| Message | tenant_id + msg_part |
msg_created_at |
| TaskQueue | tenant_id + consumer_group |
task_created_at |
| Delivery | tenant_id + consumer_group |
delivery_created_at |
| Error | date only | error_created_at |
| Type / Offset | none | — |
7. Running Jobs Manually
By default migration and cleanup run as background jobs via Sa.Schedule. You can also trigger them on demand:
// The outbox migration service lives in the outbox's own DI scope ("outbox").
var migrationService = host.Services.GetRequiredKeyedService<IMigrationService>(OutboxScope.DefaultKey);
bool ok = await migrationService.WaitMigration(TimeSpan.FromSeconds(30), ct);
// Or check current state
if (!migrationService.OnMigrated.IsCancellationRequested)
{
// No migration has completed yet: delivery jobs are held back until the first one does
}
Message Lifecycle
Publish(messages)
│ COPY (BINARY) — one transaction + advisory lock (tenant, part)
▼
outbox__msg$ ← the message lives here, msg_seq = nextval()
│
│ ① materialize — once per consumer group
│ INSERT INTO outbox … SELECT … WHERE msg_seq > @offset
│ under the (consumer_group, tenant) advisory lock;
│ __offset$ is advanced in the same transaction
▼
outbox (task queue) ← one row per "message × group" pair
│
│ ② rent — SELECT … FOR UPDATE SKIP LOCKED
│ → delivery_status_code = Processing, task_lock_expires_on
▼
IConsumer<T>.Consume
│
│ ③ finish — ReturnDelivery
│ __log$: one record per delivery attempt
│ __error$: terminal failures only
│ outbox: result status + attempt counter
▼
the row stays in outbox with its status and is not rented again
(Pending/Processing/Postpone/Retry/Warn stay rentable);
it is physically removed only with its daily partition, on retention
Note: __offset$ is a materialisation watermark, not a delivery watermark. It moves when the
tasks are created, before the consumer has seen them. That is safe: once it has moved, the tasks are
already in outbox, so a crash between materialisation and delivery loses nothing — the tasks are
simply rented later.
Consumer groups
Every group is an independent copy of the stream. A message in __msg$ is materialised once per
group that subscribes to its type (part + payload type), so groups neither block nor consume each
other's messages:
outbox__msg$ (shared: LIST(tenant) → LIST(part) → RANGE(day))
│
┌─────────────────┼─────────────────┐ one copy per group
▼ ▼ ▼
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│ "orders" │ │ "audit" │ │ "billing" │
│ cursor │ │ cursor │ │ cursor │
│ (grp,tenant)│ │ (grp,tenant)│ │ (grp,tenant)│
└──────┬──────┘ └──────┬──────┘ └──────┬──────┘
▼ ▼ ▼
task queue task queue task queue ← per-group partition
│ │ │
IConsumer<Order> IConsumer<Order> IConsumer<Invoice> ← per-group handler
│ │ │
▼ ▼ ▼
__log$ (grp) __log$ (grp) __log$ (grp) ← per-group delivery log
"orders" and "audit" above are two groups on the same message type: both receive every
message, each into its own queue. "billing" subscribes to a different type and only gets its own.
What is per group and what is shared:
| Resource | Where | Granularity |
|---|---|---|
| Cursor | __offset$ |
(consumer_group, tenant) |
| Task queue | outbox |
LIST(tenant) → LIST(consumer_group) → RANGE(day) |
| Delivery log | __log$ |
LIST(tenant) → LIST(consumer_group) → RANGE(day) |
Schedule and settings (Interval, MaxBatchSize, MaxProcessingIterations, LockDuration, …) |
the DeliveryJob in the scheduler |
per group |
| Materialisation lock | advisory lock | (consumer_group, tenant) — groups never block each other |
| Messages | __msg$ |
shared |
| Type registry | __type$ |
shared |
| Terminal errors | __error$ |
shared, not split by group (partitioned by day) |
Practical consequences:
- consumption capacity is per group, not per database:
MaxBatchSize × MaxProcessingIterationsperInterval(with the defaults 128 × 10 per minute ≈ 1 280 messages/min per group); - to consume faster, add groups (different types/parts) or raise the batch and poll frequency — but not the number of workers on one group, which is bounded by that group's own lock;
- each group has its own cursor, so a lagging group does not slow the others down;
- a message whose type no group subscribes to stays in
__msg$and leaves only with its partition on retention.
Result Statuses
After processing each message, call exactly one method:
| Method | When to use | Next action |
|---|---|---|
msg.Ok() |
Everything went fine | Status written, row is not rented again (removed with its partition) |
msg.Created() |
Side-effect resource created | Status written, row is not rented again |
msg.Accepted() |
Accepted for async processing | Status written, row is not rented again |
msg.NoContent() |
Processed, no data to return | Status written, row is not rented again |
msg.Fail(ex) |
Unrecoverable failure | Logged to __error$, no retry |
msg.Warn(ex) |
Transient issue (network, timeout) | Rented again after the batching window |
msg.Postpone(ts) |
Need to wait before retry | Requeued after ts |
msg.Retry(ts, reason) |
Retry with metadata | Requeued with attempt info |
msg.Aborted(reason) |
Intentionally skip | Marked skipped, no retry |
msg.ErrorMaxAttempts() |
Max attempts exhausted | Logged to __error$, no retry |
Error/Error5xx and ErrorMaxAttempts are additionally handed to IPgOutboxPoisonHandler before
the status is written; see Poison Handler.
Database Architecture
📂 my_outbox (schema)
├── 📄 outbox__msg$ — Source messages (read-only, binary COPY)
├── 📄 outbox — Task queue (read-write, SKIP LOCKED)
├── 📄 outbox__log$ — Delivery history (read-only)
├── 📄 outbox__error$ — Permanent errors (read-only)
├── 📄 outbox__type$ — Type ↔ hash registry (read-only)
└── 📄 outbox__offset$ — Consumer group offsets (advisory lock)
Under the hood
| Mechanism | What it solves |
|---|---|
msg_seq (BIGSERIAL) |
Database-assigned insertion order, and the consumption cursor (__offset$.group_offset_seq). Denormalised into the task table so RETURNING msg_seq advances the cursor in the same statement |
| UUID v7 message id | Application-minted id, used as the join key between __msg$ and the task queue. Deliberately not used for ordering: it comes from application clocks, which skew and get backdated |
| BINARY COPY | Maximum throughput for bulk message insertion |
| SKIP LOCKED | Multiple workers safely compete for tasks — and only for tasks. On the single cursor row "skip" would mean reporting an empty queue while a backlog exists |
| Advisory Locks | Offset coordination per (consumer group, tenant) |
| Publish lock | pg_advisory_xact_lock per (tenant, part) around the COPY — makes msg_seq order equal commit order, so the max(msg_seq) cursor cannot pass a row that is still committing |
| Cursor index | (msg_seq) on __msg$ — turns the loader's cursor read into an index range scan instead of a Seq Scan of the whole lookback window (measured 56.8 ms → 0.166 ms on 7 × 40k rows) |
| Daily partitions | Pruning keeps one poll on a few leaves, and retention is a DROP PARTITION (indexes leave with the leaf) instead of a DELETE |
| murmurHash3 | Compact type identification cached in __type$ — see TypeResolve/Readme.md |
| SqlCacheSplitter | Splits large multi-row INSERT/UPDATE statements into chunks up to the shared DefaultMaxLen (1024) — see SqlBuilder/Readme.md and Commands/Readme.md |
Bulk Publishing at Scale
Rules
- Batch 256–1024, not less. Throughput grows with the batch (4.7k → 58k rows/s without the
lock), because every publish pays a fixed price: BEGIN, lock, COPY, fsync on COMMIT. One publish =
one
COPY= one transaction. - One producer per part. The lock serializes not only the COPY but the commits of the part, so group commit stops amortizing fsync: 8 producers into one part give 14.8k rows/s against 55.3k without the lock (−67 %). With one producer per part the cost is zero; 8 parts give 43.7k rows/s and p50 32 ms against 23.5k/77 ms over three.
- Do not mix parts/tenants in one
Publishcall — otherwise a singleCOPYtakes N locks and holds all of them until commit. - Keep transactions short. Publishing inside a business transaction holds the part lock for its whole duration. Atomicity requires the same transaction context — which means the business transaction must be short, not "5 seconds with an external HTTP call inside".
- Do not add indexes to
__msg$. Each one costs ~140–200 B of WAL per row and 30–50 % of throughput. Exactly one is needed — the cursor index(msg_seq). The PK is the most expensive; drop it only deliberately (you lose the duplicate-msg_idprotection). - Do not
COPYdirectly into a leaf — measured: not faster (72–78k against 87k), routing is not the bottleneck. - Leave
msg_seqalone. The sequence costs ~4 B of WAL per row and contributes nothing to the ceiling.CACHEbreaks the cursor (message loss). - Server:
wal_compression=lz4(orzstd) — WAL is the biggest item, theUNLOGGEDcontrol showed up to 2.9× of headroom;max_wal_size/checkpoint_timeoutso checkpoints do not run constantly; keepfull_page_writeson; keepsynchronous_commiton. - Connection pooling:
pool_mode=transaction(notstatement— the library works in transactions). The locks are transaction-scoped, so PgBouncer does not break them. - Partitions: many daily partitions do not slow inserts down, and retention via
DROP PARTITIONis free (indexes go away with the leaf).
Measurements
Stand: PostgreSQL 15 in Docker, __msg$ = LIST(tenant) → LIST(part) → RANGE(day), 256 leaves,
PK (msg_id, tenant_id, msg_part, msg_created_at) plus the cursor index (msg_seq), 200 B payload,
synchronous_commit=on (the real mode), fsync ≈ 11 ms. Client — Npgsql 10, NpgsqlBinaryImporter,
exactly as in BulkInsertMsgCommand. 8 producers, 8-second runs, median of 3.
The absolute numbers are a property of that stand. Carry them over with the formula at the end of the section rather than directly; run-to-run spread on this stand is ±30 % (latencies are steadier than throughput).
Batch and the cost of the lock — 8 producers, one part:
| batch | no lock, rows/s | with lock, rows/s | lock cost |
|---|---|---|---|
| 16 | 4 685 | 1 198 | −74 % |
| 64 | 17 578 | 3 524 | −80 % |
| 256 | 44 946 | 14 808 | −67 % |
| 1024 | 58 450 | 26 222 | −55 % |
Scenarios — batch 256, lock enabled (the "1 part" configuration is the 256 row above, measured in a different run):
| parts | rows/s | tx/s | p50, ms | p99, ms |
|---|---|---|---|---|
| 1 | 17 183 | 67 | 99.1 | 413.4 |
| 3 | 23 498 | 92 | 77.5 | 156.3 |
| 8 | 43 683 | 171 | 32.2 | 176.0 |
WAL per row — batch 1024, 8 producers:
| table | WAL/row |
|---|---|
| PK + cursor index | 526 B |
| cursor index only | 386 B |
| no indexes | 319 B |
UNLOGGED (control) |
3 B |
Column legend:
batch— rows in oneCOPY, i.e. in one publish and one transaction (OutboxPublishSettings.MaxBatchSize).rows/s— rows per second, summed over all producers.tx/s— transactions (publishes) per second; with a knownbatchthat isrows/s / batch.no lock/with lock— before and after the(tenant, part)advisory lock was introduced;lock costis the throughput lost to it.parts— how many distinct(tenant, part)are in play; 8 producers over 3 parts means ~2.7 producers per key.p50/p99— percentiles of one publish's latency on the client, including lock wait.WAL/row— WAL bytes per inserted row.
Formula for carrying this over to your hardware:
t_txn ≈ F + batch × t_row
rows/s per part ≈ batch / t_txn
rows/s total ≈ min(N_parts × rows/s per part, server ceiling)
F is the effective cost of one commit in your installation (group commit included), t_row ≈
0.02 ms per row. Both follow from two points of the batch table: 1/R(B) = F/B + t_row, so with
B = 256 and B = 1024 (R = 44 946 and 58 450) we get F ≈ 1.8 ms and t_row ≈ 15 µs — effective
values for 8 producers, not the raw disk fsync.
What to monitor in production
xact_commit/sandrows/son__msg$— the actual cost of publishing;- WAL bytes/row (
pg_current_wal_lsnbefore/after a window) — expect ~350–550 B; growth means extra indexes or bloat; pg_stat_activity.wait_event—Lock:advisoryfor more than a few ms means the key is overloaded (too many producers per part);- delivery lag:
max(msg_seq) − group_offset_seqper group; - publish p99 on the client, not only server-side time.
Migrating 0.17 → 0.18 (existing deployments)
The database schema does not change. The tables, columns, partitions, __offset$ and the
msg_seq cursor are the same; OutboxRentToken lives in memory only. No data migration is needed.
Code changes
| Before | After |
|---|---|
IOutboxBulkWriter: 4 overloads, the legacy one abstract |
one abstract InsertBulk(batch, context, ct); the legacy overloads and OutboxMessage<T> are removed |
OutboxDeliveryMessage<T>(OutboxId, OutboxMessage<T>, DeliveryInfo) |
a flat record (OutboxId, PayloadId, Payload, PartInfo, DeliveryInfo, RentToken) |
IOutboxPlugin (commented out) |
AddOutboxProvider(descriptor, builder) |
| — | IOutboxProviderRegistry, IOutboxPublisherResolver, IOutboxHealth |
- A custom writer (if any): implement
InsertBulk(batch, context, ct)and register it withAddOutboxProvider(...).WithBulkWriter(sp => new MyWriter(...)). - A custom consumer keeps working: string keys and the string overloads of
IOutboxConsumerManagerare preserved. - The PG/Redis packages are already on the new contracts — an application that only uses
AddSaOutbox/AddSaOutboxUsingPostgreSqlandIOutboxMessagePublisherneeds no changes.
Registration
- The default provider key stays
"outbox"; its unkeyed aliases are kept:IOutboxMessagePublisher,IOutboxProviderRegistry,PgOutboxSettings(+ components),PgPollingSettings,IDatabase(Redis), and the three storage contracts. Existing code that resolves them without a key keeps working. - New one-call overload:
AddSaOutboxUsingPostgreSql(cfg, outbox), withscopeKeywritten once. WithOutboxSettings/WithPollingactions are now per provider; one registration no longer affects a second provider.
Adding a second PG provider
- Give it its own schema (or its own table names):
WithOutboxSettings((_, s) => s.TableSettings.WithSchema("billing")). - The startup
PgMultiProviderGuardfails when two providers point at the same tables (every message would be delivered twice), i.e. the same(host, port, database)with the same schema/table names. - The second provider's partitions are created by its own migration job (in its own schema).
- Consumer groups may now share an id across providers: the manager and consumers are addressed by
(providerKey, groupId); the string overloads throw an ambiguity error.
Moving an existing outbox to a non-default key
- The PG storage does not change: same schema/tables/cursors, so stored messages and offsets are read as before.
- The data source becomes keyed (
AddSaPostgreSqlDataSource(key, ...)); a borrowed unkeyedNpgsqlDataSourceis still found throughPgDataSourceResolver. - The schedule and the partitional module move to the new scope key.
Rollback
0.18 writes no new schema, so rolling back to 0.17 is safe. The only trace is the second provider's tables if it already created partitions; they simply stay unused.
Redis: moving an existing provider to a non-default key changes the stream prefix (
sa:outbox→sa:outbox:{key}). Set the prefix explicitly viaWithOutboxSettingsto keep the old keys.
Requirements
- .NET 10.0 or higher
- PostgreSQL 15+
- Compatible with Native AOT
License
MIT
| Product | Versions Compatible and additional computed target framework versions. |
|---|---|
| .NET | net10.0 is compatible. net10.0-android was computed. net10.0-browser was computed. net10.0-ios was computed. net10.0-maccatalyst was computed. net10.0-macos was computed. net10.0-tvos was computed. net10.0-windows was computed. |
-
net10.0
- Microsoft.Extensions.Configuration (>= 10.0.14)
- Microsoft.Extensions.DependencyInjection.Abstractions (>= 10.0.14)
- Microsoft.Extensions.Logging.Abstractions (>= 10.0.14)
- Microsoft.Extensions.Options (>= 10.0.14)
- Sa.Outbox (>= 0.20.0)
- Sa.Partitional.PostgreSql (>= 0.20.0)
NuGet packages
This package is not used by any NuGet packages.
GitHub repositories
This package is not used by any popular GitHub repositories.
| Version | Downloads | Last Updated |
|---|---|---|
| 0.20.0 | 64 | 11/13/2026 |
| 0.18.0 | 98 | 11/3/2026 |
| 0.17.0 | 96 | 10/29/2026 |
| 0.16.0 | 104 | 10/14/2026 |
| 0.15.0 | 96 | 10/2/2026 |
| 0.12.0 | 168 | 7/28/2026 |
| 0.11.0 | 150 | 7/14/2026 |
| 0.10.0 | 260 | 7/2/2026 |
| 0.9.0 | 123 | 4/29/2026 |
| 0.8.1 | 148 | 2/13/2026 |
| 0.8.0 | 142 | 2/10/2026 |
| 0.7.2 | 137 | 2/2/2026 |
| 0.7.1 | 131 | 1/28/2026 |
| 0.7.0 | 138 | 1/27/2026 |
| 0.6.0 | 132 | 1/15/2026 |
| 0.5.0 | 134 | 12/29/2025 |
| 0.4.1 | 313 | 12/18/2025 |
| 0.4.0 | 311 | 12/17/2025 |
| 0.3.0 | 186 | 10/17/2025 |
| 0.2.0 | 249 | 9/3/2025 |