PgOutput2Json.Kafka
2.0.0
See the version list below for details.
dotnet add package PgOutput2Json.Kafka --version 2.0.0
NuGet\Install-Package PgOutput2Json.Kafka -Version 2.0.0
<PackageReference Include="PgOutput2Json.Kafka" Version="2.0.0" />
<PackageVersion Include="PgOutput2Json.Kafka" Version="2.0.0" />
<PackageReference Include="PgOutput2Json.Kafka" />
paket add PgOutput2Json.Kafka --version 2.0.0
#r "nuget: PgOutput2Json.Kafka, 2.0.0"
#:package PgOutput2Json.Kafka@2.0.0
#addin nuget:?package=PgOutput2Json.Kafka&version=2.0.0
#tool nuget:?package=PgOutput2Json.Kafka&version=2.0.0
PgOutput2Json
PgOutput2Json is a .NET library that uses PostgreSQL logical replication to stream JSON-encoded change events from your database to a .NET application or a message broker like RabbitMQ, Kafka, or Redis.
π Use Cases
Use PgOutput2Json when you need to react to DB changes in near real-time:
- π Background tasks triggered by inserts/updates
- β‘ Low-latency application cache updates
- π Asynchronous logging (to file or another DB)
- π Real-time data aggregation
- π οΈ Lightweight ETL for DWH solutions
All with minimal latency β events are dispatched shortly after a transaction is committed, though large transactions may introduce additional delay due to processing and transmission overhead.
π Supported Outputs
- β .NET application via a simple delegate handler
- β Kafka (used by PgHook.Kafka)
- β RabbitMQ (Streams + Classic Queues - used by PgHook.RabbitMqStreams)
- β Redis (Streams + Pub/Sub Channels - used by PgHook.Redis)
- β SQLite (used by PgFreshCache)
- β MongoDB
- β Amazon Kinesis
- β Amazon DynamoDB
- β Azure Event Hubs
- β Webhooks (used by PgHook)
Plug-and-play adapters handle the heavy lifting β or handle messages directly in your app for maximum control.
π§ How It Works
PostgreSQL 10+ ships with a built-in logical decoding plugin called pgoutput, which emits row-level changes (INSERT/UPDATE/DELETE) over a replication slot.
PgOutput2Json:
- Connects to a replication slot via
pgoutput - Converts change events into clean JSON
- Sends them to your application or a supported message broker
No extra plugins needed β pgoutput comes with PostgreSQL by default.
The change events JSON format:
{
"c": "U", // Change type: I (insert), U (update), D (delete)
"w": 2485645760, // Transaction final LSN β part of the deduplication key
"n": 1, // Message number within the transaction β part of the deduplication key
"t": "schema.table", // Table name (if enabled in JSON options)
"k": { ... }, // Key values β included for deletes, and for updates if the key changed,
// or old row values, if the table uses REPLICA IDENTITY FULL
"r": { ... } // New row values (not present for deletes)
}
π§ͺ Testing
Every output is covered by integration tests, which run on each change against PostgreSQL 14 to 18.
Quick Start
Configure postgresql.conf
To enable logical replication, add the following setting in your postgresql.conf:
wal_level = logical
# If needed increase the number of WAL senders, replication slots.
# The default is 10 for both.
max_wal_senders = 10
max_replication_slots = 10
Other necessary settings usually have appropriate default values for a basic setup.
Note: PostgreSQL must be restarted after modifying this setting.
Create a Replication User
Next, create a user in PostgreSQL that will be used for replication. Run the following SQL command:
CREATE USER pgoutput2json WITH
PASSWORD '_your_password_here_'
REPLICATION;
If you are connecting from a remote machine, don't forget to modify pg_hba.conf to allow non-local connections for this user. After modifying pg_hba.conf, apply the changes by either:
- Sending a SIGHUP to the PostgreSQL process
- Running
pg_ctl reload - Or executing:
SELECT pg_reload_conf();
Create a Publication
Now, connect to the target database and create a publication to specify which tables and actions should be replicated:
CREATE PUBLICATION my_publication
FOR TABLE my_table1, my_table2
WITH (publish = 'insert, update, delete');
In the example code below, we'll assume the database name is my_database.
Set Up Kafka
Ensure you have a running Kafka broker accessible to your .NET application. This example assumes Kafka is running locally on the default port 9092.
Create a topic in Kafka named my_topic. You can do this using Kafkaβs command-line tools or through a Kafka UI.
Create .NET Worker Service
Set up a new .NET Worker Service and add the following package reference:
dotnet add package PgOutput2Json.Kafka
In your Worker.cs, use the following code to publish changes to Kafka:
using PgOutput2Json;
using PgOutput2Json.Kafka;
public class Worker : BackgroundService
{
private readonly ILoggerFactory _loggerFactory;
public Worker(ILoggerFactory loggerFactory)
{
_loggerFactory = loggerFactory;
}
protected override async Task ExecuteAsync(CancellationToken stoppingToken)
{
// This code assumes PostgreSQL and Kafka are on localhost
using var pgOutput2Json = PgOutput2JsonBuilder.Create()
.WithLoggerFactory(_loggerFactory)
.WithPgConnectionString("server=localhost;database=my_database;username=pgoutput2json;password=_your_password_here_")
.WithPgPublications("my_publication")
.UseKafka(options =>
{
options.ProducerConfig.BootstrapServers = "localhost:9092";
options.Topic = "my_topic";
})
.Build();
await pgOutput2Json.StartAsync(stoppingToken);
}
}
Once the service is running, any changes made to the tables listed in my_publication will be published to the my_topic topic in Kafka as JSON messages.
Note: This example uses a temporary replication slot, meaning it wonβt capture changes made while the worker was stopped.
Delivery Guarantees
Messages are confirmed to PostgreSQL in batches, and only after Kafka has acknowledged every message in the batch. If the batch fails, it is not confirmed: the publisher reconnects and PostgreSQL replays the unconfirmed changes.
Delivery is at-least-once: after a failure, the replayed changes are published to Kafka again, and consumers deduplicate on the w/n fields. Unless you set them yourself, the publisher configures the producer with:
| Setting | Value | Why |
|---|---|---|
EnableIdempotence |
true |
The broker rejects out-of-sequence writes, so retries cannot reorder or duplicate messages. |
MessageTimeoutMs |
0 (infinite) |
librdkafka times out each message separately. A timed-out message is skipped while later ones are still delivered, which leaves a gap. |
Partitioner |
Murmur2Random |
The partitioner of the Java client, so a key lands in the same partition as with other Kafka producers. |
The time limit applies to the whole batch instead, through KafkaPublisherOptions.ConfirmTimeout (5 minutes by default). While Kafka is unavailable, the confirm waits and logs a warning every 30 seconds. When the timeout expires, the producer is discarded and the listener reconnects.
Things to be aware of:
- Idempotence requires
acks=all, at most 5 in-flight requests and retries above 0. If you set an incompatible value explicitly (for exampleAcks = Acks.Leader), the producer fails to start. - If a message is permanently rejected (for example, it is larger than the topic's
max.message.bytes), the publisher keeps failing on that message until the problem is fixed. Skipping it would lose the change. Each attempt publishes the other messages of the unconfirmed batch again. - While the confirm waits, no further changes are read, and PostgreSQL retains the WAL for the replication slot until the batch is confirmed. A long Kafka outage therefore grows
pg_walon the database server.
Messages out of order after a rejected message
When the broker rejects a message, Kafka can still accept the messages sent after it. The batch is not confirmed and is replayed, so the rejected message reaches the topic once the problem is fixed, but after messages that follow it in the WAL. For example, if message 2 of 1, 2, 3 is rejected, the partition ends up with 1, 3, 1, 2, 3.
- A consumer that applies changes idempotently is not affected: a row may show an older version for a moment, and the replay brings it up to date.
- A consumer that keeps a watermark (the highest
(w, n)pair processed) skips the late message as a duplicate, and loses it.
To prevent this, set ProducerConfig.EnableGaplessGuarantee = true. librdkafka then stops the producer with a fatal error instead of sending anything after a failed message, and the publisher reconnects and replays in order. It is not enabled by default because librdkafka marks the setting as experimental ("subject to change or removal") and logs a warning on start when it is set.
Routing
Messages are routed by the message key. With partition key columns configured for a table (WithPartitionKeyFields), its rows are routed by those columns instead, with the same hash (murmur2). Messages without a key go to a random partition.
Working with Permanent Replication Slots
If you want to capture changes made while the worker is stopped, you need a permanent replication slot.
In the context of logical replication, a slot represents a stream of changes that can be replayed to a client in the order they were made on the origin server. Each slot streams a sequence of changes from a single database.
To create a replication slot, call the pg_create_logical_replication_slot function in the same database that holds the tables being tracked:
SELECT * FROM pg_create_logical_replication_slot('my_slot', 'pgoutput');
The first parameter is the name of the slot, that must be unique across all databases in a PostgreSQL cluster. Make sure you specify pgoutput as the second parameter - that specifies the correct logical decoding plugin.
The current position of each slot is persisted only at checkpoint, so in the case of a crash, the slot may return to an earlier log sequence number (LSN), which will then cause recent changes to be sent again when the server restarts.
Note: It is the responsibility of the receiving client to avoid the ill effects from handling the same message more than once.
β οΈ Important: Replication slots persist across crashes and know nothing about the state of their consumer(s). They will prevent removal of required resources even when there is no connection using them. This consumes storage because neither required WAL nor required rows from the system catalogs can be removed by VACUUM as long as they are required by a replication slot.
In extreme cases, this could cause the database to shut down to prevent transaction ID wraparound.
So, if a slot is no longer required, it should be dropped. To drop a replication slot, use:
SELECT * FROM pg_drop_replication_slot('my_slot');
Once the replication slot is created, to use it, simply specify the name of the slot in the PgOutput2JsonBuilder:
// ...
using var pgOutput2Json = PgOutput2JsonBuilder.Create()
.WithLoggerFactory(_loggerFactory)
.WithPgConnectionString("server=localhost;database=my_database;username=pgoutput2json;password=_your_password_here_")
.WithPgPublications("my_publication")
.WithPgReplicationSlot("my_slot") // <-- slot specified here
//...
| Product | Versions Compatible and additional computed target framework versions. |
|---|---|
| .NET | net8.0 is compatible. net8.0-android was computed. net8.0-browser was computed. net8.0-ios was computed. net8.0-maccatalyst was computed. net8.0-macos was computed. net8.0-tvos was computed. net8.0-windows was computed. net9.0 is compatible. net9.0-android was computed. net9.0-browser was computed. net9.0-ios was computed. net9.0-maccatalyst was computed. net9.0-macos was computed. net9.0-tvos was computed. net9.0-windows was computed. net10.0 is compatible. net10.0-android was computed. net10.0-browser was computed. net10.0-ios was computed. net10.0-maccatalyst was computed. net10.0-macos was computed. net10.0-tvos was computed. net10.0-windows was computed. |
-
net10.0
- Confluent.Kafka (>= 2.15.0)
- Microsoft.Extensions.Logging.Abstractions (>= 10.0.0)
- PgOutput2Json (>= 2.0.0)
-
net8.0
- Confluent.Kafka (>= 2.15.0)
- Microsoft.Extensions.Logging.Abstractions (>= 8.0.0)
- PgOutput2Json (>= 2.0.0)
-
net9.0
- Confluent.Kafka (>= 2.15.0)
- Microsoft.Extensions.Logging.Abstractions (>= 9.0.0)
- PgOutput2Json (>= 2.0.0)
NuGet packages
This package is not used by any NuGet packages.
GitHub repositories
This package is not used by any popular GitHub repositories.
| Version | Downloads | Last Updated |
|---|---|---|
| 2.1.1 | 47 | 11/20/2026 |
| 2.1.0 | 199 | 11/16/2026 |
| 2.0.0 | 94 | 11/14/2026 |
| 2.0.0-rc.1 | 76 | 9/15/2026 |
| 1.5.0 | 172 | 3/29/2026 |
| 1.4.0 | 231 | 11/23/2025 |
| 1.3.0 | 236 | 10/12/2025 |
| 1.2.1 | 184 | 10/11/2025 |
| 1.2.0 | 219 | 8/17/2025 |
| 1.1.2 | 170 | 8/16/2025 |
| 1.1.1 | 244 | 8/11/2025 |
| 1.1.0 | 236 | 8/11/2025 |
| 1.0.6 | 245 | 8/10/2025 |
| 1.0.3 | 227 | 8/10/2025 |
| 1.0.2 | 239 | 8/10/2025 |
| 1.0.1 | 210 | 8/9/2025 |
| 1.0.0 | 325 | 8/5/2025 |
| 0.9.20 | 226 | 5/19/2025 |
| 0.9.17 | 244 | 5/18/2025 |
| 0.9.16 | 327 | 5/12/2025 |