ITB.HangfirePostgreSql.ValkeyQueue 0.1.2-alpha

ITB.HangfirePostgreSql.ValkeyQueue

A Valkey/Redis-backed queue provider for Hangfire.PostgreSql. PostgreSQL stays the durable source of truth (job data, state history, scheduled/recurring jobs, dashboard); only the hot queue path (job IDs) moves to Valkey — LPUSH to enqueue, blocking BLMOVE to dequeue — so idle worker polling load on PostgreSQL drops toward zero.

In a measured run, an idle Hangfire server's PostgreSQL load fell by 97.6 % (1.37 → 0.03 queries/s). Per-job cost fell 12.5 %, and throughput was unchanged. Full methodology, environment and caveats: docs/LOAD-TEST.md.

Prerelease. On Hangfire.PostgreSql 1.21.1 the baseline already wakes idle workers via PostgreSQL LISTEN/NOTIFY, so the pickup-latency win is marginal and the reduction under sustained load is modest. The win is idle database load. Validate against your own workload before relying on it, and keep pure-PostgreSQL as the fallback.

Installation

dotnet add package ITB.HangfirePostgreSql.ValkeyQueue

Targets .NET 10. Requires Hangfire.PostgreSql 1.21.x and a Valkey (or Redis) 7+ server.

Usage

using Hangfire;
using Hangfire.PostgreSql;
using Hangfire.PostgreSql.Factories;
using Hangfire.PostgreSql.ValkeyQueue;
using Hangfire.Server;
using StackExchange.Redis;

var storageOptions = new PostgreSqlStorageOptions { UseSlidingInvisibilityTimeout = true };
var storage = new PostgreSqlStorage(
    new NpgsqlConnectionFactory(pgConnectionString, storageOptions), storageOptions);

var redisOptions = ConfigurationOptions.Parse(valkeyConnectionString);
var mux = ConnectionMultiplexer.Connect(redisOptions);

var valkey = new ValkeyQueueOptions
{
    InvisibilityTimeout = TimeSpan.FromMinutes(30),   // MUST exceed your longest job
    BlockingConnectionConfig = redisOptions,          // required on managed Valkey (TLS + AUTH)
};

storage.UseValkeyQueues(mux, valkey);                 // point the "default" queue at Valkey

services.AddHangfire(c => c.UseStorage(storage));
services.AddHangfireServer();

// Safety nets — recover from Valkey data loss / worker death. Register as a singleton
// IBackgroundProcess so AddHangfireServer picks it up.
services.AddSingleton<IBackgroundProcess>(
    new ValkeyQueueMaintenance(mux, valkey, pgConnectionString));

A complete, runnable worked example — including the ValkeyQueueMaintenance registration that is easy to forget — is in docs/EXAMPLE.md.

Configuration

ValkeyQueueOptions:

Correctness

The enqueue is no longer transactional with job creation (it pushes to Valkey, not PostgreSQL), so two safety nets close every failure window:

• Reconciler re-enqueues jobs PostgreSQL reports as Enqueued but missing from Valkey (Valkey data loss / failover).
• Orphan sweep requeues jobs stuck in a processing list past InvisibilityTimeout (worker died mid-job).

Both live in ValkeyQueueMaintenance. Without it registered, those failures are silent job loss — this is the one registration step you cannot skip.

Duplicates are execution-safe: only one worker wins the Enqueued → Processing state transition in PostgreSQL; the loser is discarded.

Notes

• One dedicated blocking connection per worker thread (BLMOVE monopolises its connection) — fine for tens of workers, revisit for hundreds.
• On managed Valkey (ElastiCache with TLS + AUTH) set BlockingConnectionConfig; otherwise the blocking connections are derived from the shared multiplexer's connection string, which masks the password and every blocking connection fails to authenticate.
• InvisibilityTimeout must exceed your longest-running job or the sweep will re-flag it.
• Keys are wrapped in a {hash tag} so all three keys for a queue land in the same cluster hash slot; without it, cluster mode rejects the BLMOVE with CROSSSLOT.

Development

# Postgres + Valkey for the tests (pg_stat_statements is only needed for benchmarks)
docker run -d --name hfvq-pg -e POSTGRES_PASSWORD='Asd123!1' \
  -e POSTGRES_DB=hangfire_bench -p 55432:5432 postgres:17-alpine
docker run -d --name hfvq-valkey -p 56379:6379 valkey/valkey:8-alpine

dotnet test

The suite covers the queue hand-off, both recovery paths, and the cluster hash-slot invariant. Endpoints are overridable via POSTGRES_TEST_CONNECTION, REDIS_TEST_HOST and REDIS_TEST_PORT.

License

MIT — see LICENSE.