redb.Tsak.Core.Pro 2.0.0

redb.Tsak

The runtime container for redb.Route integration pipelines. Deploy DLLs into a folder. Start, stop, reload them at runtime. Scale across nodes with leader election. Watch every message, every error, every restart — from a built-in REST API, a 30-command CLI, or a Blazor dashboard. Zero downtime. Production-tested.

---

What you get out of the box

Module-based deployment	Drop a .dll into Libs/ — Tsak loads it, builds an IRouteContext, starts the routes. Update the DLL — Tsak hot-swaps it without dropping a single in-flight message in other contexts.
Three deployment modes	Standalone (in-memory, no DB) · Single-node + redb (durable EAV state) · Cluster (leader election + automatic context redistribution across nodes).
Three API modes	Full (REST API + management) · Headless (zero ports, embedded use) · Readonly (only GET endpoints — perfect for monitoring sidecars).
5-layer configuration	Module ships defaults. DevOps overrides. Config hot-reloads. No code changes, no restarts.
Built-in observability	Per-process metrics (CPU/RAM/threads/GC, 12h history), per-route metrics (count/error rate/latency), ring-buffer logs, OpenTelemetry traces, optional Prometheus scrape.
Watchdog	Detects suspected and hung routes. Optionally auto-restarts them.
Quartz scheduler	Built-in IScheduler injected into every context. RAMJobStore standalone, AdoJobStore cluster-safe — schema auto-created on first start.
Security	API Key + HMAC-SHA256 + roles + expiry + revocation. Constant-time comparison. EAV-backed key store. Optional user binding (disabled user → key dies).
Module isolation	Per-module AssemblyLoadContext — dependencies don't conflict between modules.
Clients ready to ship	Typed C# ITsakApiClient, 30-command tsak CLI with profiles and JSON output, Blazor Server dashboard.

---

Why redb.Tsak exists

Most .NET teams either:

1. Bake their integration code into a Worker Service — every change means redeploy the entire process. Multiple unrelated pipelines fight in the same Program.cs.
2. Buy a heavyweight enterprise ESB — pay six figures for a UI, a runtime, a management API, and a deployment workflow they barely use.
3. Roll their own management plane — and re-invent metrics, hot-reload, leader election, and an admin API for the third time this decade.

Tsak is the missing piece between "a Worker Service" and "an enterprise ESB":

	Hand-rolled Worker	Tsak	Heavy ESB
Deploy a new pipeline	Redeploy whole process	cp module.dll Libs/ (hot-reload)	Vendor wizard, hours
Stop one pipeline without affecting others	Custom code	tsak context stop orders	Vendor UI
Distribute pipelines across nodes	Custom coordinator	Built-in leader election + auto-rebalance	Vendor cluster
REST API for ops/CI	Build it yourself	32 endpoints, typed client	Yes, vendor-locked
CLI for CI/CD	Build it yourself	30 commands, profiles, JSON output	Maybe
Web dashboard	Build it yourself	Blazor Server, 10 pages	Yes, vendor-locked
Cost	Engineering time	MIT, free	$$$$ + lock-in
Routing engine	DIY or MassTransit	redb.Route — 22 transports, 24 EIP processors, fluent DSL	Vendor's DSL

---

For redb.Route users

If you already write RouteBuilder classes for redb.Route, Tsak is what runs them in production. You do not change a single line of route code.

┌──────────────────────┐         ┌──────────────────────┐         ┌──────────────────────┐
│   Your routes        │  --->   │   class library      │  --->   │  Tsak picks it up    │
│   (RouteBuilder)     │ publish │   (Orders.dll)       │   cp    │  hot-reload + run    │
└──────────────────────┘         └──────────────────────┘         └──────────────────────┘

What Tsak adds on top of a plain redb.Route worker:

Pure redb.Route worker (Host.CreateApplicationBuilder)	redb.Route inside Tsak
One Program.cs wires every RouteBuilder at compile time.	Drop the DLL into Libs/ — Tsak loads it.
Add a route → redeploy the whole process.	Hot-reload one module without touching the others.
One process = one bag of routes.	Multiple named contexts, each with its own properties, lifecycle and assembly load context.
Stop one route = stop the process.	tsak route stop <ctx> <id> or POST /api/routes/{id}/stop.
Distribute across nodes = roll your own coordinator.	Built-in leader election + per-context assignment across cluster nodes.
Schedule a job = wire Quartz yourself in Program.cs.	IScheduler injected into every context, schema auto-bootstrapped.
Operate it = parse logs, build a dashboard, expose metrics yourself.	REST + CLI + Blazor + Prometheus + OTel out of the box.

A RouteBuilder you wrote against redb.Route's plain IHostedService runs unchanged inside a Tsak module — the same Configure() is called, the same IExchange flows through, the same OnException and .Transacted() semantics apply. Tsak only owns how the routes are loaded, scheduled, observed and torn down.

// MyRoutes/InitRoute.cs — the only Tsak-specific piece
public static class InitRoute
{
    public static IRouteContext main(IRouteContext context)
    {
        // Plain redb.Route — exactly what you would write outside Tsak
        ((RouteContext)context).AddRoutes(new OrderRoutes());
        ((RouteContext)context).AddRoutes(new ShipmentRoutes());
        return context;
    }
}

For the routing DSL itself (transports, EIP patterns, expressions, transactional pipelines, OnException, error handling), see the redb.Route README. Everything below in this document is about the container.

---

Architecture

Process layout

┌────────────────────────────────────────────────────────────────────┐
│                        redb.Tsak.Worker                            │
│                                                                    │
│  ┌──────────────────────────────────────────────────────────────┐ │
│  │  _system context (protected, cannot be stopped or removed)   │ │
│  │  ────────────────────────────────────────────────────────── │ │
│  │  HTTP listener  →  AuthMiddleware  →  ControllerDispatcher   │ │
│  │  /api/auth        Bearer / X-Api-Key   AuthController         │ │
│  │  /api/contexts    role check           ContextsController     │ │
│  │  /api/modules                          ModulesController      │ │
│  │  /api/cluster                          ClusterController      │ │
│  │  /api/system                           SystemController       │ │
│  │  /api/scheduler                        SchedulerController    │ │
│  │  /api/logs                             LogsController         │ │
│  │  /api/users                            UsersController        │ │
│  │  /api/watchdog  /api/diagnostics  /api/lifecycle  /api/routes │ │
│  └──────────────────────────────────────────────────────────────┘ │
│                                                                    │
│  ┌────────────────┐  ┌────────────────┐  ┌────────────────┐      │
│  │   Context A    │  │   Context B    │  │   Context C    │      │
│  │   "orders"     │  │   "payments"   │  │   "analytics"  │      │
│  │ ────────────── │  │ ────────────── │  │ ────────────── │      │
│  │ Module:        │  │ Module:        │  │ Modules:       │      │
│  │  Api.Orders    │  │  Pay.Stripe    │  │  Etl.Loader    │      │
│  │ Route 1: Kafka │  │ Route 1: HTTP  │  │  Etl.Reports   │      │
│  │ Route 2: Cron  │  │ Route 2: AMQP  │  │ Route 1..N     │      │
│  │ Route N: HTTP  │  │ Route N: Mail  │  │                │      │
│  │ ALC: isolated  │  │ ALC: isolated  │  │ ALC: isolated  │      │
│  └────────────────┘  └────────────────┘  └────────────────┘      │
│                                                                    │
│  ┌──────────────────────────────────────────────────────────────┐ │
│  │  Infrastructure services                                      │ │
│  │  ────────────────────────────────────────────────────────── │ │
│  │  HotReloadService   ClusterCoordinator    QuartzScheduler    │ │
│  │  ModuleRegistry     LeaderElection        WatchdogService    │ │
│  │  ContextManager     NodeRegistry          MetricsService     │ │
│  │  ConfigMerger       AssignmentManager     LogRingBuffer      │ │
│  │                                            HealthCheckService │ │
│  └──────────────────────────────────────────────────────────────┘ │
└────────────────────────────────────────────────────────────────────┘
                                  │
                                  ▼
                ┌─────────────────────────────────┐
                │  redb EAV (Postgres or MSSQL)   │
                │  - API keys (RedbApiKeyStore)   │
                │  - Cluster topology (Tree)      │
                │  - Lifecycle events             │
                │  - Idempotent state             │
                │  - Quartz QRTZ_* tables         │
                └─────────────────────────────────┘

Anatomy of a request

External call:  curl -H "Authorization: Bearer $KEY" http://node:9090/api/contexts/orders/start
       │
       ▼
HTTP listener (Kestrel) inside _system context — port 9090
       │
       ▼
HeaderBridge processor      → normalizes route.path = "contexts/orders/start", route.method = "POST"
       │
       ▼
AuthorizeProcessor          → resolves API key (5min cache) → validates role → enriches Exchange with claims
       │
       ▼
ControllerDispatcher        → looks up [Route("contexts/{name}/start")] on ContextsController
       │
       ▼
ContextsController.Start    → ITsakContextManager.StartAsync("orders")
       │
       ▼
                            → graceful start: load assemblies → wire routes → connect transports → mark Running
       │
       ▼
JSON response               → { "name": "orders", "status": "Running", "startedAt": "..." }

The same ContextsController can also be invoked over RabbitMQ RPC, gRPC, SignalR, or any other redb.Route InOut transport — the dispatcher is transport-agnostic. This is how Tsak avoids tying its management API to HTTP-only.

---

Project structure

redb.Tsak/
├── src/
│   ├── redb.Tsak.Core/          Kernel: contracts, controllers, services,
│   │                            security, cluster, hot-reload, monitoring
│   ├── redb.Tsak.Core.Pro/      Pro extensions (cluster, hooks, monitoring)
│   ├── redb.Tsak.Worker/        Hosted process: DI wiring, Serilog,
│   │                            Quartz, appsettings.json, Dockerfile
│   ├── redb.Tsak.Contracts/     Wire DTOs (System.Text.Json only) shared
│   │                            between Core, Client, Web, CLI
│   ├── redb.Tsak.Client/        ITsakApiClient + TsakApiClient (HTTP)
│   ├── redb.Tsak.CLI/           tsak — 30 commands, profiles, JSON output
│   ├── redb.Tsak.Web/           Blazor Server dashboard, custom design system
│   └── redb.Tsak.Web.Pro/       Pro web extensions (auth, node providers)
├── tests/
│   ├── redb.Tsak.Tests/         287 unit + integration tests
│   └── redb.Tsak.CLI.Tests/     64 CLI command tests
├── docs/
│   ├── PLAN.md                  Architecture reference
│   ├── CONFIG_GUIDE.md          5-layer configuration deep-dive
│   ├── LT_TSAK_ANALYSIS.md      Analysis of the original lt.tsak
│   ├── ENTERPRISE_AUDIT.md
│   └── phases/                  Per-phase implementation notes (0..8B)
├── README.md                    This file
├── CHANGELOG.md                 Per-version changes
├── CONTRIBUTING.md              Contribution guide
├── SECURITY.md                  Security policy
├── DEPLOYMENT_SECRETS.md        Secrets management for production
└── LICENSE                      MIT

---

Quick start

1. Run a Tsak worker (no database, in-memory)

cd redb.Tsak/src/redb.Tsak.Worker
dotnet run

[INF] redb.Tsak.Worker starting…
[INF] Storage: InMemory
[INF] Cluster: disabled
[INF] HotReload: enabled (scan every 10s)
[INF] HTTP API listening on http://0.0.0.0:9090
[INF] _system context started
[INF] Discovered 0 modules in Libs/
[INF] Ready

curl http://localhost:9090/api/system/health
# { "status": "Healthy", "checks": [...] }

2. Deploy your first module

A Tsak module is a plain .NET class library. Convention: a static InitRoute.main(IRouteContext) entry point, or one or more RouteBuilder subclasses.

// MyRoutes/InitRoute.cs
public static class InitRoute
{
    public static IRouteContext main(IRouteContext context)
    {
        ((RouteContext)context).AddRoutes(new OrderRoutes());
        return context;
    }
}

// MyRoutes/OrderRoutes.cs
public class OrderRoutes : RouteBuilder
{
    protected override void Configure()
    {
        From("kafka://orders?brokers=localhost:9092&groupId=svc")
            .Filter(e => e.Message.GetHeader<string>("type") == "new")
            .Process(async (e, ct) => await ProcessAsync(e, ct))
            .To("rabbitmq://processed?host=localhost");
    }
}

# Build and deploy
dotnet publish MyRoutes -c Release -o publish/
cp -r publish/* /opt/tsak/Libs/MyRoutes/

# Tsak picks it up within HotReload:ScanIntervalSeconds (default 10s)

tsak context list
# Name      Status   Routes  Endpoints  Modules
# _system   Running       11          1   (system)
# MyRoutes  Running        1          2   MyRoutes

3. Manage from the CLI

tsak login http://localhost:9090 --key $TSAK_KEY
tsak health
tsak context restart MyRoutes
tsak logs --level Error --limit 50
tsak metrics
tsak scheduler jobs
tsak cluster overview

4. Open the dashboard

cd redb.Tsak/src/redb.Tsak.Web
dotnet run
# Browse to http://localhost:5100

---

Module workflow

Tsak supports two equivalent deployment formats — pick whichever fits your workflow:

Format	What it is	When to use
Directory layout (Libs/{Module}/*.dll)	Loose .dll files alongside manifest.json and config	Local dev, fast iteration, atomic file replace
.tpkg package (Libs/{Module}.tpkg)	A single ZIP archive containing the manifest, DLLs, configs	CI/CD, atomic deploys, immutable artifacts, easy versioning

Both are scanned by HotReloadService. A .tpkg is unpacked into an isolated AssemblyLoadContext, then loaded the same way as a directory module.

Format 1 — Directory layout

Libs/
└── Orders/
    ├── Orders.dll                  ← compiled assembly
    ├── Orders.deps.json            ← dependency graph (from dotnet publish)
    ├── manifest.json               ← module metadata (Tsak)
    ├── context.json                ← infrastructure defaults (Layer 3)
    └── Orders.config.json          ← module business settings (Layer 4)

manifest.json

{
  "Name": "Orders",
  "Version": "2026.05.06.1430",
  "Description": "Order intake and validation pipeline",
  "Author": "Platform Team",
  "Dependencies": ["redb.Route", "redb.Route.Kafka", "redb.Route.RabbitMQ"]
}

context.json — infrastructure defaults (shipped with the module)

{
  "RabbitMQ": { "Host": "localhost", "Port": 5672, "Vhost": "/" },
  "Kafka":    { "Brokers": "localhost:9092", "GroupId": "orders-svc" },
  "Redis":    { "Connection": "localhost:6379", "Database": 0 }
}

Orders.config.json — module-shipped business settings

{
  "MaxRetries": 5,
  "BackoffSeconds": 30,
  "FeatureFlags": { "ValidateInventory": true, "EnrichCustomer": true }
}

How Tsak loads it

1. HotReloadService notices a new directory under Libs/.
2. ModuleAssemblyLoadContext (per-module isolated AssemblyLoadContext) loads Orders.dll.
3. ModuleLoader finds InitRoute.main() (or any RouteBuilder subclass).
4. ConfigMerger deep-merges the 5 layers into a single IDictionary<string, object?> and exposes it as context properties.
5. ContextManager creates an IRouteContext, registers Quartz IScheduler, calls InitRoute.main(context).
6. RouteBuilder.Configure() runs — routes are defined.
7. If AutoStart = true, the context starts: transports connect, consumers begin reading, the route is live.

Hot-swap

Replace the DLL on disk:

cp -r ./publish/* /opt/tsak/Libs/Orders/

Tsak detects the file timestamp change. With HotReload:RollingUpdate = true and Cluster:Enabled = true, nodes update one by one — there is never a moment when zero nodes are running the new version, but also never a moment when in-flight messages are dropped.

Configurable knobs:

Key	Default	Behavior
HotReload:ScanIntervalSeconds	10	How often to scan Libs/ for changes.
HotReload:KeepVersions	2	Old versions kept on disk for one-command rollback.
HotReload:StartupTimeoutSeconds	60	Wait time for the new version to settle before retiring the old one.
HotReload:RollingUpdate	true	In a cluster, update nodes sequentially.
HotReload:Collectible	false	Enable AssemblyLoadContext.Unload() for full GC reclamation. Do not enable if your modules use Reflection.Emit (XmlSerializer, source-gen serializers, compiled regex).
HotReload:RemovalDebounceScans	2	Number of scan cycles a missing DLL must persist before its module is unloaded — protects against false positives during atomic file replacement.

---

Format 2 — .tpkg package (CAR-style hot deploy)

A .tpkg is a plain ZIP archive that bundles the manifest, all DLLs, and per-module configs into a single file. Drop it into Libs/, Tsak picks it up atomically — one file, one deploy.

Anatomy

Orders.tpkg            (ZIP archive)
├── manifest.json
├── Orders.dll                  ← entry point
├── Orders.config.json          ← optional module config (Layer 4)
├── Orders.Models.dll           ← bundled dependency
├── Orders.Domain.dll           ← bundled dependency
└── Newtonsoft.Json.dll         ← any other transitive deps NOT provided by host

manifest.json for a .tpkg

{
  "Name": "Orders",
  "Version": "2026.05.06.1430",
  "EntryPoints": [ "Orders.dll" ],
  "Dependencies": [ "redb.Route.Kafka", "redb.Route.RabbitMQ" ]
}

EntryPoints lists only DLLs that contain InitRoute.main() or RouteBuilder subclasses. All other DLLs in the ZIP are loaded into the same isolated ALC as transitive dependencies — Tsak does not scan them for entry points. A single .tpkg may declare multiple entry points (e.g. Orders.dll + Orders.Api.dll), each one becomes its own module within the same ALC.

Why .tpkg

Benefit	Why it matters
Atomic deploy	One file appears or disappears — no partial-state window where some DLLs are updated and others aren't
Immutable artifact	Drop the same .tpkg into ten environments → byte-identical deployment
Easy versioning	Orders-2026.05.06.tpkg next to Orders-2026.05.05.tpkg — ship to artifactory, sign, attach SBOMs
Smaller diff	Replace one ZIP, not 50 DLLs — friendlier to file watchers and rsync
Atomic removal	Delete the .tpkg → all its modules unload together (no orphaned dependencies)
Host-DLL exclusion	Pack only what the host doesn't already provide — keeps packages small and avoids version conflicts

Building a .tpkg

Option A — MSBuild target in your .csproj

Embed packing into the build itself. Used by redb.Route.Demo:

<PropertyGroup>
  <CopyLocalLockFileAssemblies>true</CopyLocalLockFileAssemblies>
  <TsakModuleName>Orders</TsakModuleName>
  <TsakLibsDir>$(MSBuildThisFileDirectory)..\redb.Tsak\src\redb.Tsak.Worker\Libs</TsakLibsDir>
</PropertyGroup>

<Target Name="PackTpkg" AfterTargets="Build">
  <PropertyGroup>
    <_TpkgStaging>$(IntermediateOutputPath)tpkg</_TpkgStaging>
    <_TpkgFile>$(MSBuildThisFileDirectory)output\$(TsakModuleName).tpkg</_TpkgFile>
  </PropertyGroup>

  <RemoveDir Directories="$(_TpkgStaging)" />
  <MakeDir Directories="$(_TpkgStaging);$(MSBuildThisFileDirectory)output" />

  <Copy SourceFiles="$(MSBuildThisFileDirectory)manifest.json" DestinationFolder="$(_TpkgStaging)" />
  <Copy SourceFiles="$(TargetPath)" DestinationFolder="$(_TpkgStaging)" />
  <Copy SourceFiles="$(MSBuildThisFileDirectory)$(TsakModuleName).config.json"
        DestinationFolder="$(_TpkgStaging)"
        Condition="Exists('$(MSBuildThisFileDirectory)$(TsakModuleName).config.json')" />

  <ZipDirectory SourceDirectory="$(_TpkgStaging)"
                DestinationFile="$(_TpkgFile)" Overwrite="true" />

  <MakeDir Directories="$(TsakLibsDir)" Condition="!Exists('$(TsakLibsDir)')" />
  <Copy SourceFiles="$(_TpkgFile)" DestinationFolder="$(TsakLibsDir)" SkipUnchangedFiles="false" />
  <Touch Files="$(TsakLibsDir)\$(TsakModuleName).tpkg" />

  <Message Importance="high" Text="Packed $(TsakModuleName) → $(_TpkgFile)" />
</Target>

CopyLocalLockFileAssemblies=true is critical — without it, NuGet PackageReference assemblies are not copied to bin/ for class-library SDK projects, and your .tpkg will be missing third-party deps.

After this target, every dotnet build produces output/Orders.tpkg and copies it into Libs/. Tsak hot-reload picks it up within ScanIntervalSeconds. Reference: redb.Route.Demo.csproj.

Option B — Standalone PowerShell script

For multi-project packages where you want full control over which DLLs to include and which to exclude (host-provided assemblies, version conflicts, force-include patterns).

A complete reference script with host-DLL exclusion, force-include patterns (for binary-incompatible NuGet versions), and multi-entry-point packaging lives at redb.Identity/scripts/pack-tpkg.ps1. Minimal version (used by tsum):

param(
    [ValidateSet("Debug","Release")] [string]$Configuration = "Release"
)
$ErrorActionPreference = "Stop"

$tfm     = "net9.0"
$pkgName = "Orders"
$staging = Join-Path $env:TEMP "${pkgName}_staging"
$tpkg    = Join-Path $PSScriptRoot "output\$pkgName.tpkg"
$libs    = "..\redb.Tsak\src\redb.Tsak.Worker\Libs"

# Build
dotnet build .\Orders.csproj -c $Configuration --nologo
if ($LASTEXITCODE -ne 0) { exit 1 }

# Stage
if (Test-Path $staging) { Remove-Item -Recurse -Force $staging }
New-Item -ItemType Directory -Path $staging | Out-Null

Copy-Item .\manifest.json $staging
Copy-Item ".\bin\$Configuration\$tfm\Orders.dll" $staging          # entry point
Copy-Item ".\bin\$Configuration\$tfm\Orders.Models.dll" $staging   # dependency
Copy-Item .\Orders.config.json $staging -ErrorAction SilentlyContinue

# Zip
if (Test-Path $tpkg) { Remove-Item $tpkg -Force }
Add-Type -AssemblyName System.IO.Compression.FileSystem
[IO.Compression.ZipFile]::CreateFromDirectory($staging, $tpkg)

# Deploy
Copy-Item $tpkg $libs -Force
(Get-Item (Join-Path $libs "$pkgName.tpkg")).LastWriteTime = Get-Date
Remove-Item -Recurse -Force $staging

Write-Host "Packed $tpkg" -ForegroundColor Green

Reference scripts in this repo:

• redb.Identity/scripts/pack-tpkg.ps1 — production-grade: two-package split (Core + Http), automatic exclusion of host-provided DLLs, force-include patterns for binary-incompatible NuGet versions
• tsum_web-develop/tsum.deploy/pack-tsum-tpkg.ps1 — multi-entry-point: bundles tsum.Core + tsum.Api + shared dependencies into a single tsum.tpkg

Atomic reload semantics

When a .tpkg is replaced or removed:

1. HotReloadService detects the file timestamp change (or removal).
2. Silently unregisters every module that was loaded from this package — no transient "missing module" alerts.
3. Drains the old isolated ALC (graceful stop, in-flight messages complete).
4. (For replace) Loads the new .tpkg, registers the new modules, starts contexts.
5. (For remove) Cleans up the ALC; Collectible mode reclaims the memory immediately.

Multiple entry points inside a single .tpkg reload as one unit — you cannot get into a state where one entry point is on the old version and another on the new.

Excluding host-provided DLLs

Tsak Worker already loads dozens of assemblies (redb.Core, redb.Route, all transports it ships, Quartz, Serilog, etc.). Repacking those into every .tpkg wastes space and risks version conflicts when the module references a different version than the host.

The recommended pattern (see redb.Identity/scripts/pack-tpkg.ps1):

# Build the exclude set from the host's bin directory
$ExcludeSet = New-Object System.Collections.Generic.HashSet[string] (
    [System.StringComparer]::OrdinalIgnoreCase)

Get-ChildItem "$TsakWorkerBin\*.dll" | ForEach-Object {
    [void]$ExcludeSet.Add($_.Name)
}

# When copying DLLs into staging, skip anything in the exclude set
Get-ChildItem ".\bin\$Configuration\net9.0\*.dll" | Where-Object {
    -not $ExcludeSet.Contains($_.Name)
} | Copy-Item -Destination $staging

For binary-incompatible cases (e.g. host has Microsoft.IdentityModel.* 6.x but your module needs 8.x), use force-include patterns to override the exclusion:

$ForceIncludePatterns = @( 'Microsoft.IdentityModel.*.dll' )

if ($ForceIncludePatterns | Where-Object { $dll.Name -like $_ }) {
    Copy-Item $dll.FullName $staging   # always include, even if in exclude set
}

The module's isolated ALC then resolves these DLLs from its own directory before falling through to the host's default ALC.

The most powerful feature in Tsak's configuration model: deep merge across five layers, with predictable precedence.

Layer 1: Tsak:Contexts:default              ← base for all contexts (lowest priority)
Layer 2: Tsak:Contexts:{name}               ← named context overrides
Layer 3: Libs/{Module}/context.json         ← module infrastructure defaults
Layer 4: Libs/{Module}/{Module}.config.json ← module business settings
Layer 5: Tsak:Contexts:{name}:Override      ← DevOps final word (highest priority)

Later layers deep-merge over earlier layers. Nested objects merge recursively — they do not replace each other wholesale.

Worked example

// appsettings.json
{
  "Tsak": {
    "Contexts": {
      "default": {
        "AutoStart": true,
        "RabbitMQ": { "Host": "localhost", "Port": 5672 }
      },
      "orders": {
        "Modules": ["Orders"],
        "RabbitMQ": { "Host": "rabbit-orders.local" },
        "Override": {
          "RabbitMQ": { "Password": "from-secret-manager" }
        }
      }
    }
  }
}

// Libs/Orders/context.json
{ "RabbitMQ": { "Vhost": "/orders", "Username": "orders-svc" } }

// Libs/Orders/Orders.config.json
{ "MaxRetries": 10 }

Effective configuration for the orders context:

{
  "AutoStart": true,
  "MaxRetries": 10,
  "RabbitMQ": {
    "Host": "rabbit-orders.local",       // from Layer 2 (overrode Layer 1 "localhost")
    "Port": 5672,                         // from Layer 1 (preserved through merge)
    "Vhost": "/orders",                   // from Layer 3 (module ships default)
    "Username": "orders-svc",             // from Layer 3
    "Password": "from-secret-manager"     // from Layer 5 (DevOps wins)
  }
}

Hot-reload of config

Edit context.json or {Module}.config.json while Tsak is running. The hot-reload scan re-merges all five layers and restarts the affected context with the new properties. No worker restart required.

Named vs anonymous contexts

• Named — explicitly defined in Tsak:Contexts:{name} with a Modules array. Multiple modules share one IRouteContext.
• Anonymous — for any module not assigned to a named context. One module = one context, named after the module.

{
  "Tsak": { "Contexts": {
    "api": { "Modules": ["Api.Orders", "Api.Catalog"], "AutoStart": true }
  }}
}

Api.Orders and Api.Catalog share the api context (and its property bag). Any other module gets its own anonymous context.

Full reference: CONFIG_GUIDE.md.

---

REST API

32 endpoints organized into 12 controllers. Every endpoint speaks JSON. Auth is opt-in (Tsak:Auth:Enabled) — when enabled, all endpoints except /api/system/health require an API key.

Controllers

Group	Endpoints	Purpose
/api/auth	3	Create / list / revoke API keys
/api/users	5	User CRUD (Pro feature, EAV-backed)
/api/contexts	7	List / get / start / stop / restart / reset-route-states / remove
/api/routes	6	List / get / start / stop / force-stop / inflight per route
/api/modules	3	List / get / remove loaded modules
/api/cluster	4	Status / nodes / rebalance / remove-node
/api/system	4	Health / metrics / metrics-history / info
/api/scheduler	9	Status / jobs / running / start / standby / pause-job / resume-job / fire-job
/api/watchdog	2	State / alerts
/api/lifecycle	1	Recent lifecycle events (filtered)
/api/diagnostics	2	Dump (cluster-wide) / route-level diagnostics
/api/logs	3	Tail / list-files / download-file

Sample calls

# Authenticate
KEY="$(tsak auth keys create --name ci --roles admin --output json | jq -r .rawKey)"

# Health
curl -s http://localhost:9090/api/system/health | jq
# {
#   "status": "Healthy",
#   "checks": [
#     { "name": "Contexts", "status": "Healthy", "data": { "running": 4, "failed": 0 } },
#     { "name": "Memory", "status": "Healthy", "data": { "workingSetMB": 184 } },
#     { "name": "Cluster", "status": "Healthy", "data": { "leader": "node-1", "epoch": 7 } }
#   ]
# }

# Start a context
curl -s -X POST -H "Authorization: Bearer $KEY" \
  http://localhost:9090/api/contexts/orders/start | jq

# Live in-flight exchanges in a route (debugging stuck routes)
curl -s -H "Authorization: Bearer $KEY" \
  http://localhost:9090/api/contexts/orders/routes/route-1/inflight | jq

# Force-stop a hung route
curl -s -X POST -H "Authorization: Bearer $KEY" \
  http://localhost:9090/api/contexts/orders/routes/route-1/force-stop

# Recent lifecycle events
curl -s -H "Authorization: Bearer $KEY" \
  "http://localhost:9090/api/lifecycle?contextName=orders&limit=20" | jq

# Diagnostic dump (CPU profiles, GC info, thread states)
curl -s -H "Authorization: Bearer $KEY" \
  http://localhost:9090/api/diagnostics/dump > tsak-dump.json

Typed C# client

services.AddTsakClient(opts =>
{
    opts.BaseUrl = "http://tsak-prod:9090";
    opts.ApiKey = builder.Configuration["TsakKey"];
});

public class MyService(ITsakApiClient tsak)
{
    public async Task RestartAllAsync(CancellationToken ct)
    {
        var contexts = await tsak.ListContextsAsync(ct);
        foreach (var c in contexts.Where(c => c.Status == "Failed"))
            await tsak.RestartContextAsync(c.Name, ct);
    }
}

---

CLI

tsak is a single binary that talks to any Tsak worker. It supports multiple connection profiles, JSON output for CI, and rich tabular output for humans.

# Profile management
tsak login http://prod-1:9090 --key $PROD_KEY --profile prod
tsak login http://stg-1:9090 --key $STG_KEY  --profile staging
tsak profile use prod
tsak profile list

# Per-call override
tsak --server http://other:9090 --key $OTHER_KEY  context list

# Output format
tsak context list                  # tabular (default)
tsak context list --output json    # JSON
tsak context list --output plain   # raw lines (grep-friendly)

Command groups (30 commands)

Group	Commands
profile	login, logout, use, list
auth	auth keys list, auth keys create, auth keys revoke
context	context list, context get, context start, context stop, context restart, context reset-routes, context delete
route	route list, route get, route start, route stop, route force-stop, route inflight
module	module list, module get, module deploy, module delete
scheduler	scheduler status, scheduler jobs, scheduler running, scheduler start, scheduler standby, scheduler pause-job, scheduler resume-job, scheduler fire-job
cluster	cluster overview, cluster nodes, cluster rebalance, cluster remove-node
monitoring	health, metrics, metrics history, info, logs, logs files, logs download, lifecycle, diagnostics, route-diagnostics
watchdog	watchdog status, watchdog alerts
users	users list, users get, users create, users update, users delete

---

Web dashboard

A separate Blazor Server process (redb.Tsak.Web) — works in two modes:

Mode	Storage	Node discovery
Standalone	None	Static node list in appsettings.json
Cluster	Required (Postgres / MSSQL)	Discovered dynamically from EAV cluster topology

Pages

Page	Highlights
Dashboard	Cluster overview, status donut chart, sparkline metrics, node list grid (sortable, filterable, paginated).
Cluster	3-level topology tree (cluster → group → node), module assignment, per-node health, click-through to NodeDetail.
NodeDetail	5 tabs: Contexts · Scheduler · Modules · Monitoring · Logs. Live Chart.js graphs (CPU, GC, memory, threads), 10s auto-refresh.
Routes	All routes across all contexts. Status, message count, error rate, click-through to route detail.
RouteView	Per-route deep-dive: definition, current state, in-flight exchanges, recent diagnostics.
Endpoints	Consumer / producer endpoints per route.
Watchdog	Suspected and hung route alerts with manual stop/restart actions.
Logs	Searchable ring-buffer log viewer with level filter and tail mode.
Auth	API key management UI — create, revoke (with confirmation).
Login	Credential-based dashboard access (cluster mode = EAV users; standalone = config).

Custom design system

The dashboard uses a custom CSS design system — no Bootstrap, no MUI, no Tailwind. Built on CSS variables, supports dark/light theme, system fonts, inline SVG icons.

Reusable components: TsakGrid<T>, TsakChart, TsakCard, TsakBadge, TsakIcon, TsakConfirmDialog, TsakToast, TsakPageHeader, TsakErrorBoundary.

---

Cluster mode

Enable with Tsak:Cluster:Enabled = true and a Postgres/MSSQL connection string. Tsak handles the rest.

What you get

• Leader election — distributed lock in redb EAV with TTL and epoch fencing. A new leader is elected automatically when the current one dies or loses its lock.
• Node registry — each worker registers itself with periodic heartbeats. Dead nodes are evicted after DeadNodeTimeoutSeconds.
• Automatic context assignment — the leader distributes contexts across live nodes (currently round-robin; weighted strategies on the roadmap). Contexts are reassigned automatically when nodes join or leave.
• Rolling hot-reload — module updates roll across nodes in sequence, never updating multiple nodes concurrently.
• Cluster-wide diagnostics — tsak cluster overview aggregates state from every node.

Topology in EAV

Stored as a polymorphic 3-level tree using redb.Tree (so it shows up nicely in any redb-aware tool):

cluster:default                     ← scheme: _tsak_clusters
 └── group:default:default          ← scheme: _tsak_groups
      ├── node:default:worker-1     ← scheme: _tsak_nodes
      ├── node:default:worker-2     ← scheme: _tsak_nodes
      └── node:default:worker-3     ← scheme: _tsak_nodes

Cluster operations (assignment, leader change, rebalance) mutate this tree atomically. Every operation is fenced by the leader's epoch token — a stale leader cannot corrupt state after losing election.

Cluster configuration

{
  "Tsak": {
    "Cluster": {
      "Enabled": true,
      "ClusterName": "default",
      "GroupName": "default",
      "NodeId": "",
      "ApiEndpoint": "http://node-1.local:9090",
      "HeartbeatIntervalSeconds": 15,
      "DeadNodeTimeoutSeconds": 60,
      "LeaderLockTtlSeconds": 30,
      "Strategy": "round-robin"
    }
  }
}

Pluggable cluster backends

All cluster coordination is hidden behind interfaces in redb.Tsak.Core.Pro:

Interface	Default implementation (redb EAV)	Drop-in alternative
ILeaderElection	RedbLeaderElection (epoch-fenced lock in EAV)	KubernetesLeaderElection (Lease API), EtcdLeaderElection, ZK
IDistributedLock	RedbDistributedLock (TTL row in EAV)	RedisDistributedLock, KubernetesLeaseLock
INodeRegistry	RedbNodeRegistry (heartbeat rows)	KubernetesPodRegistry (label selector), Consul
IClusterCoordinator	ClusterCoordinator (background loop)	implementation owns the loop
IClusterBootstrap	RedbClusterBootstrap	bootstrap from K8s ConfigMap
IAssignmentManager	RoundRobinAssignmentManager	WeightedAssignmentManager, custom

Replace any of them with a single DI registration before AddTsakCluster(). Nothing else in the codebase changes — the ClusterCoordinator only talks to these interfaces.

builder.Services
    .AddSingleton<ILeaderElection, KubernetesLeaderElection>()  // override default
    .AddTsakCluster(builder.Configuration);                     // everything else stays

This is the design path for native Kubernetes integration without ever touching redb EAV for coordination — handy when the cluster runs against an external operational database that you do not want to use as a synchronization primitive.

---

Watchdog

WatchdogService continuously inspects every route and classifies it into one of three states:

                                 (no progress for SuspectedThresholdMinutes)
       ┌────────────┐                            │
       │  Healthy   │ ─────────────────────────────────────► ┌─────────────┐
       └────────────┘                                         │  Suspected  │
              ▲                                               └─────────────┘
              │                                                       │
              │ (progress resumes)            (no progress for HungThresholdMinutes)
              │                                                       │
              │                                                       ▼
              │                                                ┌─────────────┐
              └─────────────────────────────────────────────── │    Hung     │
                                                                └─────────────┘
                                                                       │
                                            (AutoRestartHungRoutes = true)
                                                                       │
                                                                       ▼
                                                       Force-stop + restart route

Configurable:

{
  "Tsak": {
    "Watchdog": {
      "Enabled": true,
      "CheckIntervalSeconds": 10,
      "SuspectedThresholdMinutes": 0.5,
      "HungThresholdMinutes": 1.5,
      "AutoRestartHungRoutes": false
    }
  }
}

Alerts surface in /api/watchdog/alerts, in the CLI (tsak watchdog alerts), and in the Web dashboard's Watchdog page.

---

Lifecycle & graceful shutdown

Tsak's startup and shutdown sequences are deterministic — LifecycleHookOrdering enforces the order so operators can rely on what is up before what depends on it, and what drains before what stops.

Startup order (Worker DI)

1. QuartzSchemaInitializer — applies Quartz DDL (Postgres / MSSQL) idempotently via raw ADO.NET. No redb dependency at this stage to avoid bootstrap deadlocks.
2. QuartzHostedService — starts the scheduler (RAMJobStore standalone, AdoJobStore cluster).
3. MetricsCollectionService — periodic process / GC sampling into the circular buffer.
4. TsakHostedService — main coordinator: shared assembly loader → module discovery → context start → cluster register.
5. RouteWatchdogService — hung-exchange detector (Pro).

Shutdown sequence (TsakHostedService.StopAsync)

1. Logs "Tsak graceful shutdown, stopping all contexts...".
2. Runs every ITsakLifecycleHook.OnStoppingAsync() in reverse registration order — last hook to start is first to stop. Cluster deregistration is a hook, so leadership is released before contexts begin draining.
3. For each context, calls ITsakContextManager.StopContextAsync(ctx, CancellationToken.None) — deliberately not the host shutdown token. One slow context cannot cancel the drain of the others. Each context owns its own context:graceful-stop-timeout for in-flight exchanges.
4. base.StopAsync() releases hosted-service resources.
5. Quartz scheduler stops, Serilog sinks flush.

Stop / restart REST responses include DrainTimeMs and InflightAfter — operators see exactly how long the drain took and whether anything was force-killed. SIGTERM handling is delegated to the standard .NET IHostApplicationLifetime.

Lifecycle hooks

public class MyHook : ITsakLifecycleHook
{
    public Task OnStartingAsync(CancellationToken ct)  => /* before module discovery */;
    public Task OnStartedAsync (CancellationToken ct)  => /* after all contexts running */;
    public Task OnStoppingAsync(CancellationToken ct)  => /* before context drain (reverse) */;
    public Task OnStoppedAsync (CancellationToken ct)  => /* after all contexts stopped */;
}

builder.Services.AddSingleton<ITsakLifecycleHook, MyHook>();

Every start / stop / error event is also persisted by LifecycleAuditService and queryable through /api/lifecycle.

---

Storage modes

Mode	API keys	Modules	Cluster	State	Use case
InMemory	ConfigApiKeyStore (read-only, from appsettings)	In-process registry	Not supported	Lost on restart	Dev, CI, embedded scenarios
Redb (Postgres)	RedbApiKeyStore (EAV, runtime CRUD)	Persistent	Supported	Survives restart	Single-node production, lightweight clusters
Redb (MSSQL)	RedbApiKeyStore (EAV, runtime CRUD)	Persistent	Supported	Survives restart	Single-node production, MSSQL shops

Switch modes with one config setting:

{ "Tsak": { "Storage": { "Type": "Redb" }, "Redb": { "Provider": "postgres" } } }

When UsePro = true, redb.Core.Pro is enabled — gives you EAV change tracking (faster writes), distributed locking primitives, and the cluster topology features.

---

Security

Layer	Mechanism
Wire	API Key in Authorization: Bearer or X-Api-Key header
Storage	SHA-256 hash, raw key never persisted
Comparison	CryptographicOperations.FixedTimeEquals (timing-attack safe)
Authorization	Per-endpoint role checks (admin, reader, custom)
Lifecycle	Optional expiry per key, runtime revocation, immediate cache invalidation
User binding	Optional UserId link — when the user is disabled or deleted, the key dies (via IUserProvider)
Cache	5-minute TTL key lookup cache, invalidated on revoke
Cluster trust	Inter-node calls use the same API key auth — no implicit trust between nodes
Protected resources	The _system context cannot be stopped or removed by any caller, including admins

Full policy: SECURITY.md. Production secrets handling: DEPLOYMENT_SECRETS.md.

---

Observability

Metrics

• MetricsService — circular buffer with CPU, working set, managed memory, threads, GC pressure. Default window: 12 hours × 10s sample = 4320 points.
• ContextMetricsCollector — per-context aggregation: messages/sec, error rate, in-flight count.
• Per-route metrics from redb.Route's .Metered() step.
• Cluster-wide periodic state report: ClusterReportIntervalSeconds.

OpenTelemetry & Prometheus

First-class OTel integration via the standard NuGet packages:

• OpenTelemetry.Extensions.Hosting
• OpenTelemetry.Exporter.Prometheus.HttpListener
• OpenTelemetry.Instrumentation.Process
• OpenTelemetry.Instrumentation.Runtime

When Tsak:Metrics:Prometheus:Enabled = true, Tsak calls AddOpenTelemetry().WithMetrics(b => b.AddPrometheusHttpListener(...)) and exposes a Prometheus-format endpoint on http://*:9464/metrics (port configurable via Tsak:Metrics:Prometheus:Port).

The OTel pipeline registers the redb.Route ActivitySource (RouteActivitySource.SourceName) so any tracing emitted by route processors is collected automatically — point an OTLP collector / Jaeger / Tempo at the activity source, or scrape :9464/metrics from Prometheus.

"Tsak": {
  "Metrics": {
    "Prometheus": { "Enabled": true, "Port": 9464 }
  }
}

Health probes — Kubernetes-ready

Distinct probes for the three K8s lifecycle phases. All three are auth-exempt by default (Tsak:Api:AuthExempt):

Endpoint	Probe type	Returns 200 when
GET /api/system/health/startup	startup	process is up
GET /api/system/health/live	liveness	process is not deadlocked
GET /api/system/health/ready	readiness	contexts are running and healthy (stricter than liveness)

HealthCheckService aggregates probes (worst status wins: Unhealthy > Degraded > Healthy) and never throws — exceptions inside a probe become Unhealthy, never a 500. Modules can contribute custom probes by implementing IModuleHealthContributor. Pro ships ClusterHealthContributor reporting leader / member health.

Logs

• LogRingBuffer — Serilog in-memory sink, default 2000 entries, queryable via REST and Web UI.
• File logs via Serilog (rolling files, configurable path).
• MemoryUsageEnricher — every log event carries {MemoryUsage} for cheap memory pressure correlation.

Tracing

• OpenTelemetry traces via redb.Route's .Traced() step — per-route, per-step spans.
• Standard OTLP exporter (configurable in appsettings.json).

Diagnostics

• Per-route diagnostic dumps via /api/contexts/{ctx}/routes/{id}/diagnostics.
• Cluster-wide dumps via /api/diagnostics/dump.
• Lifecycle events feed (route start/stop/restart, hot-swap, cluster reassignment, watchdog alerts).
• In-flight exchange tracking — see exactly which messages are sitting where right now.

---

Quartz scheduler

Every Tsak context gets an IScheduler injected for free. Modules use it via redb.Route.Quartz:

From("cron:0 */5 * * * ?")           // every 5 minutes
    .Setbody(() => DateTime.UtcNow)
    .To("rabbitmq://heartbeats");

Mode	Storage	Cluster-safe	Use
RAMJobStore	In-memory	No	Standalone, dev
AdoJobStore	Postgres / MSSQL QRTZ_* tables	Yes	Production

Schema is auto-created on first start by QuartzSchemaInitializer — embedded SQL scripts for both Postgres and MSSQL, idempotent, runs before Quartz initializes its own connection pool. No DBA action required.

REST endpoints under /api/scheduler cover: status, listing scheduled jobs, listing currently-running jobs, start/standby, pause/resume/fire-now per job key.

---

Three deployment recipes

Recipe A — Standalone, single binary

For development, demos, embedded use.

{
  "Tsak": {
    "Storage": { "Type": "InMemory" },
    "Cluster": { "Enabled": false },
    "Auth":    { "Enabled": false }
  }
}

Just dotnet run. No DB, no auth, no cluster. Fastest path to running a Tsak module locally.

Recipe B — Single node + redb (Postgres)

For production single-node deployments where state must survive restarts.

{
  "ConnectionStrings": { "Postgres": "Host=db;Database=redb_tsak;Username=tsak;Password=$$" },
  "Tsak": {
    "Storage": { "Type": "Redb" },
    "Redb":    { "Provider": "postgres", "UsePro": true, "License": "$$" },
    "Cluster": { "Enabled": false },
    "Auth":    { "Enabled": true, "Secret": "$$" }
  }
}

API keys persist. Quartz jobs persist. Lifecycle events persist. Module assignments persist. One node, durable state.

Recipe C — Cluster

For HA and horizontal scaling. Same appsettings on every node, only NodeId and ApiEndpoint differ:

{
  "ConnectionStrings": { "Postgres": "Host=db.cluster;Database=redb_tsak;Username=tsak;Password=$$" },
  "Tsak": {
    "Storage": { "Type": "Redb" },
    "Redb":    { "Provider": "postgres", "UsePro": true },
    "Cluster": {
      "Enabled": true,
      "ClusterName": "production",
      "GroupName": "default",
      "ApiEndpoint": "http://node-1.local:9090",
      "Strategy": "round-robin"
    },
    "HotReload": { "RollingUpdate": true },
    "Auth":     { "Enabled": true, "Secret": "$$" }
  }
}

Start three workers. They'll discover each other through the shared EAV store, elect a leader, distribute your modules, and roll updates without downtime.

---

Docker

A multi-stage Dockerfile is included with the worker. Final image is based on mcr.microsoft.com/dotnet/runtime:9.0 — slim, no SDK, no app host. Two ports are typically exposed: 9090 (REST API + UI) and 9464 (Prometheus, when enabled).

# From the repository root
docker build -f redb.Tsak/src/redb.Tsak.Worker/Dockerfile -t redb-tsak:1.0.4 .
docker run -d --name tsak \
  -p 9090:9090 -p 9464:9464 \
  -v /opt/tsak/Libs:/app/Libs \
  -e ConnectionStrings__Postgres="$PG_CONN" \
  -e Tsak__Auth__Secret="$TSAK_SECRET" \
  -e Tsak__Metrics__Prometheus__Enabled=true \
  redb-tsak:1.0.4

Mount Libs/ from the host (or from a shared volume) so module updates can be deployed without rebuilding the image. appsettings.json overrides flow through environment variables in the standard ASP.NET pattern (__ for nesting).

---

Kubernetes deployment

Tsak is built from the ground up for container deployment — distinct K8s probes, OTel/Prometheus exporter, env-var configuration, and pluggable cluster backends are all in the box.

Probes

Map the three health endpoints to the matching K8s probe types:

startupProbe:
  httpGet: { path: /api/system/health/startup, port: 9090 }
  failureThreshold: 30
  periodSeconds: 4           # up to 120s to boot

livenessProbe:
  httpGet: { path: /api/system/health/live, port: 9090 }
  periodSeconds: 10
  failureThreshold: 3        # 30s before pod restart

readinessProbe:
  httpGet: { path: /api/system/health/ready, port: 9090 }
  periodSeconds: 5
  failureThreshold: 2        # 10s before removal from load balancer

Liveness intentionally does not check module health — that avoids restart loops during rolling updates. Readiness is stricter: any context in a non-running state → pod is removed from service endpoints (no restart), allowing the cluster to rebalance via IAssignmentManager.

Graceful termination

Kubernetes sends SIGTERM, then SIGKILL after terminationGracePeriodSeconds. Set Tsak:Shutdown:TimeoutSeconds to terminationGracePeriodSeconds − 5 so cluster deregistration has its buffer. The shutdown sequence then becomes:

SIGTERM → cluster deregister (lifecycle hook, reverse-order) →
context drain (isolated CT, up to TimeoutSeconds each) →
Quartz shutdown → log flush → SIGKILL never needed

spec:
  terminationGracePeriodSeconds: 60
  containers:
    - name: tsak
      env:
        - name: Tsak__Shutdown__TimeoutSeconds
          value: "55"

Pod identity in cluster mode

StableNode IDs across pod restarts matter for assignment continuity. Inject pod metadata via the downward API and bind NodeId to metadata.name:

env:
  - name: POD_NAME
    valueFrom: { fieldRef: { fieldPath: metadata.name } }
  - name: POD_NAMESPACE
    valueFrom: { fieldRef: { fieldPath: metadata.namespace } }
  - name: POD_IP
    valueFrom: { fieldRef: { fieldPath: status.podIP } }
  - name: Tsak__Cluster__NodeId
    valueFrom: { fieldRef: { fieldPath: metadata.name } }
  - name: Tsak__Cluster__ApiEndpoint
    value: http://$(POD_IP):9090

Use a StatefulSet (or a Deployment + headless service) for predictable pod names. Tsak treats Tsak:Cluster:NodeId as the cluster identity — mapping it to metadata.name keeps assignment stable across restarts.

Prometheus scraping

metadata:
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port:   "9464"
    prometheus.io/path:   "/metrics"

Set Tsak:Metrics:Prometheus:Enabled = true to activate the OTel Prometheus exporter on port 9464.

Native cluster integration

All coordination interfaces (ILeaderElection, IDistributedLock, INodeRegistry, IClusterCoordinator, IClusterBootstrap, IAssignmentManager) are pluggable — see Pluggable cluster backends. A Kubernetes-native Lease implementation (KubernetesLeaderElection) can be dropped in without touching redb EAV for coordination, leaving redb only for module / lifecycle / API key state.

---

Testing

Suite	Count	What's covered
redb.Tsak.Tests	287	Module loading, context lifecycle, config merge, security, controllers, monitoring, scheduler, cluster (leader election, heartbeat, rebalance, epoch fencing), hot-reload (ALC isolation, rolling update, rollback, removal debounce), in-memory and Redb stores, host startup, full API integration
redb.Tsak.CLI.Tests	64	All 30 CLI commands — output format (table/JSON/plain), error handling, auth failure, profile resolution
Total	351	All passing on net8.0, net9.0, net10.0

dotnet test redb.Tsak/tests/redb.Tsak.Tests
dotnet test redb.Tsak/tests/redb.Tsak.CLI.Tests

---

Implementation status

All 9 phases are complete and merged. See STATUS.md for the per-phase breakdown.

#	Phase	Status	Tests
0	Infrastructure	Done	1
1	Controller Dispatcher (transport-agnostic)	Done	20
2	Container Core (modules, contexts, coordinator)	Done	32
3	Cluster (leader election, registry, assignment)	Done	9
4	Hot Reload (collectible ALC, rolling update, rollback)	Done	25
5	Monitoring (metrics, health, logs, watchdog)	Done	50
6	REST API & Auth (12 controllers, EAV key store)	Done	42
7	Quartz Scheduler (DI, schema initializer, controller)	Done	30
8A	CLI (30 commands, profiles, JSON)	Done	64
8B	Web UI (Blazor Server, 10 pages, design system)	Done	—

Total: 351 tests passing.

---

Frequently asked questions

Is Tsak a routing engine? No. Routing is redb.Route. Tsak is the runtime container that hosts route contexts, manages their lifecycle, and exposes a management surface.

Can Tsak load any .NET DLL? Tsak loads class libraries that follow its module convention: a static InitRoute.main(IRouteContext) method, or one or more public RouteBuilder subclasses. Random DLLs are ignored.

Is module isolation a sandbox? No. It is a dependency isolation boundary (separate AssemblyLoadContext per module). A malicious module runs with the same OS privileges as the Tsak process. Restrict write access to Libs/ accordingly. See SECURITY.md.

Why is Collectible: false the default for hot-reload? Because Reflection.Emit-based code (XmlSerializer, source-gen serializers, compiled regex, MEF) crashes inside collectible AssemblyLoadContexts. The .NET runtime forbids non-collectible dynamic assemblies (generated by Emit) from referencing types in a collectible ALC. Default is the safe choice; opt in only when you know your modules are Emit-free.

What happens to in-flight messages during hot-swap? The old context drains (existing exchanges complete). The new context starts in parallel. Once the new context is healthy and StartupTimeoutSeconds elapses, the old context is retired. Zero message loss for transports that support graceful shutdown.

Can I write my own assignment strategy? Yes — IAssignmentManager is the extensibility point. The round-robin strategy is the only one shipped today; weighted strategies are on the roadmap.

Does Tsak support multi-region clusters? Out of the box, no. The cluster coordination assumes low-latency access to the shared EAV database. For multi-region, run one cluster per region and federate above the Tsak layer (e.g. via redb.Route.RabbitMQ shovels).

Why a custom design system instead of Bootstrap or MUI? The dashboard is small and focused. Custom CSS keeps the bundle tiny, eliminates a major source of UI churn (vendor breaking changes), and gives full control over theming. CSS variables enable dark/light themes with no JS.

Is there an OpenAPI spec? Not yet. The 12 controllers are documented in STATUS.md and exposed via the typed ITsakApiClient. OpenAPI / Swagger generation is on the roadmap.

---

Roadmap

• OpenAPI / Swagger generation for the REST API
• Weighted cluster assignment strategies (CPU + memory composite)
• Batch operations (POST /api/contexts/batch/start)
• Webhook subscriptions for lifecycle events (push instead of poll)
• Live config editor in the dashboard
• Multi-region federation primitives

See docs/ for design notes on each.

---

Part of

• redb.Route — ESB & EIP framework for .NET (the routing engine Tsak hosts)
• redb.Core / redb.Core.Pro — EAV storage backend (the persistence layer Tsak uses)
• RedBase — full ecosystem

---

License

Apache License 2.0 — see LICENSE.

Pro components (redb.Tsak.Core.Pro, redb.Tsak.Web.Pro) require a commercial license — see LICENSE-PRO.txt.

Part of the RedBase ecosystem.

Copyright © 2024–2026 RedBase.