Juice.Observability.Rules 1.1.0

Juice Observability Platform

Turns telemetry from Juice services into a resource model, health, rules, alerts and investigation workflows.

Build and test

Needs the .NET 10 SDK. Integration tests need the local stack running with --profile test and are skipped otherwise.

deploy/local/scripts/init.sh && (cd deploy/local && docker compose --profile test up -d --wait)
dotnet test
deploy/local/scripts/run-api.sh          # API on http://localhost:5180, migrates and provisions the tenant
(cd console && npm ci && npm start)      # Console on http://localhost:4200, proxying /api to :5180

docker compose --profile platform up -d --build serves the Console and the API together on http://localhost:5180. console/e2e/journey.mjs signs in through Keycloak and walks the investigation journey (Console requirements §21) in a headless browser.

Build output goes to build/. Add a migration with dotnet tool restore && dotnet ef migrations add <Name> --project src/Juice.Observability.Data --output-dir Migrations; MigrationTests fails when the model changes without one.

Packages and CI

Besides the image, the platform ships as libraries for a host app of the customer's own:

src/Juice.Observability.Host and console/src are the reference host and app: they use the packages the way a customer would.

var builder = WebApplication.CreateBuilder(args);
builder.Configuration.AddJuiceObservabilityFiles();
builder.Services.AddJuiceObservability(builder.Configuration);
var app = builder.Build();
app.UseCorrelationId();
app.UseAuthentication();
app.UseAuthorization();
app.MapJuiceObservability();
app.Run();

Publishing (.github/workflows): the backend and the console library are versioned and released independently.

Each next version is the one to release: raise it after a release so alphas sort above it. Tags have no v.

The console library calls the API over HTTP, so record which versions work together:

API

All routes need a bearer token from the deployment's OIDC provider, with audience juice-platform and a platform role (viewer, operator or admin) in the roles claim. Errors are RFC 9457 problem details with correlationId and traceId; send X-Correlation-Id to set your own. Lists take limit (1–200, default 50) and cursor (the previous page's nextCursor).

A resource's detail also carries its health per environment and each instance's state. With Integrations:Grafana:BaseUrl set, a service and an alert also carry a link to the optional Grafana overview; without it, links is empty. Telemetry queries are platform-owned templates, tenant-filtered and bounded; services, instances, resources and dependencies are named by platform id, never by telemetry names from the caller.

{tenantId} must be the caller's tenant; any other id, existing or not, gets the same 403. /health/live and /health/ready are anonymous.

A local token, for example for victor (viewer):

source deploy/local/.env
curl -s -d grant_type=password -d client_id=juice-platform-cli -d username=victor \
  -d "password=$KEYCLOAK_DEMO_PASSWORD" http://localhost:8080/realms/juice/protocol/openid-connect/token

juice-platform-cli exists only in the local realm, for development and tests.

Discovery

Every 30 s the worker reads telemetry from its checkpoint to one minute ago, in windows of at most 15 minutes, as platform_reader. Every query filters on the Collector-stamped tenant. It reconciles each window and moves the checkpoint in one transaction.

Each window also re-reads the 10 minutes before the checkpoint (Discovery:Overlap). Telemetry is stored by its own time, so a row can land after its window was read: a long span is written when it ends but timed by its start, and the Collector holds data back while ClickHouse is slow. Upserts make the re-read harmless. A dependency's span count adds only spans after the checkpoint, unless the dependency is first seen in the overlap. Telemetry later than one minute plus the overlap is still missed.

• Services are keyed by namespace and name, across environments. Each namespace is one application.

• Instances are keyed by service.instance.id, or by host and process id (marked derived). An instance goes inactive after 15 minutes without telemetry and is deleted 30 days later.

• Resources:

  Resource	Key
  Host	host.id, else host.name
  Container	container.id
  Pod	k8s.pod.uid
  Messaging destination	system + destination
  Database	system + server + namespace

• Dependencies:

  • A client or producer span whose child server or consumer span is in another service gives a dependency between the two services, confidence observed.
  • A span naming a destination or a database gives a dependency on that resource, confidence inferred, since the resource itself never reports.

• Renamed services map through Discovery:Aliases.

• Limits. A window that fills Discovery:RowLimit rows is split, down to 10 s. Values are cut to 256 characters and attributes to 64. Both are counted in juice.observability.discovery.truncated.

Health

Every 30 s the health worker reads a 5-minute evidence window ending 10 s ago. It evaluates:

• each active instance;
• each service in every environment it was seen in;
• each application per environment;
• each host.

It writes one current snapshot per subject, and a transition only when the state changes.

• Instances. The error rate of server and consumer spans: 1 % is Degraded and 5 % is Unhealthy, counted only once there are at least 3 requests. With less traffic, recent telemetry alone is a heartbeat.

• Services:

  • a failed critical dependency (Health:CriticalDependencies) makes the service Critical;
  • if half or more of its fresh instances are unhealthy, it is Unhealthy;
  • any partial failure, or an elevated service error rate, makes it Degraded.

  Instances that went silent after a restart don't count.

• Applications. Children are weighted, optionally critical (Health:ServiceWeights), with a minimum coverage of 50 %. Missing optional services never make an application unhealthy.

• Reasons are stable codes, such as partial_instance_failure or critical_dependency_failed:shop/billing.

• Policy version. Each snapshot and transition carries the policy version, a hash of the rule settings.

• Replays. A transition is unique per subject, evidence window and policy version, so re-evaluating the same evidence never adds one.

• ClickHouse unavailable. The pass fails and the states are kept. The API marks a state stale once it is older than three intervals.

• Freshness. A new failure is visible after about the evidence lag (10 s) plus ingestion; tested at 10–11 s with a 1 s evaluation loop. With the 30 s interval the bound is about 42 s, against the 60 s target. juice.observability.health.evidence_age records it in operation.

Rules and alerts

Rules are deployment configuration. Rules:File names a JSON file (deploy/local/rules.json) that is validated and loaded at startup:

• an invalid rule stops the host;
• each create, update or disable writes an audit record (actor deployment-config);
• the API only reads rules.

Each rule is scoped by Service and Environment (or Host). It also has a schedule (default 1 min), a window, a threshold and a recovery threshold (hysteresis), minimum samples, ForDuration, RecoveryDuration and Cooldown.

• Lifecycle. A breach opens a Pending alert. It becomes Firing once the breach has held for ForDuration, and Resolved after RecoveryDuration at or below the recovery threshold.
  • A Pending alert that stops breaching resolves without ever firing, so a transient breach never fires.
  • Values between the two thresholds keep a Firing alert firing.
  • After an alert that fired resolves, a new one can't fire until Cooldown has passed.
• Deduplication. There is one open alert per rule and subject, enforced by a unique index.
• No data. Too few samples, or no rows, means no data. It is recorded on the alert and holds the alert's state; it is never read as zero.
• Scheduler. It runs 4 evaluations at once, each cut off after 20 s. A rule still running when due again is skipped, never stacked.
  • Metrics: juice.observability.rules.evaluations, .duration, .lag, .skipped and juice.observability.alerts.transitions.
• Breach-to-alert. Firing comes at most lag (10 s) plus one schedule after ForDuration: 70 s for a 1-minute rule, against the 2-minute target. Measured in tests at 22 s with a 5 s schedule and a 10 s ForDuration.

Webhooks

When an alert fires, or resolves after firing, one delivery per matching endpoint is written in the same transaction as the alert change. A webhook can therefore never roll an alert back, and replaying an evaluation never notifies twice. The dispatcher sends every 2 s.

• Signing. HMAC-SHA256, Standard Webhooks form: webhook-id, webhook-timestamp, webhook-signature: v1,<base64> over {id}.{timestamp}.{body}.
• Destinations. HTTPS only, no redirects. DNS is checked on every connection: one internal answer blocks the destination. Private addresses need an allowlist entry. Loopback, link-local and cloud metadata are never reachable.
• Retries. Transient failures retry with exponential backoff and jitter (10 s to 10 min, 8 attempts). Other 4xx fail at once. Every attempt is recorded.

Configuration, secrets and receiver guidance are in docs/deployment.md.

License

MIT