Egil.Orleans.Messaging.Streams.EventHubs 0.1.80

• net10.0

  • Egil.Orleans.Messaging (>= 0.1.80)
  • Microsoft.Orleans.Streaming.EventHubs (>= 10.1.0)

BREAKING CHANGES:
- prevent grain-derived stream ID collisions
 Derive namespace-based explicit subscription IDs from the complete Orleans GrainId and expose the same helper to producers. Reject textual identities that do not round-trip instead of silently collapsing them.
 Cover compound keys, hex-parseable strings, cross-kind collisions, Orleans binary serialization, and StreamId text parsing. Document producer wiring and migration.
- split provider-specific messaging packages
 Move Event Hubs stream enrichment into Egil.Orleans.Messaging.Streams.EventHubs so the core package no longer carries the Event Hubs dependency.
 Add Egil.Orleans.Messaging.State.AzureStorage with Azure Table/Blob storage-aware state manager registration and failure classification for optimistic concurrency failures.
 Rename WriteFailureKind to StorageFailureKind so write and clear recovery can share the same provider classification model.
 Update CI to pack all messaging packages and document the companion package model.
- make stream resume tokens configurable
 Make MessageTracker fully optional for StreamManager registration and let each implicit or explicit subscription opt out of passing the tracked resume token to Orleans.
 Remove the StreamCursor constructor that accepted StreamId so cursors are built from the stream namespace directly. Add tests and docs for the resume-token policy and include the outbox postman API plan.
- split messaging tools by capability
 Move messaging tools into capability namespaces and matching test folders, with Outboxes used to avoid a namespace/type name collision.
 Replace StreamManager's AddSubscription/SubscribeAsync surface with explicit implicit/explicit configuration APIs and IImplicitStreamGrain forwarding through an attached grain component.
New Features:
- coordinate outbox postman processing
 Refactor outbox processing into dedicated dispatch, postman registry, and reconciliation collaborators.
 Coordinate manual and timer-backed drains through a single active drain so callers wait for the current pass before processing the next pending snapshot. Preserve per-postman ordering while allowing different postmen to dispatch concurrently.
 Remove the prototype OutboxProcessor2 implementation from the compiled package.
- preserve stream token json types
 Add an explicit JsonConverter registry for concrete StreamSequenceToken types and route StreamCursor and MessageTracker token payloads through that registry.
 Register Event Hubs token converters from the enriched adapter setup so enriched checkpoints round-trip without downcasting provider metadata.
- post outbox items concurrently
 Dispatch pending outbox items concurrently on the Orleans activation scheduler and remove the thread-pool execution mode.
 Background posting now allows delivery to interleave by default while durable acknowledgement and failure reconciliation remain non-interleaving unless explicitly opted in.
- classify azure storage failures
 Classify deterministic Azure Storage failures as non-persistence outcomes and keep ambiguous transient results on the read-back recovery path.
 This lets StateManager skip unnecessary recovery reads for rejected mutations while preserving correctness for throttling, timeouts, and service failures.
- add outbox postman helper APIs
 Add built-in stream and grain postman helpers so common Orleans fan-out targets can use the outbox processor without hand-written dispatch callbacks.
 Cover helper behavior through Orleans tests and fill validation coverage for keyed postman registration edge cases.
- add keyed outbox postman services
 Add IPostman<TMessage>, named postman attributes, IServiceCollection registration helpers, and an OutboxProcessor.AddPostman overload that resolves keyed postman services from the grain activation service provider.
 Cover keyed registration, scoped multi-contract resolution, successful keyed dispatch, and keyed dispatch failure reconciliation.
- add stream token tracker overloads
 Allow MessageTracker callers to process stream sequence tokens directly with explicit stream namespace or provider-qualified namespace, and expose latest stream sequence token lookup for resume-token use cases.
 Clarify stream manager tests by using distinct provider-name and namespace constants for explicit subscriptions.
- make stream tracking optional
 Allow grains to register StreamManager without providing a MessageTracker when they do not persist stream high-water marks.
 The tracked overloads remain source-compatible and still supply resume tokens when a tracker snapshot is provided.
- implement outbox processor stream flow
 Implements the outbox processor registration and draining path, including postman dispatch, success acknowledgement, failure reconciliation, retry timer/reminder scheduling, and stream-backed integration coverage.
 Adds outbox and stream receive telemetry plus focused Orleans in-process tests for stream delivery, error paths, resume behavior, JSON converter branches, and tracker semantics.
- implement stream manager subscriptions
 StreamManager now separates subscription configuration from activation-time subscription setup via RegisterStreamManager, AddSubscription, and SubscribeAsync. This lets grains await Orleans stream subscription establishment without blocking activation and keeps subscription failures visible.
 MessageTracker now records receive-lag histograms for accepted enriched stream cursors and outbox tokens, with shared telemetry plumbing kept in MessagingTelemetry. The Messaging test suite adds an in-process Orleans cluster with memory streams covering ValueTask, Task, default error handling, and resume-token behavior.
- add keyed state manager registration and remove write policy
 Add InitializeStateManager(...) that resolves keyed IStateManagerFactory<T> by storage name, with IServiceCollection and ISiloBuilder helper APIs for keyed singleton registrations.
 Refactor state management to StateManagerBase<T>/DefaultStateManager<T>, remove WritePolicy and force-write behavior, and drop reflection-based ETag mutation.
 Update state manager tests and API design docs to match the new factory wiring and write semantics.
- implement enriched event hub adapter token stamping
 Implement EnrichedEventHubAdapter overrides to stamp traceparent on queued messages, create EnrichedEventHubSequenceToken from cached Event Hub messages, and return enriched stream positions that preserve offset/sequence/event index and optional traceparent.
 Add focused tests for traceparent stamping behavior and enriched token construction from cached message metadata.
- wire enriched event hub data adapter registration
 Implement UseEnrichedDataAdapter extension to validate input and register EnrichedEventHubAdapter through the Event Hub configurator using Orleans Serializer from DI.
 Add focused tests for null-argument validation and configurator callback registration.
- implement StreamCursor enriched token accessors
 Enable StreamCursor JSON converter attribute and implement TryGetEnqueuedTime, TryGetStreamProviderName, and TryGetTraceParent to expose enrichment metadata when backed by EnrichedEventHubSequenceToken.
 Add focused tests for positive and negative cases across enqueued time, stream provider name, and traceparent access.
- implement StateManager extension wrapper
 Implement AsStateManager<T> to validate input and wrap IPersistentState<T> with StateManager<T> for grain-facing usage.
 Add focused tests for successful wrapping and null-argument validation.
- implement StateManager persistence workflow
 Implement StateManager<T> state cache, read/write/clear operations, force-write ETag override, and conflict handling that reconciles equivalent persisted state before rethrowing non-equivalent failures.
 Add focused tests for read refresh, successful writes, write-failure reconciliation behavior, rollback on conflicting writes, rollback when post-failure read fails, and VersionedState version stamping.
- implement OutboxMessageEnvelope STJ converter factory
 Enable JSON converter factory attribute on OutboxMessageEnvelope<T> and implement OutboxMessageEnvelopeJsonConverterFactory with closed generic converters for read/write round-trip serialization.
 Add tests for attribute wiring, CanConvert behavior, round-trip serialization, and missing-token payload validation.
- implement OutboxSequenceToken STJ converter
 Enable JSON converter attribute on OutboxSequenceToken and implement OutboxSequenceTokenJsonConverter read/write paths. The converter now round-trips sequence, sender GrainId shape, timestamp, and epoch without requiring caller-side serializer options.
 Add focused tests for converter attribute registration, round-trip behavior (including multiple GrainId key shapes), and payload validation for missing sender.
- MessageTracker
- Outbox<T> implemented
- stub types with XML docs
Bug Fixes:
- deliver enriched Event Hub tokens
 Wrap Event Hubs batch containers so stream consumers receive broker enqueue time, provider name, and trace context on batch and per-event sequence tokens. Cover the cached-message and Orleans serialization pipeline deterministically.
- preserve clear concurrency conflicts
 Keep InconsistentStateException observable when clear recovery finds a missing record, while adopting the provider's recovered state. Document the throwing-path state contract and cover it with a deterministic regression test.
- expose provider state as nullable
 Annotate IStateManager<T>.State for storage providers that can expose null after reads or clears, and handle null recovery values without dereferencing them. Preserve Orleans activated defaults while documenting and testing both nullable and non-null missing-record shapes.
- decouple Azure storage provider
 Keep only Azure.Core as a runtime dependency of the Azure Storage state-manager companion. Move Table and Blob SDK references to tests and leave Orleans provider selection to consumers, avoiding unwanted provider and version coupling.
- stabilize message tracker JSON schema
 Pin every private tracker model property to its canonical wire name so serializer naming policies cannot change persisted state. Require both root collections so mismatched or malformed payloads fail instead of silently clearing deduplication state.
- tolerate additive stream token JSON
 Read cursor, discriminator-envelope, built-in token, and Event Hub token properties by name while ignoring unknown fields. Required fields, known value types, and duplicate properties remain strict so rolling upgrades gain forward tolerance without accepting ambiguous payloads.
- preserve concurrent state after write failures
 Adopt durable state after a successful recovery read before rethrowing mismatches and concurrency conflicts. This keeps recovered state paired with the refreshed ETag so retries cannot overwrite concurrent writes.
- retry inherited reminder cleanup
 Retain a reminder handle loaded from a previous activation before unregistering it, so a transient unregister failure can be retried by the next empty outbox drain.
- reject duplicate outbox processors
 Fail immediately when a grain activation attempts to register a second processor, preserving the original reminder component and its durable retry path. Document the one-processor invariant and cover different-type registration through the reminder DIM.
- preserve active outbox reconciliation
 Keep reconciliation ownership visible until callbacks complete so an interleaving empty post cannot cancel acknowledged work. Processing timeouts now accept a scoped TimeProvider, and tests use deterministic gates and clocks.
- keep transient clocks out of persisted outboxes
 Accept an explicit UTC append timestamp while preserving the system-clock convenience overload. Persisted outboxes no longer retain a TimeProvider, so serialization and rehydration need no clock re-registration.
- prevent outbox drain handoff deadlock
 Release the drain gate between background dispatch and non-interleaving reconciliation, and let a foreground drain finish any pending batch before taking a new snapshot. This prevents non-reentrant grain calls from blocking the reconciliation turn while preserving direct PostAsync completion semantics.
- preserve outbox behavior after rehydration
 Fall back to TimeProvider.System when Orleans rehydrates an outbox without its non-serialized clock. Document custom-provider re-registration and cover the Orleans deep-copy path so the first mutation no longer fails.
- preserve sender timestamp in tracked outbox tokens
 MessageTracker.LatestOutbox reconstructed the last accepted
 OutboxSequenceToken using the receiver's Received wall-clock time as
 the token Timestamp. Token equality includes Timestamp, so whenever
 sender and receiver clocks differ the returned token did not equal the
 token that was actually accepted, breaking equality-based consumers and
 any logic reading the timestamp as sender time.
 Outbox entries now store the sender-stamped timestamp of the last
 accepted token alongside the receiver-side Received time (which remains
 the basis for eviction). LatestOutbox returns the original timestamp.
 JSON payloads written before the new property fall back to Received,
 matching the previous behavior.
- prune in-memory outbox attempt counters after reconciliation
 The outbox processor tracks per-item dispatch attempt counts in an
 in-memory dictionary keyed by item equality. Entries were only removed
 when an item posted successfully, so items that left the outbox any
 other way - dead-lettered or dropped in ReconcileFailedAsync, or removed
 directly by the grain - kept their entries for the lifetime of the
 activation, a slow memory leak invisible to the owning grain.
 The processor now prunes counters for items no longer pending after
 each reconciliation. A consequence is that an equal item enqueued after
 its predecessor was dead-lettered starts a fresh attempt sequence
 instead of inheriting stale counts.
 Also documents the attempt-count semantics on ReconcileFailedAsync:
 counts are per-activation and keyed by item equality, so retry and
 dead-letter policies that must survive activation restarts need to
 persist their own counters.
- detect diverged outbox tails during write recovery
 Outbox equality now compares the full first and last pending tokens,
 including their timestamps, instead of only their sequence numbers.
 Items are only appended at the tail, so two outbox histories that diverged
 by adding different messages - for example duplicate grain activations
 racing an ambiguous storage write - always differ in their highest pending
 token. Comparing the full token lets the state-manager recovery read-back
 distinguish such diverged states instead of mistaking them for a lost
 response, which could silently drop the local activation's message.
 Histories that diverged only by removals can still compare equal; recovery
 then at worst re-delivers an already posted item (at-least-once) but never
 loses a pending message. Equality remains O(1) and still ignores payloads.
- arm outbox retry when a foreground post run fails
 A failed PostAsync run - processing timeout, cancellation, or an exception
 from an acknowledgement or reconciliation callback - previously skipped the
 retry scheduling that normally happens after reconciliation. On a first-ever
 foreground post no timer or reminder existed yet, so announced messages
 stayed in the durable outbox until the next post or activation.
 The processor now arms its retry timer and reminder before rethrowing the
 post-run failure, but only on the failure path, so successful posts still
 avoid the reminder registration write.
- accept tokenless stream messages
 Do not persist a stream tracking entry when Orleans delivers a null StreamSequenceToken. Without a token there is no high-water mark to compare, so tokenless stream messages are accepted without changing the tracker instead of causing later tokenless messages to look like duplicates.
- harden outbox review behavior
 Remove posted outbox tokens by identity so successful items after a failed gap can still be acknowledged. Also makes outbox token/envelope types Orleans-friendly with required init properties and rewrites the STJ converters to delegate through typed converters while preserving the existing JSON shape.
- tighten stream subscription configuration
 Fail fast when Orleans invokes an implicit stream subscription without a configured handler, and add explicit StreamId subscription overloads so explicit streams are not limited to the grain-keyed convention.
 Clarify outbox ownership, first-match postman dispatch, thread-pool postman safety, and future provider-specific package boundaries in public docs.
- fail fast for missing grain components
 Throw clear configuration errors when implicit stream and outbox marker interfaces receive callbacks before their corresponding components have been attached.
 This avoids silently swallowing stream subscription or reminder callbacks when a grain implements the marker interface but forgets to call the registration helper during activation.
- resume streams from tracked cursors
 Use the activation-time MessageTracker snapshot to provide provider-aware resume tokens when StreamManager attaches implicit handles, resumes explicit handles, or creates missing explicit subscriptions.
 Add focused tests for provider-specific cursor lookup and the Orleans stream APIs that receive those tokens.
- tighten messaging manager invariants
 Fix stream subscription initialization so handles are collected without concurrent List mutation, repeated SubscribeAsync calls are rejected, and stream handlers receive nullable sequence tokens instead of relying on null-forgiving calls.
 Recover StateManager clear failures with a read-back path, clamp negative receive-lag telemetry, validate malformed outbox JSON with JsonException, and document the intentional outbox equality fingerprint invariant.
 Rename the deferred outbox processor activation API references to RegisterOutboxProcessor to align with the other activation-time helpers while leaving the implementation deferred.
- restrict StreamCursor JSON tokens to built-in Orleans types
 Limit StreamCursorJsonConverter to Orleans built-in stream sequence token types and remove custom token adapter serialization paths. Unsupported token kinds now throw a NotSupportedException with an actionable message that explains the stability rationale and points users to the GitHub issues page for feature requests.
 Update StreamCursorJsonConverterTests to assert unsupported custom tokens and unknown token kinds fail with the new guidance message while keeping built-in token round-trip coverage.
Performance:
- register outbox retry reminder lazily
 PostInBackgroundAsync and follow-up drain scheduling no longer register the
 durable reminder up front; they arm only the in-memory grain timer. The
 reminder - which costs a storage write - is now registered only when a post
 run fails or completes with items still pending, including when a background
 dispatch or reconciliation callback throws. Unregistering also caches the
 one-time lookup for a leftover reminder from a previous activation, so an
 empty outbox no longer pays a reminder-table read on every successful drain.
 Successful posts therefore incur no reminder I/O. The trade-off is that a
 silo crash between scheduling a background post and the first failed run
 loses the cross-activation retry trigger; pending items then wait for the
 next activation's post instead of a reminder tick.