Excalibur.Data.DataProcessing
10.0.0-alpha.16
See the version list below for details.
dotnet add package Excalibur.Data.DataProcessing --version 10.0.0-alpha.16
NuGet\Install-Package Excalibur.Data.DataProcessing -Version 10.0.0-alpha.16
<PackageReference Include="Excalibur.Data.DataProcessing" Version="10.0.0-alpha.16" />
<PackageVersion Include="Excalibur.Data.DataProcessing" Version="10.0.0-alpha.16" />
<PackageReference Include="Excalibur.Data.DataProcessing" />
paket add Excalibur.Data.DataProcessing --version 10.0.0-alpha.16
#r "nuget: Excalibur.Data.DataProcessing, 10.0.0-alpha.16"
#:package Excalibur.Data.DataProcessing@10.0.0-alpha.16
#addin nuget:?package=Excalibur.Data.DataProcessing&version=10.0.0-alpha.16&prerelease
#tool nuget:?package=Excalibur.Data.DataProcessing&version=10.0.0-alpha.16&prerelease
Excalibur.Data.DataProcessing
Data processing utilities for the Excalibur framework.
Installation
dotnet add package Excalibur.Data.DataProcessing
Features
- Batch processing
- ETL operations
- Data transformation
- Bulk operations
The queue table is transient
AddDataTaskForRecordTypeAsync writes one row and returns its identifier. That row does not stay:
| outcome | the row |
|---|---|
| the processor completed the task | deleted |
| the processor failed | kept, with its attempt count raised |
| the attempt count reached the maximum | kept, and no longer selected — a failed backfill stays visible |
So an absent row is not evidence that the enqueue failed. With a drain running — a background service, or a scheduled job — a task whose processor succeeds can be gone within one polling interval. Query by the returned identifier to tell an enqueue that never happened from one already processed.
Each enqueue logs, at Information, the identifier it wrote, the schema-qualified table, the database
the write actually landed in, and whether an ambient transaction was in scope. That line is the fastest
way to settle where a task went:
Queued data task 0190c3f1-… for record type 'Orders' into [DataProcessor].[DataTaskRequests]
on database 'AppDb'. Ambient transaction: none.
Two further reasons a row can appear missing, both of which the enqueue reports no error for:
- It is in another schema than the one you are looking at. The default is
[DataProcessor].[DataTaskRequests], not the connection's default schema — and it is written on the connection you supplied to the registration, which need not be the database the rest of the application uses. - An ambient transaction rolled it back. The enqueue participates in
Transaction.Currenton purpose, so queueing work can be part of the business transaction that decided to queue it. A scope disposed without being completed therefore discards the row. The drain refuses an ambient transaction instead, because processing must not be undone by one.
Tenancy
The data task queue is a global, type-wide sweep: it runs outside any tenant scope, and its enqueue and
drain surfaces (IDataOrchestrationManager.AddDataTaskForRecordTypeAsync, the pending-task drain) accept
no tenant identity. If your IRecordFetcher records are tenant-owned, your fetcher must apply the tenant
predicate itself -- the queue applies none.
License
This project is multi-licensed under:
See LICENSE for details.
Database schema
This package does not create its tables. The DDL ships inside the .nupkg under scripts/, and
you apply it — so a schema change to your database is always something you did deliberately, never
something a version upgrade did to you while you were not looking.
| Script |
|---|
scripts/001_CreateDataProcessingSchema.sql |
Apply them in filename order before starting the application, and again — for any new ones — before starting a new version after an upgrade.
The scripts/ folder lives inside the package, not in your build output. To read it:
# the package folder, then the scripts inside it
dotnet nuget locals global-packages --list
ls ~/.nuget/packages/excalibur.data.dataprocessing/<version>/scripts/
The folder is authoritative. If any list of scripts — including this one — disagrees with what is actually in the package you installed, the package is right.
Task ownership and recovery
The SQL Server orchestrator requires a factory that returns a new, closed connection. It claims each task with a database-scoped session application lock, rereads its eligibility, and retains the connection until processing and asynchronous scope cleanup finish. Competing workers skip owned tasks. Each checkpoint, failure update, and deletion verifies ownership on that session. Claim connections disable pooling so a lost grant or release response cannot strand an application lock in the pool; this adds one physical connection per claimed task, not per record.
A lost database session fences subsequent task-state writes. It cannot undo external handler effects already underway. Handlers must remain idempotent and tolerate replay and, after connection loss, overlapping external effects. All workers must use the ownership-aware implementation; older workers do not honor these locks. Drain tasks outside ambient transactions; enqueue may participate in a business transaction.
DispatcherTimeoutMilliseconds requests cancellation for each claimed task, including scope cleanup. Cancellation is cooperative: a handler that ignores it can delay completion. The manager joins processing before releasing ownership. An observed timeout consumes a failed attempt; host cancellation does not. Attempts counts observed failures, not process crashes. CompletedCount counts successful processing attempts and may include replayed records, rather than distinct records.
Repeated failed hosted cycles degrade health and become unhealthy at UnhealthyThreshold; a successful cycle resets the failure streak. Exceeding shutdown drain time reports the service as stopped and unhealthy even if noncooperative work has not yet finished.
| Product | Versions Compatible and additional computed target framework versions. |
|---|---|
| .NET | net10.0 is compatible. net10.0-android was computed. net10.0-browser was computed. net10.0-ios was computed. net10.0-maccatalyst was computed. net10.0-macos was computed. net10.0-tvos was computed. net10.0-windows was computed. |
-
net10.0
- Ben.Demystifier (>= 0.4.1)
- CloudNative.CloudEvents (>= 2.8.0)
- CloudNative.CloudEvents.SystemTextJson (>= 2.8.0)
- Cronos (>= 0.12.0)
- Dapper (>= 2.1.72)
- Excalibur.Data (>= 10.0.0-alpha.16)
- Excalibur.Dispatch (>= 10.0.0-alpha.16)
- Excalibur.Dispatch.Abstractions (>= 10.0.0-alpha.16)
- Excalibur.Hosting (>= 10.0.0-alpha.16)
- FluentValidation (>= 12.1.1)
- FluentValidation.DependencyInjectionExtensions (>= 12.1.1)
- IdentityModel (>= 7.0.0)
- JsonNet.ContractResolvers (>= 2.0.0)
- Medo.Uuid7 (>= 3.2.0)
- MemoryPack (>= 1.21.4)
- Microsoft.Extensions.Caching.Abstractions (>= 10.0.10)
- Microsoft.Extensions.Caching.Memory (>= 10.0.10)
- Microsoft.Extensions.Configuration (>= 10.0.10)
- Microsoft.Extensions.Configuration.Abstractions (>= 10.0.10)
- Microsoft.Extensions.Configuration.Binder (>= 10.0.10)
- Microsoft.Extensions.Configuration.CommandLine (>= 10.0.10)
- Microsoft.Extensions.Configuration.EnvironmentVariables (>= 10.0.10)
- Microsoft.Extensions.Configuration.Json (>= 10.0.10)
- Microsoft.Extensions.DependencyInjection (>= 10.0.10)
- Microsoft.Extensions.DependencyInjection.Abstractions (>= 10.0.10)
- Microsoft.Extensions.Diagnostics.HealthChecks (>= 10.0.10)
- Microsoft.Extensions.Diagnostics.HealthChecks.Abstractions (>= 10.0.10)
- Microsoft.Extensions.Hosting (>= 10.0.10)
- Microsoft.Extensions.Hosting.Abstractions (>= 10.0.10)
- Microsoft.Extensions.Http (>= 10.0.10)
- Microsoft.Extensions.Logging (>= 10.0.10)
- Microsoft.Extensions.Logging.Abstractions (>= 10.0.10)
- Microsoft.Extensions.Logging.Console (>= 10.0.10)
- Microsoft.Extensions.ObjectPool (>= 10.0.10)
- Microsoft.Extensions.Options (>= 10.0.10)
- Microsoft.Extensions.Options.ConfigurationExtensions (>= 10.0.10)
- Microsoft.Extensions.Options.DataAnnotations (>= 10.0.10)
- Microsoft.IdentityModel.Tokens (>= 8.17.0)
- Polly (>= 8.6.6)
- System.IdentityModel.Tokens.Jwt (>= 8.17.0)
- System.Threading.RateLimiting (>= 10.0.7)
NuGet packages (1)
Showing the top 1 NuGet packages that depend on Excalibur.Data.DataProcessing:
| Package | Downloads |
|---|---|
|
Excalibur.Jobs.DataProcessing
Data processing job types for the Excalibur Jobs framework. PRE-RELEASE: the public API is not frozen and may change between pre-release builds. Not recommended for production use. Known issues: https://docs.excalibur-dispatch.dev/docs/known-issues |
GitHub repositories
This package is not used by any popular GitHub repositories.
| Version | Downloads | Last Updated |
|---|---|---|
| 10.0.0-alpha.18 | 0 | 10/25/2026 |
| 10.0.0-alpha.17 | 51 | 10/23/2026 |
| 10.0.0-alpha.16 | 47 | 10/21/2026 |
| 10.0.0-alpha.15 | 58 | 10/11/2026 |
| 10.0.0-alpha.14 | 66 | 10/2/2026 |
| 10.0.0-alpha.13 | 67 | 9/29/2026 |
| 10.0.0-alpha.12 | 71 | 9/25/2026 |
| 10.0.0-alpha.11 | 80 | 9/15/2026 |
| 10.0.0-alpha.10 | 88 | 9/6/2026 |
| 10.0.0-alpha.9 | 90 | 8/31/2026 |
| 10.0.0-alpha.8 | 93 | 8/14/2026 |
| 10.0.0-alpha.7 | 93 | 8/13/2026 |
| 10.0.0-alpha.6 | 94 | 8/11/2026 |
| 10.0.0-alpha.5 | 85 | 8/10/2026 |
| 10.0.0-alpha.4 | 83 | 8/10/2026 |
| 3.0.0-alpha.216 | 96 | 6/30/2026 |
| 3.0.0-alpha.215 | 86 | 6/23/2026 |
| 3.0.0-alpha.214 | 94 | 6/23/2026 |
| 3.0.0-alpha.208 | 83 | 6/11/2026 |
| 3.0.0-alpha.207 | 93 | 6/11/2026 |
Release notes and versioning policy: https://docs.excalibur-dispatch.dev/docs/migration/version-upgrades