Excalibur.Data.DataProcessing 10.0.0-alpha.17

.NET 10.0 This package targets .NET 10.0. The package is compatible with this framework or higher.
This is a prerelease version of Excalibur.Data.DataProcessing.
There is a newer prerelease version of this package available.
See the version list below for details. 
dotnet add package Excalibur.Data.DataProcessing --version 10.0.0-alpha.17
                    


                    

                
NuGet\Install-Package Excalibur.Data.DataProcessing -Version 10.0.0-alpha.17
This command is intended to be used within the Package Manager Console in Visual Studio, as it uses the NuGet module's version of Install-Package. 
<PackageReference Include="Excalibur.Data.DataProcessing" Version="10.0.0-alpha.17" />
For projects that support PackageReference, copy this XML node into the project file to reference the package. 
<PackageVersion Include="Excalibur.Data.DataProcessing" Version="10.0.0-alpha.17" />
                    


                            Directory.Packages.props
                        

                    

                
<PackageReference Include="Excalibur.Data.DataProcessing" />
                    


                            Project file
For projects that support Central Package Management (CPM), copy this XML node into the solution Directory.Packages.props file to version the package. 
paket add Excalibur.Data.DataProcessing --version 10.0.0-alpha.17
                    


                    

                
#r "nuget: Excalibur.Data.DataProcessing, 10.0.0-alpha.17"
#r directive can be used in F# Interactive and Polyglot Notebooks. Copy this into the interactive tool or source code of the script to reference the package. 
#:package Excalibur.Data.DataProcessing@10.0.0-alpha.17
#:package directive can be used in C# file-based apps starting in .NET 10 preview 4. Copy this into a .cs file before any lines of code to reference the package. 
#addin nuget:?package=Excalibur.Data.DataProcessing&version=10.0.0-alpha.17&prerelease
                    


                            Install as a Cake Addin
                        

                    

                
#tool nuget:?package=Excalibur.Data.DataProcessing&version=10.0.0-alpha.17&prerelease
                    


                            Install as a Cake Tool

Excalibur.Data.DataProcessing

Data processing utilities for the Excalibur framework.

Installation

dotnet add package Excalibur.Data.DataProcessing

Features

  • Batch processing
  • ETL operations
  • Data transformation
  • Bulk operations

The queue table is transient

AddDataTaskForRecordTypeAsync writes one row and returns its identifier. That row does not stay:

outcome the row
the processor completed the task deleted
the processor failed kept, with its attempt count raised
the attempt count reached the maximum kept, and no longer selected — a failed backfill stays visible

So an absent row is not evidence that the enqueue failed. With a drain running — a background service, or a scheduled job — a task whose processor succeeds can be gone within one polling interval. Query by the returned identifier to tell an enqueue that never happened from one already processed.

Each enqueue logs, at Information, the identifier it wrote, the schema-qualified table, the database the write actually landed in, and whether an ambient transaction was in scope. That line is the fastest way to settle where a task went:

Queued data task 0190c3f1-… for record type 'Orders' into [DataProcessor].[DataTaskRequests]
on database 'AppDb'. Ambient transaction: none.

Two further reasons a row can appear missing, both of which the enqueue reports no error for:

  • It is in another schema than the one you are looking at. The default is [DataProcessor].[DataTaskRequests], not the connection's default schema — and it is written on the connection you supplied to the registration, which need not be the database the rest of the application uses.
  • An ambient transaction rolled it back. The enqueue participates in Transaction.Current on purpose, so queueing work can be part of the business transaction that decided to queue it. A scope disposed without being completed therefore discards the row. The drain refuses an ambient transaction instead, because processing must not be undone by one.

Tenancy

The data task queue is a global, type-wide sweep: it runs outside any tenant scope, and its enqueue and drain surfaces (IDataOrchestrationManager.AddDataTaskForRecordTypeAsync, the pending-task drain) accept no tenant identity. If your IRecordFetcher records are tenant-owned, your fetcher must apply the tenant predicate itself -- the queue applies none.

License

This project is multi-licensed under:

See LICENSE for details.

Database schema

This package does not create its tables. The DDL ships inside the .nupkg under scripts/, and you apply it — so a schema change to your database is always something you did deliberately, never something a version upgrade did to you while you were not looking.

Script
scripts/001_CreateDataProcessingSchema.sql

Apply them in filename order before starting the application, and again — for any new ones — before starting a new version after an upgrade.

The scripts/ folder lives inside the package, not in your build output. To read it:

# the package folder, then the scripts inside it
dotnet nuget locals global-packages --list
ls ~/.nuget/packages/excalibur.data.dataprocessing/<version>/scripts/

The folder is authoritative. If any list of scripts — including this one — disagrees with what is actually in the package you installed, the package is right.

Task ownership and recovery

The SQL Server orchestrator requires a factory that returns a new, closed connection. It claims each task with a database-scoped session application lock, rereads its eligibility, and retains the connection until processing and asynchronous scope cleanup finish. Competing workers skip owned tasks. Each checkpoint, failure update, and deletion verifies ownership on that session. Claim connections disable pooling so a lost grant or release response cannot strand an application lock in the pool; this adds one physical connection per claimed task, not per record.

A lost database session fences subsequent task-state writes. It cannot undo external handler effects already underway. Handlers must remain idempotent and tolerate replay and, after connection loss, overlapping external effects. All workers must use the ownership-aware implementation; older workers do not honor these locks. Drain tasks outside ambient transactions; enqueue may participate in a business transaction.

DispatcherTimeoutMilliseconds requests cancellation for each claimed task, including scope cleanup. Cancellation is cooperative: a handler that ignores it can delay completion. The manager joins processing before releasing ownership. An observed timeout consumes a failed attempt; host cancellation does not. Attempts counts observed failures, not process crashes. CompletedCount counts successful processing attempts and may include replayed records, rather than distinct records.

Repeated failed hosted cycles degrade health and become unhealthy at UnhealthyThreshold; a successful cycle resets the failure streak. Exceeding shutdown drain time reports the service as stopped and unhealthy even if noncooperative work has not yet finished.

Product Compatible and additional computed target framework versions.
.NET net10.0 is compatible.  net10.0-android was computed.  net10.0-browser was computed.  net10.0-ios was computed.  net10.0-maccatalyst was computed.  net10.0-macos was computed.  net10.0-tvos was computed.  net10.0-windows was computed. 
Compatible target framework(s)
Included target framework(s) (in package)
Learn more about Target Frameworks and .NET Standard.

NuGet packages (1)

Showing the top 1 NuGet packages that depend on Excalibur.Data.DataProcessing:

Package Downloads
Excalibur.Jobs.DataProcessing

Data processing job types for the Excalibur Jobs framework. PRE-RELEASE: the public API is not frozen and may change between pre-release builds. Not recommended for production use. Known issues: https://docs.excalibur-dispatch.dev/docs/known-issues

GitHub repositories

This package is not used by any popular GitHub repositories.

Version Downloads Last Updated
10.0.0-alpha.18 0 10/25/2026
10.0.0-alpha.17 51 10/23/2026
10.0.0-alpha.16 47 10/21/2026
10.0.0-alpha.15 58 10/11/2026
10.0.0-alpha.14 66 10/2/2026
10.0.0-alpha.13 67 9/29/2026
10.0.0-alpha.12 71 9/25/2026
10.0.0-alpha.11 80 9/15/2026
10.0.0-alpha.10 88 9/6/2026
10.0.0-alpha.9 90 8/31/2026
10.0.0-alpha.8 93 8/14/2026
10.0.0-alpha.7 93 8/13/2026
10.0.0-alpha.6 94 8/11/2026
10.0.0-alpha.5 85 8/10/2026
10.0.0-alpha.4 83 8/10/2026
3.0.0-alpha.216 96 6/30/2026
3.0.0-alpha.215 86 6/23/2026
3.0.0-alpha.214 94 6/23/2026
3.0.0-alpha.208 83 6/11/2026
3.0.0-alpha.207 93 6/11/2026
Loading failed

Release notes and versioning policy: https://docs.excalibur-dispatch.dev/docs/migration/version-upgrades