Excalibur.Data.ElasticSearch 10.0.0-alpha.13

.NET 10.0 This package targets .NET 10.0. The package is compatible with this framework or higher.
This is a prerelease version of Excalibur.Data.ElasticSearch. 
dotnet add package Excalibur.Data.ElasticSearch --version 10.0.0-alpha.13
                    


                    

                
NuGet\Install-Package Excalibur.Data.ElasticSearch -Version 10.0.0-alpha.13
This command is intended to be used within the Package Manager Console in Visual Studio, as it uses the NuGet module's version of Install-Package. 
<PackageReference Include="Excalibur.Data.ElasticSearch" Version="10.0.0-alpha.13" />
For projects that support PackageReference, copy this XML node into the project file to reference the package. 
<PackageVersion Include="Excalibur.Data.ElasticSearch" Version="10.0.0-alpha.13" />
                    


                            Directory.Packages.props
                        

                    

                
<PackageReference Include="Excalibur.Data.ElasticSearch" />
                    


                            Project file
For projects that support Central Package Management (CPM), copy this XML node into the solution Directory.Packages.props file to version the package. 
paket add Excalibur.Data.ElasticSearch --version 10.0.0-alpha.13
                    


                    

                
#r "nuget: Excalibur.Data.ElasticSearch, 10.0.0-alpha.13"
#r directive can be used in F# Interactive and Polyglot Notebooks. Copy this into the interactive tool or source code of the script to reference the package. 
#:package Excalibur.Data.ElasticSearch@10.0.0-alpha.13
#:package directive can be used in C# file-based apps starting in .NET 10 preview 4. Copy this into a .cs file before any lines of code to reference the package. 
#addin nuget:?package=Excalibur.Data.ElasticSearch&version=10.0.0-alpha.13&prerelease
                    


                            Install as a Cake Addin
                        

                    

                
#tool nuget:?package=Excalibur.Data.ElasticSearch&version=10.0.0-alpha.13&prerelease
                    


                            Install as a Cake Tool

Excalibur.Data.ElasticSearch

Elasticsearch data provider for the Excalibur framework, providing enterprise-grade document storage, full-text search, and projection management with comprehensive resilience, performance, and monitoring capabilities.

Overview

This package provides Elasticsearch integration for Excalibur applications, enabling:

  • Document Storage: Type-safe repository pattern with CRUD operations
  • Full-Text Search: Rich query DSL support with aggregations
  • Index Management: Lifecycle management, templates, and schema evolution
  • Projections: Event sourcing projection support with rebuild capabilities
  • Resilience: Circuit breaker, retry policies, and dead letter handling
  • Performance: Connection pooling with configurable pool types and node sniffing
  • Monitoring: OpenTelemetry integration, metrics, and health checks
  • Security: API key, certificate, and basic authentication

Installation

dotnet add package Excalibur.Data.ElasticSearch

Dependencies:

  • Elastic.Clients.Elasticsearch (8.x)
  • Microsoft.Extensions.DependencyInjection
  • Microsoft.Extensions.Options

Configuration

Basic Connection

services.Configure<ElasticsearchConfigurationOptions>(options =>
{
    // Single node
    options.Url = new Uri("https://localhost:9200");

    // Or multiple nodes
    options.Urls = new[]
    {
        new Uri("https://node1:9200"),
        new Uri("https://node2:9200"),
        new Uri("https://node3:9200")
    };
});

Elastic Cloud

services.Configure<ElasticsearchConfigurationOptions>(options =>
{
    options.CloudId = "my-deployment:dXMtY2VudHJhbDE...";
    options.Connection.ApiKey = "your-api-key";
});

Authentication Options

options.Connection.ApiKey = "your-api-key";
// Or Base64-encoded
options.Connection.Base64ApiKey = "base64-encoded-api-key";
Basic Authentication
options.Connection.Username = "elastic";
options.Connection.Password = "your-password";
Certificate Fingerprint
options.Connection.CertificateFingerprint = "A1:B2:C3:...";
options.Connection.DisableCertificateValidation = false;  // Keep false in production

Environment Variables

ELASTICSEARCH__URL=https://localhost:9200
ELASTICSEARCH__CONNECTION__APIKEY=your-api-key
ELASTICSEARCH__CONNECTION__USERNAME=elastic
ELASTICSEARCH__CONNECTION__PASSWORD=your-password

services.Configure<ElasticsearchConfigurationOptions>(
    configuration.GetSection("Elasticsearch"));

Connection Settings

services.Configure<ElasticsearchConfigurationOptions>(options =>
{
    // Timeouts
    options.Connection.RequestTimeout = TimeSpan.FromSeconds(30);
    options.Connection.PingTimeout = TimeSpan.FromSeconds(5);

    // Connection pooling
    options.ConnectionPoolType = ConnectionPoolType.Sniffing;
    options.Connection.MaximumConnectionsPerNode = 80;

    // Node discovery
    options.EnableSniffing = true;
    options.Connection.SniffingInterval = TimeSpan.FromHours(1);
});

Security

Two distinct secret-handling capabilities live in this package, and they are deliberately separate abstractions:

Connection-credential storage

IElasticsearchKeyStorage stores the opaque credentials this package uses to authenticate to Elasticsearch itself -- OAuth tokens, service-account secrets, passwords, API keys. An in-memory development store is wired by default. To back it with Azure Key Vault instead, install the optional Excalibur.Data.ElasticSearch.Azure package and call AddAzureKeyVaultCredentialStorage(configuration) before registering authentication. That provider lives in its own package so this one does not put the Azure SDK on consumers who never use Key Vault.

Field-level encryption key management

Field-level encryption (AddFieldEncryption()) delegates key resolution and cryptographic operations entirely to Excalibur.Compliance's IKeyManagementProvider and IEncryptionProviderRegistry -- this package never sees a raw encryption key. Call AddKeyManagement(configuration) to select a provider:

  • Local -- the in-process development provider (Excalibur.Compliance's AddDevEncryption()). Keys are lost on restart; never use in production.
  • AzureKeyVault / AwsKms / GoogleCloudKms / HashiCorpVault -- register the corresponding Excalibur.Compliance.Azure / .Aws / .Vault package's key-management extension (e.g. services.AddEncryption(e => e.UseKeyManagement<AzureKeyVaultProvider>())) before calling AddKeyManagement; it is refused otherwise, so a cloud selection can never silently fall back to the development provider.

Index Management

Index Lifecycle Management

// Inject IIndexLifecycleManager
public class MyService
{
    private readonly IIndexLifecycleManager _lifecycleManager;

    public async Task ConfigureIndexAsync()
    {
        var policy = new IndexLifecyclePolicy
        {
            Hot = new HotPhaseConfiguration
            {
                RolloverConditions = new RolloverConditions
                {
                    MaxAge = TimeSpan.FromDays(7),
                    MaxSize = "50gb"
                }
            },
            Warm = new WarmPhaseConfiguration
            {
                MinAge = TimeSpan.FromDays(30)
            },
            Delete = new DeletePhaseConfiguration
            {
                MinAge = TimeSpan.FromDays(90)
            }
        };

        await _lifecycleManager.CreatePolicyAsync("my-policy", policy);
    }
}

Index Templates

// Inject IIndexTemplateManager
var template = new IndexTemplateConfiguration
{
    Name = "my-template",
    IndexPatterns = new[] { "logs-*" },
    Priority = 100
};

await _templateManager.CreateTemplateAsync(template);

Projections

Projection Store

public class OrderProjection
{
    public string OrderId { get; set; }
    public string CustomerId { get; set; }
    public decimal Total { get; set; }
    public DateTime LastModified { get; set; }
}

// Configure projection store
services.AddElasticSearchProjectionStore<OrderProjection>(options =>
{
    options.IndexPrefix = "orders";
});

Projection Rebuild

// Inject IProjectionRebuildManager
var request = new ProjectionRebuildRequest
{
    ProjectionType = nameof(OrderProjection),
    SourceIndexName = "orders-v1",
    TargetIndexName = "orders-v2",
    CreateNewIndex = true,
    UseAliasing = true,
    BatchSize = 1000
};

var result = await _rebuildManager.StartRebuildAsync(request);

// Check status
var status = await _rebuildManager.GetRebuildStatusAsync(result.OperationId);

Eventual Consistency Tracking

// Inject IEventualConsistencyTracker
var eventId = Guid.NewGuid().ToString();
await _tracker.TrackWriteModelEventAsync(eventId, "order-1", "OrderCreated", DateTime.UtcNow);
await _tracker.TrackReadModelProjectionAsync(eventId, nameof(OrderProjection), DateTime.UtcNow);

// Check lag
var lag = await _tracker.GetConsistencyLagAsync(nameof(OrderProjection));
if (!lag.IsWithinSLA)
{
    _logger.LogWarning("Projection lagging by {Events} events", lag.PendingEvents);
}

Schema Evolution

// Inject ISchemaEvolutionHandler
var comparison = await _schemaHandler.CompareSchemaAsync("orders-v1", "orders-v2");

if (!comparison.IsBackwardsCompatible)
{
    var migrationRequest = new SchemaMigrationRequest
    {
        ProjectionType = nameof(OrderProjection),
        SourceIndex = "orders-v1",
        TargetIndex = "orders-v2",
        Strategy = MigrationStrategy.AliasSwitch,
        NewSchema = new Elastic.Clients.Elasticsearch.Mapping.Properties
        {
            { "orderId", new Elastic.Clients.Elasticsearch.Mapping.KeywordProperty() },
            { "status", new Elastic.Clients.Elasticsearch.Mapping.KeywordProperty() },
            { "total", new Elastic.Clients.Elasticsearch.Mapping.DoubleNumberProperty() }
        }
    };

    var plan = await _schemaHandler.PlanMigrationAsync(migrationRequest);
    await _schemaHandler.ExecuteMigrationAsync(plan);
}

Resilience

Circuit Breaker

services.Configure<ElasticsearchConfigurationOptions>(options =>
{
    options.Resilience = new ElasticsearchResilienceOptions
    {
        CircuitBreaker = new CircuitBreakerOptions
        {
            Enabled = true,
            MinimumThroughput = 10,        // Minimum requests before evaluation
            BreakDuration = TimeSpan.FromSeconds(30),
            SamplingDuration = TimeSpan.FromSeconds(60),
            FailureRatio = 0.5             // opens at a 50% failure ratio
        }
    };
});

Retry Policy

services.Configure<ElasticsearchConfigurationOptions>(options =>
{
    options.Resilience = new ElasticsearchResilienceOptions
    {
        Retry = new ElasticSearchRetryPolicyOptions
        {
            Enabled = true,
            MaxAttempts = 3,
            BaseDelay = TimeSpan.FromMilliseconds(100),
            MaxDelay = TimeSpan.FromSeconds(30),
            UseExponentialBackoff = true,
            JitterFactor = 0.1
        }
    };
});

Dead Letter Handling

services.Configure<ElasticsearchDeadLetterOptions>(options =>
{
    options.DeadLetterIndexPrefix = "dead-letters";
    options.MaxRetryCount = 3;
    options.RetentionPeriod = TimeSpan.FromDays(30);
});

Monitoring

OpenTelemetry Metrics

services.AddOpenTelemetry()
    .WithMetrics(metrics =>
    {
        metrics.AddMeter("Excalibur.Data.ElasticSearch");
    })
    .WithTracing(tracing =>
    {
        tracing.AddSource("Excalibur.Data.ElasticSearch");
    });

Monitoring Settings

services.Configure<ElasticsearchConfigurationOptions>(options =>
{
    options.Monitoring = new ElasticsearchMonitoringOptions
    {
        Metrics = new MetricsOptions { Enabled = true },
        RequestLogging = new RequestLoggingOptions
        {
            Enabled = true,
            LogFailuresOnly = true,
            MaxBodySizeBytes = 1024
        },
        Performance = new PerformanceDiagnosticsOptions
        {
            Enabled = true,
            SlowOperationThreshold = TimeSpan.FromSeconds(1),
            SamplingRate = 0.1  // 10% sampling
        }
    };
});

Health Checks

Registration

services.AddHealthChecks()
    .AddElasticSearchHealthCheck(tags: new[] { "ready", "elasticsearch" });

Custom Configuration

services.AddHealthChecks()
    .AddCheck<ElasticClientHealthCheck>(
        "elasticsearch",
        failureStatus: HealthStatus.Degraded,
        tags: new[] { "ready" },
        timeout: TimeSpan.FromSeconds(10));

Repository Pattern

Define Repository

public class OrderRepository : ElasticRepositoryBase<Order, string>
{
    public OrderRepository(IElasticClient client) : base(client, "orders") { }

    public async Task<IEnumerable<Order>> FindByCustomerAsync(string customerId)
    {
        return await SearchAsync(q => q
            .Match(m => m.Field(f => f.CustomerId).Query(customerId)));
    }
}

Register and Use

services.AddScoped<IOrderRepository, OrderRepository>();

// In your service
public class OrderService
{
    private readonly IOrderRepository _orders;

    public async Task<Order> GetAsync(string id)
    {
        return await _orders.GetByIdAsync(id);
    }
}

Troubleshooting

Common Issues

Connection Refused
Elasticsearch.Net.ElasticsearchClientException: Connection refused

Solutions:

  • Verify Elasticsearch is running
  • Check URL and port configuration
  • Verify firewall allows connections
  • Check certificate fingerprint for HTTPS
Authentication Failed
Elasticsearch.Net.ElasticsearchClientException: 401 Unauthorized

Solutions:

  • Verify API key or credentials
  • Check user has required permissions
  • Ensure authentication method matches server configuration
Index Not Found
Elasticsearch.Net.ElasticsearchClientException: index_not_found_exception

Solutions:

  • Verify index exists: GET /_cat/indices
  • Check index name spelling (case-sensitive)
  • Create index if using auto-create

Logging Configuration

{
  "Logging": {
    "LogLevel": {
      "Excalibur.Data.ElasticSearch": "Debug",
      "Elastic.Transport": "Warning"
    }
  }
}

Complete Configuration Reference

services.Configure<ElasticsearchConfigurationOptions>(options =>
{
    // Endpoint
    options.Url = new Uri("https://localhost:9200");
    options.CloudId = null;
    options.ConnectionPoolType = ConnectionPoolType.Static;
    options.EnableSniffing = false;

    // Connection (authentication, timeouts, SSL/TLS)
    options.Connection.Username = null;
    options.Connection.Password = null;
    options.Connection.ApiKey = "your-api-key";
    options.Connection.CertificateFingerprint = null;
    options.Connection.DisableCertificateValidation = false;
    options.Connection.RequestTimeout = TimeSpan.FromSeconds(30);
    options.Connection.PingTimeout = TimeSpan.FromSeconds(5);
    options.Connection.MaximumConnectionsPerNode = 80;
    options.Connection.SniffingInterval = TimeSpan.FromHours(1);

    // Resilience
    options.Resilience = new ElasticsearchResilienceOptions();

    // Monitoring
    options.Monitoring = new ElasticsearchMonitoringOptions();

    // Projections
    options.Projections = new ProjectionOptions();
});

See Also

License

This project is multi-licensed under:

See LICENSE for details.

Product Compatible and additional computed target framework versions.
.NET net10.0 is compatible.  net10.0-android was computed.  net10.0-browser was computed.  net10.0-ios was computed.  net10.0-maccatalyst was computed.  net10.0-macos was computed.  net10.0-tvos was computed.  net10.0-windows was computed. 
Compatible target framework(s)
Included target framework(s) (in package)
Learn more about Target Frameworks and .NET Standard.

NuGet packages (3)

Showing the top 3 NuGet packages that depend on Excalibur.Data.ElasticSearch:

Package Downloads
Excalibur.Inbox.ElasticSearch

Elasticsearch implementation of the inbox pattern for Excalibur, providing idempotent message processing. PRE-RELEASE: the public API is not frozen and may change between pre-release builds. Not recommended for production use. Known issues: https://docs.excalibur-dispatch.dev/docs/known-issues
Excalibur.Outbox.ElasticSearch

Elasticsearch implementation of the outbox pattern for Excalibur. PRE-RELEASE: the public API is not frozen and may change between pre-release builds. Not recommended for production use. Known issues: https://docs.excalibur-dispatch.dev/docs/known-issues
Excalibur.Data.ElasticSearch.Azure

Azure Key Vault connection-credential storage for Excalibur.Data.ElasticSearch. Optional: install this only if you back Elasticsearch authentication secrets with Key Vault. PRE-RELEASE: the public API is not frozen and may change between pre-release builds. Not recommended for production use. Known issues: https://docs.excalibur-dispatch.dev/docs/known-issues

GitHub repositories

This package is not used by any popular GitHub repositories.

Version Downloads Last Updated
10.0.0-alpha.13 0 9/29/2026
10.0.0-alpha.12 64 9/25/2026
10.0.0-alpha.11 77 9/15/2026
10.0.0-alpha.10 75 9/6/2026
10.0.0-alpha.9 79 8/31/2026
10.0.0-alpha.8 97 8/14/2026
10.0.0-alpha.7 89 8/13/2026
10.0.0-alpha.6 88 8/11/2026
10.0.0-alpha.5 80 8/10/2026
10.0.0-alpha.4 79 8/10/2026
3.0.0-alpha.216 102 6/30/2026
3.0.0-alpha.215 88 6/23/2026
3.0.0-alpha.214 92 6/23/2026
3.0.0-alpha.208 86 6/11/2026
3.0.0-alpha.207 84 6/11/2026
3.0.0-alpha.205 83 6/10/2026
3.0.0-alpha.204 93 6/8/2026
3.0.0-alpha.203 91 6/8/2026
3.0.0-alpha.202 79 6/8/2026
3.0.0-alpha.201 85 6/8/2026
Loading failed

Release notes and versioning policy: https://docs.excalibur-dispatch.dev/docs/migration/version-upgrades