ISC.Kafka.Logging 1.0.0

ISC.Kafka.Logging

Lightweight, non-blocking, fail-safe Kafka logging client for .NET.

Why This Package?

The legacy IKafkaProducer has critical issues in production:

Migration from Legacy

4 steps — minimal code changes

Step 1. Install package

dotnet add package ISC.Kafka.Logging

Step 2. Replace using

- using legacy.Repositories.Kafka;
+ using ISC.Kafka.Logging;

Step 3. Replace DI registration

- services.AddSingleton<IKafkaProducer, KafkaProducer>();
+ services.AddKafkaProducer(configuration.GetSection("Kafka"));

Step 4. Remove await from all ProduceAsync calls

- await _producer.ProduceAsync(message);
+ _producer.ProduceAsync(message);

That's it. Same interface name IKafkaProducer, same method name ProduceAsync. No other code changes needed.

Installation

dotnet add package ISC.Kafka.Logging

<details> <summary>Other install methods</summary>

Package Manager:

Install-Package ISC.Kafka.Logging

PackageReference:

<PackageReference Include="ISC.Kafka.Logging" Version="1.0.0" />

</details>

Compatibility

Quick Start

1. Add configuration

// appsettings.json
{
  "Kafka": {
    "BootstrapServers": "localhost:9092",
    "Topic": "app-logs",
    "EnableLogging": true
  }
}

2. Register services

builder.Services.AddKafkaProducer(builder.Configuration.GetSection("Kafka"));

3. Inject and use

public class OrderService
{
    private readonly IKafkaProducer _producer;

    public OrderService(IKafkaProducer producer)
    {
        _producer = producer;
    }

    public void CreateOrder(int userId)
    {
        // Fire-and-forget — returns immediately, Kafka delivery happens in background
        _producer.ProduceAsync(new { UserId = userId, Action = "CREATE_ORDER", Timestamp = DateTime.UtcNow });
    }
}

Configuration

All options are bound from IConfiguration via KafkaOptions:

SASL/SSL Authentication

When SaslUsername is provided, SASL authentication is enabled. Defaults to SaslMechanism = Plain and SecurityProtocol = SaslSsl if not explicitly configured.

Do not store credentials in appsettings.json or source code. Use one of the methods below to inject SaslUsername and SaslPassword into IConfiguration at runtime.

appsettings.json only contains non-secret values:

{
  "Kafka": {
    "BootstrapServers": "broker.example.com:9093",
    "Topic": "app-logs",
    "SaslMechanism": "ScramSha512",
    "SecurityProtocol": "SaslPlaintext",
    "ClientSoftwareName": "my-service"
  }
}

Credentials are supplied separately via any of the following methods. All values merge into the same IConfiguration section and are automatically bound to KafkaOptions — no code changes required.

Option 1: Environment Variables (recommended for containers / CI/CD)

export Kafka__SaslUsername=your-username
export Kafka__SaslPassword=your-password

.NET maps Kafka__SaslUsername (double underscore) to IConfiguration["Kafka:SaslUsername"] automatically.

Option 2: User Secrets (recommended for local development)

dotnet user-secrets init
dotnet user-secrets set "Kafka:SaslUsername" "your-username"
dotnet user-secrets set "Kafka:SaslPassword" "your-password"

Requires AddUserSecrets() in the host builder (included by default in ASP.NET Core Development environment).

Option 3: Vault / Key Vault (recommended for production)

Install the config provider for your secrets manager and register it at startup. Credentials are pushed into IConfiguration before the app starts.

Example (Azure Key Vault):

builder.Configuration.AddAzureKeyVault(
    new Uri("https://your-vault.vault.azure.net/"),
    new DefaultAzureCredential());

Store secrets in the vault with keys Kafka--SaslUsername and Kafka--SaslPassword (double dash -- maps to the : separator).

Recommended Settings by Environment

Development:

{
  "Kafka": {
    "BootstrapServers": "localhost:9092",
    "Topic": "dev-logs",
    "MessageTimeoutMs": 1000,
    "SocketTimeoutMs": 2000,
    "MessageSendMaxRetries": 1
  }
}

Production (high-load):

{
  "Kafka": {
    "BootstrapServers": "broker1:9092,broker2:9092",
    "Topic": "prod-logs",
    "MessageTimeoutMs": 300,
    "SocketTimeoutMs": 500,
    "RetryBackoffMs": 100,
    "MessageSendMaxRetries": 0
  }
}

Transports

The package ships with three built-in transports:

Using a different transport

// Console output for development
builder.Services.AddKafkaProducer<ConsoleTransport>(builder.Configuration.GetSection("Kafka"));

// Disable logging entirely
builder.Services.AddKafkaProducer<NullTransport>(builder.Configuration.GetSection("Kafka"));

Custom transport

Implement the ILogTransport interface:

public class FileTransport : ILogTransport
{
    public async Task SendAsync(string payload)
    {
        await File.AppendAllTextAsync("logs.txt", payload + Environment.NewLine);
    }
}

// Register
builder.Services.AddKafkaProducer<FileTransport>(builder.Configuration.GetSection("Kafka"));

API Reference

IKafkaProducer

public interface IKafkaProducer
{
    void ProduceAsync(object message);
}

• void return — fire-and-forget by design, cannot be awaited
• message is serialized to JSON via System.Text.Json with JsonSerializerDefaults.Web (camelCase), using runtime type (not declared type)
• null messages are silently ignored
• Kafka delivery happens in the background, never blocks the caller
• Serialization or delivery errors are silently swallowed

ILogTransport

public interface ILogTransport
{
    Task SendAsync(string payload);
}

• payload is the JSON-serialized string of the log event
• Implement this interface to create a custom transport

ServiceCollectionExtensions

// Register with default KafkaTransport
services.AddKafkaProducer(configuration.GetSection("Kafka"));

// Register with custom transport
services.AddKafkaProducer<TTransport>(configuration.GetSection("Kafka"));

Failure Behavior

The library is designed for best-effort delivery. It should be used for logging, audit trails, and non-critical events — not for transactions that require guaranteed delivery.

Publishing

See PUBLISHING.md for instructions on building and publishing the NuGet package.

License

MIT