Noundry.Streams.Service 1.0.0

Noundry Streams

High-performance, self-contained event stream ingestion and relay service for .NET

Noundry Streams is a lightweight, production-ready service that captures event streams from any source (webhooks, APIs, message queues), persists them durably with configurable retention, and relays them to configured endpoints with built-in retry logic.

Key Features

✅ 50K+ messages/second throughput capability ✅ Sub-millisecond ingestion latency (p99 < 1ms) ✅ Zero message loss with durable SQLite persistence ✅ Crash-safe with WAL mode and configurable retention ✅ Built-in retry logic with exponential backoff ✅ Multi-stream support - one service instance handles many streams ✅ OpenTelemetry integration for observability ✅ Single executable deployment with minimal dependencies ✅ Cross-platform - Linux, macOS, Windows

Quick Start

1. Run the Service

# Download or build the binary
./NoundryStreams.Service

# Service starts on http://localhost:5000

2. Create a Stream

curl -X POST http://localhost:5000/admin/streams \
  -H "Content-Type: application/json" \
  -d '{
    "streamId": "webhook-events",
    "displayName": "Webhook Events",
    "retentionHours": 72
  }'

3. Configure Relay Destination

curl -X POST http://localhost:5000/admin/streams/webhook-events/relays \
  -H "Content-Type: application/json" \
  -d '{
    "endpointUrl": "https://my-backend.com/process",
    "retryAttempts": 3
  }'

4. Ingest Messages

# External systems POST data to the stream
curl -X POST http://localhost:5000/streams/webhook-events/ingest \
  -H "Content-Type: application/json" \
  -d '{"event": "user.signup", "userId": 12345}'

# Response: 202 Accepted (< 1ms)

That's it! Noundry Streams automatically:

• Accepts the message (< 1ms response)
• Persists to SQLite database
• Relays to configured endpoint(s)
• Retries on failure
• Deletes after retention period

What It Does

Noundry Streams acts as a durable buffer between event sources and your processing systems:

┌─────────────────┐       ┌──────────────────┐       ┌─────────────────┐
│  Event Sources  │──────▶│  Noundry Streams │──────▶│   Your Backend  │
│  (Webhooks,     │       │  • Ingest (< 1ms)│       │   • Process     │
│   Kafka, APIs)  │       │  • Persist (WAL) │       │   • Transform   │
└─────────────────┘       │  • Relay + Retry │       │   • Store       │
                          └──────────────────┘       └─────────────────┘

Benefits:

• Decouples sources from destinations
• Handles transient failures automatically
• Provides replay capability (configurable retention)
• Simplifies webhook ingestion
• Enables audit logging and compliance

Architecture

• ASP.NET Core Minimal APIs - Ultra-lightweight HTTP server
• System.Threading.Channels - Lock-free, high-performance queuing
• SQLite with WAL mode - Durable persistence (survives crashes)
• Tuxedo ORM - Drop-in Dapper replacement for blazing-fast data access
• Batched writes - Configurable batch size and interval
• Worker-based relays - Configurable parallelism per stream
• OpenTelemetry - Optional distributed tracing

Configuration

Edit appsettings.json:

{
  "NoundryStreams": {
    "DataPath": "./data/noundry-streams.db",
    "ApiKey": "your-secret-key",
    "Defaults": {
      "ChannelCapacity": 10000,
      "BatchIntervalMs": 50,
      "BatchMaxSize": 500,
      "RetentionHours": 168,
      "RelayWorkerCount": 4
    }
  }
}

See CONFIGURATION.md for all options.

API Reference

Ingest Endpoints

• POST /streams/{streamId}/ingest - Ingest a message
• POST /ingest?stream={streamId} - Alternative with query param

Admin Endpoints

• POST /admin/streams - Create a stream
• GET /admin/streams - List all streams
• GET /admin/streams/{streamId} - Get stream details
• POST /admin/streams/{streamId}/relays - Add relay config
• GET /admin/streams/{streamId}/metrics - Get stream metrics

Health Endpoints

• GET /health - Health check
• GET /health/ready - Readiness check

See API.md for detailed API documentation.

Deployment

Linux (systemd)

./scripts/publish-linux.sh
sudo ./scripts/install-service-linux.sh
sudo systemctl start noundry-streams

Windows (Windows Service)

.\scripts\publish-windows.ps1
.\scripts\install-service-windows.ps1
Start-Service NoundryStreams

Docker

cd examples/docker
docker-compose up -d

See DEPLOYMENT.md for detailed deployment instructions.

Performance

Benchmarks (8-core, 16GB RAM):

• Ingestion: 50K+ msg/sec, < 1ms latency (p99)
• Persistence: 30K+ writes/sec (batched)
• Relay: 10K+ msg/sec per worker
• Memory: ~200MB base + ~1GB cache
• Disk: ~1KB per message (avg)

See PERFORMANCE.md for tuning guide.

Use Cases

• Webhook ingestion - Stripe, GitHub, Twilio webhooks
• Event streaming - High-throughput event pipelines
• Microservices decoupling - Buffer between services
• Audit logging - Durable event log with replay
• API rate limiting - Buffer and relay at controlled rate
• Data replication - Reliable cross-region sync

Requirements

• .NET 9.0 Runtime (included in self-contained builds)
• Linux, macOS, or Windows
• ~200MB RAM minimum
• 10GB+ disk space recommended (depends on retention)

Building from Source

cd Streams
dotnet restore
dotnet build
dotnet run --project src/NoundryStreams.Service

Documentation

• Architecture - System design and internals
• Configuration - All configuration options
• API Reference - Complete API documentation
• Deployment - Production deployment guide
• Performance - Tuning and benchmarks
• Troubleshooting - Common issues and solutions

License

MIT License - see LICENSE file for details.

Contributing

Contributions welcome! Please open an issue or PR.

Support

For issues or questions, please refer to the documentation in the docs/ folder.