CosmoS3 2.3.0

CosmoS3

CosmoS3 is an Amazon S3–compatible object-storage middleware library for CosmoApiServer. It implements core S3 operations with a pluggable SQL metadata store — SQL Server, PostgreSQL, MySQL, SQLite, or the embedded CosmoKv engine — and the local disk (or a pluggable storage driver) for object data.

It can be consumed three ways: as a standalone S3 endpoint (CosmoS3.Host), as middleware inside an existing CosmoApiServer app, or embedded in-process via the IS3Repository data layer.

Table of Contents

1. Architecture
2. Security & Correctness
3. Performance
4. Recent Updates
5. Quick Start
6. Configuration
7. Database Schema
8. S3 Feature Compatibility
9. Static Website Hosting
10. Presigned URLs
11. Multipart Upload
12. Using with AWS CLI
13. Testing
14. Project Structure

Architecture

┌─────────────────────────────────────────────────────┐
│                   CosmoApiServer                     │
│  (System.IO.Pipelines transport, HTTP/1.1·2·3)       │
└──────────────────────┬───────────────────────────────┘
                       │ IMiddleware
                       ▼
┌──────────────────────────────────────────────────────┐
│                  S3Middleware                         │
│  • S3Context.CreateAsync — async, non-blocking parse  │
│  • Authenticates (SigV4 / SigV2 / presigned)          │
│  • Verifies the SigV4 signature (ValidateSignatures)  │
│  • Routes to ServiceHandler / BucketHandler /         │
│    ObjectHandler / Admin + internal handlers          │
└────────────────┬───────────────────────┬──────────────┘
                 │                       │
     ┌───────────▼───────┐   ┌───────────▼────────────┐
     │   DataAccess       │   │   Storage Driver        │
     │  → IS3Repository   │   │  (DiskStorageDriver:    │
     │  (CosmoSQLClient,  │   │   atomic write + move,  │
     │   5 SQL backends)  │   │   ArrayPool, SubStream) │
     └────────────────────┘   └─────────────────────────┘

Key types:

Security & Correctness

The design choices below are deliberate; each closes a concrete failure mode. They are enforced by the unit + integration test suite (see Testing).

Performance

CosmoS3 is optimized for high-throughput and low-latency workloads. Recent architectural improvements have pushed performance to near-NVMe speeds on local hardware.

Benchmark: CosmoS3 vs. MinIO (Throughput in MiB/s)

Tests were performed locally using the warp S3 benchmark tool with 1MiB and 16MiB object sizes.

Key Optimizations:

• Zero-Allocation I/O: Integrated ArrayPool<byte> across all read/write paths to virtually eliminate transient allocations and GC pressure.
• End-to-End Streaming: Implemented SubStream for range requests, allowing data to be streamed directly from disk to the transport without intermediate buffering.
• Lock-Free Concurrency: Migrated bucket registry to ConcurrentDictionary, enabling non-blocking lookups during high-concurrency requests.
• Metadata Acceleration: composite UNIQUE index on (bucketguid, objectkey, version DESC) for near-instant latest-version lookups, plus a lock-free in-memory bucket registry so the hot HEAD/GET/PUT path skips the DB on a cache hit.

Recent Updates

• SigV4 signature verification: requests are now cryptographically verified (constant-time, ±15-min skew), validated against the official AWS test vector — not just parsed.
• Conditional requests (RFC 7232): If-Match / If-None-Match / If-[Un]Modified-Since on GET/HEAD/PUT (304 / 412, overwrite guard, optimistic concurrency).
• Object versioning: per-key version rows with a UNIQUE (bucket, key, version) index and collision-retry assignment; GET/PUT ?versioning, ListObjectVersions, and x-amz-version-id.
• Bounded-memory listing: ListObjects(V2) streams and pages at the database level instead of loading the whole bucket.
• Single-pass multipart assembly: parts stream directly into the destination blob — no temp-file double-write.
• Hardened defaults: no default admin key, anonymous writes off unless an owner is configured, path-traversal-safe bucket names.
• Five SQL backends: SQL Server, PostgreSQL, MySQL, SQLite, and the embedded CosmoKv engine, behind one backend-portable IS3Repository.
• Fully async request path: S3Context.CreateAsync removed the sync-over-async body buffering (and the thread-pool workaround it required).
• Static website hosting, aws-chunked streaming decode, and a benchmarking suite for comparison against any S3-compatible backend.

Quick Start

1. Run a sample host

A ready-to-run host is included at samples/CosmoS3Host/, with backend-specific variants and demos alongside it:

cd samples/CosmoS3Host          # default
dotnet run

2. Wire CosmoS3 into your own CosmoApiServer app

using CosmoS3;
using CosmoS3.Settings;

var settings = new SettingsBase
{
    RegionString       = "us-east-1",
    ValidateSignatures = false,   // set true in production

    Storage = new StorageSettings
    {
        StorageType   = CosmoS3.Storage.StorageDriverType.Disk,
        DiskDirectory = "./data/objects"
    },

    Database = new DatabaseSettings
    {
        Hostname     = "localhost",
        Port         = 1433,
        DatabaseName = "MyDatabase",
        Username     = "sa",
        Password     = "your-password"
    },

    // Optional: enable CORS for browser-based S3 clients
    Cors = new CorsSettings { Enabled = true },

    // Optional: enable HTTPS
    // CertificatePath     = "./certs/server.pfx",
    // CertificatePassword = "changeme",

    // Optional: enable HTTP/2 cleartext (h2c)
    // EnableHttp2 = true,
};

// CosmoS3Application.Create() wires TLS, HTTP/2, CORS, logging, and S3Middleware.
var app = CosmoS3Application.Create(settings, port: 8100);
app.Run();

Or wire manually for full control over the middleware order:

using CosmoApiServer.Core.Hosting;
using CosmoS3;
using CosmoS3.Settings;

var app = CosmoWebApplicationBuilder.Create()
    .ListenOn(8100)
    .UseHttps("./certs/server.pfx", "changeme")   // optional TLS
    .UseHttp3()                                   // optional HTTP/3 over QUIC
    .UseHttp2()                                    // optional h2c
    .UseCors()                                     // optional CORS
    .UseLogging()
    .UseMiddleware(new S3Middleware(settings))
    .Build();

app.Run();

Configuration

SettingsBase

DatabaseSettings

CosmoS3 selects the backend from DatabaseType; the schema is created/updated at startup via DatabaseFactory.EnsureSchemaAsync.

For SQL Server the built connection string is:

server=HOSTNAME,PORT;database=DBNAME;user id=USER;password=PASS;TrustServerCertificate=true;

SQLite and the embedded CosmoKv engine need no server — point ConnectionString at a file/data-source path.

kvdirect — embedded ordered-KV metadata (no SQL)

kvdirect stores all metadata directly on the CosmoKv ordered-KV engine — the same LSM store the cosmokv backend uses, but without the SQL parser/planner/connection-pool on top. Object keys are byte-ordered (<bucket> 0x00 <key> 0x00 <invVersion>) so the latest version is a single seek and a bucket listing is a native cursor; there is no schema (the default user/credential/bucket/owner-ACL are seeded automatically on first open). Point ConnectionString at a directory:

"Database": { "DatabaseType": "kvdirect", "ConnectionString": "Data Source=./cosmos3-kvdirect" }

It is single-process / single-node by design (one embedded store, no shared DB) — the MinIO-style "just run the binary" deployment. Use a SQL backend (postgres/mssql/mysql) when multiple nodes must share one metadata store. End-to-end it runs ~6.3× faster across PUT/GET/HEAD/DELETE than the same engine via SQL; see benchmarks/results/kvdirect-spike-SUMMARY.md.

StorageSettings

Database Schema

CosmoS3 supports multiple database engines including SQL Server, PostgreSQL, MySQL, and SQLite. The schema is automatically created or updated at startup via DatabaseFactory.EnsureSchemaAsync.

Key Tables

Key Indexes for Performance & Correctness

The schema includes composite indexes that stay fast with millions of objects and also enforce integrity:

• ux_s3_objects_bucket_key_version: UNIQUE (bucketguid, objectkey, version DESC) — serves the common "get latest version" lookup and makes the versioning write race impossible at the storage layer (collisions retry with the next version). On CosmoKv the DESC qualifier is dropped (engine limitation) but the uniqueness constraint is identical.
• idx_s3_objects_guid: (guid) — blob/version resolution by object GUID.
• idx_s3_credentials_accesskey: (accesskey) — rapid authentication.
• idx_s3_buckets_name: (name) — fast bucket resolution.

All database operations go through the static DataAccess facade over the backend-portable IS3Repository (S3Repository), using parameterized SQL — no stored procedures, so the same code runs unchanged on every backend.

S3 Feature Compatibility

Service-Level Operations

Bucket Operations

Object Operations

Notes on Compatibility

• Signature versions: Both SigV4 and SigV2 are supported for authentication and presigned URLs. With ValidateSignatures enabled (default), SigV4 signatures are cryptographically verified, not merely parsed.
• Conditional requests: GET/HEAD return 304 Not Modified / 412 Precondition Failed; PUT supports If-None-Match: * (don't overwrite) and If-Match (optimistic concurrency) per RFC 7232.
• aws-chunked transfer encoding: Automatically decoded via the async body path; works with aws s3 cp for any file size.
• Versioning: Supported. Toggle with PUT ?versioning; when enabled, writes create new version rows, deletes write delete markers, and responses carry x-amz-version-id. List historic versions with list-object-versions.
• Anonymous writes: Off by default — set AnonymousOwnerGuid to a real user GUID to allow public-write buckets.
• Bucket policies / lifecycle / replication: Not implemented. (CORS is configured server-side via CorsSettings, not per-bucket.)

Static Website Hosting

A bucket can be configured to serve static files over plain HTTP (no AWS credentials required).

Configure a bucket for website hosting

# Create bucket
aws --endpoint-url http://localhost:8100 s3 mb s3://my-site

# Upload content
aws --endpoint-url http://localhost:8100 s3 cp index.html  s3://my-site/index.html  --content-type text/html
aws --endpoint-url http://localhost:8100 s3 cp error.html  s3://my-site/error.html   --content-type text/html

# Enable website hosting
aws --endpoint-url http://localhost:8100 s3 website s3://my-site \
    --index-document index.html \
    --error-document error.html

Browse the site

# Bucket root returns index.html
curl http://localhost:8100/my-site/

# Unknown path returns error.html with 404
curl http://localhost:8100/my-site/missing.html

Redirect all requests

aws --endpoint-url http://localhost:8100 s3api put-bucket-website \
    --bucket my-site \
    --website-configuration '{
        "RedirectAllRequestsTo": { "HostName": "example.com", "Protocol": "https" }
    }'

Routing rules

aws --endpoint-url http://localhost:8100 s3api put-bucket-website \
    --bucket my-site \
    --website-configuration '{
        "IndexDocument": { "Suffix": "index.html" },
        "ErrorDocument": { "Key": "error.html" },
        "RoutingRules": [
            {
                "Condition": { "KeyPrefixEquals": "old/" },
                "Redirect":  { "ReplaceKeyPrefixWith": "new/" }
            }
        ]
    }'

How it works:

• Website configuration is stored as website.xml at <DiskDirectory>/<bucketName>/website.xml.
• Requests to a website-enabled bucket without AWS authentication headers are served as static files.
• If the request path ends with /, the index document is served.
• If the object is not found, the error document is returned with HTTP 404.
• Redirect rules are evaluated before object lookup.

Presigned URLs

Presigned URLs grant time-limited access to an S3 object without requiring the caller to have AWS credentials.

Generate a presigned URL (C# SDK)

var request = new GetPreSignedUrlRequest
{
    BucketName = "my-bucket",
    Key        = "my-object.txt",
    Expires    = DateTime.UtcNow.AddMinutes(15),
    Verb       = HttpVerb.GET
};

string url = s3Client.GetPreSignedURL(request);

Use the presigned URL

# Download with curl (no AWS credentials needed)
curl "<presigned-url>" -o downloaded.txt

# Upload with a presigned PUT URL
curl -X PUT "<presigned-put-url>" --data-binary @file.txt

Signature version behavior:

AWSSDK generates SigV2 presigned URLs for custom (non-AWS) endpoints. CosmoS3 validates both:

Expired presigned URLs return HTTP 403 ExpiredToken.

Multipart Upload

Multipart upload allows large files to be uploaded in parts and assembled server-side.

Via AWS CLI (automatic for files > 8 MB by default)

# CosmoS3 handles chunked uploads transparently
aws --endpoint-url http://localhost:8100 \
    s3 cp large-file.bin s3://my-bucket/large-file.bin \
    --expected-size 1073741824   # hint for 1 GB file

Via SDK (manual)

// 1. Initiate upload
var initResponse = await s3.InitiateMultipartUploadAsync(new InitiateMultipartUploadRequest
{
    BucketName  = "my-bucket",
    Key         = "my-object"
});
string uploadId = initResponse.UploadId;

// 2. Upload parts (minimum 5 MB each, except the last)
var uploadPartResponse = await s3.UploadPartAsync(new UploadPartRequest
{
    BucketName   = "my-bucket",
    Key          = "my-object",
    UploadId     = uploadId,
    PartNumber   = 1,
    InputStream  = partStream,
    PartSize     = partStream.Length
});

// 3. Complete the upload
await s3.CompleteMultipartUploadAsync(new CompleteMultipartUploadRequest
{
    BucketName = "my-bucket",
    Key        = "my-object",
    UploadId   = uploadId,
    PartETags  = new List<PartETag> { new PartETag(1, uploadPartResponse.ETag) }
});

Internals:

• Each uploaded part is written to a part file under TempDirectory and hashed on arrival.
• CompleteMultipartUpload streams the ordered part files directly into the destination blob via ConcatReadStream (single pass — no concatenate-to-temp-then-recopy), then deletes the parts. The object length is the sum of the recorded part lengths.
• Active sessions and their parts are tracked in s3_uploads / s3_uploadparts and can be listed or aborted.

Using with AWS CLI

Configure the AWS CLI for local use

aws configure
# AWS Access Key ID:     default
# AWS Secret Access Key: default
# Default region name:   us-east-1
# Default output format: json

Common commands

ENDPOINT=http://localhost:8100

# List buckets
aws --endpoint-url $ENDPOINT s3 ls

# Create bucket
aws --endpoint-url $ENDPOINT s3 mb s3://my-bucket

# Upload file
aws --endpoint-url $ENDPOINT s3 cp file.txt s3://my-bucket/

# Download file
aws --endpoint-url $ENDPOINT s3 cp s3://my-bucket/file.txt ./

# List objects
aws --endpoint-url $ENDPOINT s3 ls s3://my-bucket/

# Sync directory
aws --endpoint-url $ENDPOINT s3 sync ./local-dir/ s3://my-bucket/prefix/

# Delete object
aws --endpoint-url $ENDPOINT s3 rm s3://my-bucket/file.txt

# Delete bucket (must be empty)
aws --endpoint-url $ENDPOINT s3 rb s3://my-bucket

# Tag a bucket
aws --endpoint-url $ENDPOINT s3api put-bucket-tagging \
    --bucket my-bucket \
    --tagging '{"TagSet":[{"Key":"env","Value":"dev"}]}'

# Get bucket tags
aws --endpoint-url $ENDPOINT s3api get-bucket-tagging --bucket my-bucket

# Get bucket website config
aws --endpoint-url $ENDPOINT s3api get-bucket-website --bucket my-bucket

# Presigned URL (60 seconds)
aws --endpoint-url $ENDPOINT s3 presign s3://my-bucket/file.txt --expires-in 60

Testing

The suite (tests/CosmoS3.Tests/) uses xUnit + AWSSDK.S3 and needs no external database or running server — it boots the real server in-process against an embedded CosmoKv DB.

Tests are split into two xUnit traits that must run as separate dotnet test invocations, because the server's data layer is a process-global singleton (DataAccess._repo), so only one server instance can exist per process:

# Server-free unit tests (signature math, helpers, conditional-header logic, ...)
dotnet test tests/CosmoS3.Tests --filter "Category=Unit"

# Integration tests — boot the in-process server, drive it through the AWS SDK
dotnet test tests/CosmoS3.Tests --filter "Category=Integration"

CI runs the two lanes as separate steps (.github/workflows/ci.yml). Current status: 62 unit + 12 integration, all passing.

Fixtures

• CosmoServerFixture (integration) boots the server with CosmoWebApplication.RunAsync on an ephemeral port over a temp CosmoKv database, and hands tests a configured AmazonS3Client (NewS3Client()). Shared via the [Collection("CosmoServer")] xUnit collection.
• S3Fixture is the legacy fixture that targets an externally running endpoint, retained for the original end-to-end tests.

Representative coverage

Project Structure

src/CosmoS3/
├── S3Middleware.cs          # IMiddleware entry point; async parse, SigV4 verify, routing
├── S3Request.cs             # HTTP → S3 request parsing (CreateAsync, aws-chunked decode)
├── S3Response.cs            # S3-formatted response writer
├── S3Context.cs             # Combined request + response context (CreateAsync factory)
├── S3Exception.cs           # Typed S3 error thrown by handlers
├── SignatureV4Verifier.cs   # Recompute + constant-time compare of the SigV4 signature
├── DataAccess.cs            # Static facade over IS3Repository
├── IS3Repository.cs         # Backend-portable data-layer contract
├── S3Repository.cs          # Parameterized-SQL implementation (all 5 SQL backends)
├── KvDirectRepository.cs    # Embedded ordered-KV implementation (no SQL — DatabaseType=kvdirect)
├── DatabaseFactory.cs       # Backend selection + schema bootstrap
├── CosmoS3Application.cs     # Turn-key host builder (TLS / HTTP2·3 / CORS / middleware)
├── Settings/                # SettingsBase, Database/Storage/Logging/Debug/Cors settings
├── Helpers/
│   ├── SignatureV4.cs       # SigV4 canonical-request + signing-key primitives
│   ├── SecurityHelper.cs    # FixedTimeEquals, ConfiguredKeyMatches, IsValidBucketName
│   ├── ConditionalHeaders.cs# RFC 7232 If-Match / If-None-Match / If-[Un]Modified-Since
│   ├── ConcatReadStream.cs  # Single-pass multipart assembly stream
│   ├── AclConverter.cs      # ACL ⇆ policy conversion (grantees validated vs DB)
│   ├── RequestValidator.cs  # Shared handler precondition checks
│   └── HashHelper.cs        # Single-pass MD5/SHA hashing
├── Classes/
│   ├── AuthManager.cs       # Credential/user resolution + authorization decision
│   ├── BucketManager.cs     # Lock-free ConcurrentDictionary bucket registry
│   ├── BucketClient.cs      # Per-bucket storage driver accessor
│   ├── ConfigManager.cs     # User / credential / bucket / object lookup
│   └── CleanupManager.cs    # Background task: expire stale temp files
├── Api/S3/
│   ├── ApiHandler.cs        # Top-level dispatcher (service/bucket/object)
│   ├── ServiceHandler.cs    # ListBuckets
│   ├── BucketHandler.cs     # Bucket operations (incl. versioning, website, ACL, tags)
│   ├── ObjectHandler.cs     # Object operations + multipart + conditional PUT
│   └── ApiHelper.cs         # Shared XML response helpers
├── Storage/
│   ├── StorageDriverBase.cs
│   └── DiskStorageDriver.cs # Atomic temp-then-move writes; ArrayPool + SubStream I/O
├── Schema/                  # Per-backend DDL: mssql · postgres · mysql · sqlite · cosmokv
├── S3Objects/               # XML DTOs (request/response bodies)
└── Logging/
    └── S3Logger.cs          # Console/callback-based logger