Flowthru.Extensions.AWS.S3 0.28.0

Flowthru.Extensions.AWS.S3

Read and write Flowthru Catalog Items against s3:// URIs. This package registers an IStorageMediumProvider for the s3:// scheme, so any format extension (Csv, Parquet, Json, Xml, …) works over S3 with no change — a Flow targets S3 by writing an s3://bucket/key path on an Item, nothing more.

Mental model

Storage in Flowthru is three independent axes: format (how bytes serialize) × medium (where bytes live) × container (the in-memory shape). This package adds one medium. It does not know about CSV or Parquet, and CSV/Parquet do not know about S3 — they meet through the medium resolver. Adding S3 is purely additive.

Install

dotnet add package Flowthru.Extensions.AWS.S3

Register the medium, then target S3 by writing an s3:// path anywhere a path is accepted:

services.AddFlowthru(b =>
{
    b.UseS3();                       // credentials via the standard AWS chain
    b.RegisterCatalog(sp => new Catalog(
        basePath, sp.GetRequiredService<IStorageMediumResolver>()));
});

// Anywhere a path is accepted:
ItemFactory.Enumerable.Csv<Order>("orders", "s3://my-bucket/raw/orders.csv", resolver);

Bare paths and file:// still resolve to the local filesystem; only s3:// routes here.

Credentials

The extension owns no credentials. The AWS-backed gateway builds its client with no explicit credentials, so the AWS SDK resolves them through its standard chain — environment variables (AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_SESSION_TOKEN), the shared profile (~/.aws/credentials), or an ECS/EC2 instance role. Flowthru never loads, stores, or sees a secret.

Configuration (Flowthru:S3)

b.UseS3(s3 =>
{
    s3.Region = "us-west-2";
    s3.ServiceUrl = "http://localhost:9000";  // MinIO / LocalStack
    s3.ForcePathStyle = true;
});

S3-compatible stores

Setting ServiceUrl + ForcePathStyle targets MinIO, LocalStack, Cloudflare R2, or any S3-API endpoint. Note: AWS SDK v4 sends request checksums by default; some older S3-compatible servers reject them — upgrade the server or pin a compatible SDK if you hit checksum errors. (AWS itself is unaffected.)

Local development without an AWS account

UseLocalS3 swaps in a shipped file-backed stub — a fully offline stand-in for S3, no account, credentials, or network. Each object lands at {root}/{bucket}/{key}, so the directory tree is an inspectable record of the "bucket".

b.UseLocalS3("/tmp/s3-stub");   // s3://demo/out.json → /tmp/s3-stub/demo/out.json

Same provider, same medium, same Flow — only the gateway changes. This is the swap point that also lets tests run offline. For shared or production storage, use UseS3().

Behavior notes

• Atomic writes — WriteStream is a single object PUT, all-or-nothing at the object level; a failed write never leaves a partial object.
• Pre-flight write-probe — InspectTarget() PUTs and deletes a zero-byte sentinel beside the target key, so a missing bucket or denied s3:PutObject fails at pre-flight with a WriteAccessDenied diagnostic, not at runtime. (It briefly creates an object, which can fire bucket notifications — the same trade-off the filesystem medium's sentinel probe makes.)
• Caching — S3-backed Items are fingerprintable via the object ETag (one HEAD request, no body transfer), so they participate in Flowthru's cache plan.