Sa.HybridFileStorage 0.20.0

dotnet add package Sa.HybridFileStorage --version 0.20.0
                    
NuGet\Install-Package Sa.HybridFileStorage -Version 0.20.0
                    
This command is intended to be used within the Package Manager Console in Visual Studio, as it uses the NuGet module's version of Install-Package.
<PackageReference Include="Sa.HybridFileStorage" Version="0.20.0" />
                    
For projects that support PackageReference, copy this XML node into the project file to reference the package.
<PackageVersion Include="Sa.HybridFileStorage" Version="0.20.0" />
                    
Directory.Packages.props
<PackageReference Include="Sa.HybridFileStorage" />
                    
Project file
For projects that support Central Package Management (CPM), copy this XML node into the solution Directory.Packages.props file to version the package.
paket add Sa.HybridFileStorage --version 0.20.0
                    
#r "nuget: Sa.HybridFileStorage, 0.20.0"
                    
#r directive can be used in F# Interactive and Polyglot Notebooks. Copy this into the interactive tool or source code of the script to reference the package.
#:package Sa.HybridFileStorage@0.20.0
                    
#:package directive can be used in C# file-based apps starting in .NET 10 preview 4. Copy this into a .cs file before any lines of code to reference the package.
#addin nuget:?package=Sa.HybridFileStorage&version=0.20.0
                    
Install as a Cake Addin
#tool nuget:?package=Sa.HybridFileStorage&version=0.20.0
                    
Install as a Cake Tool

Sa.HybridFileStorage

English | Русский

Hybrid file storage abstraction with automatic provider failover. Unifies multiple storage backends (FileSystem, S3, PostgreSQL) under a single resilient API — if one provider becomes unavailable, the system switches to another.

Installation

dotnet add package Sa.HybridFileStorage

Table of Contents


Virtual Folders (Baskets) Concept

Sa.HybridFileStorage operates on the idea of virtual folders — called baskets. A basket is a logical, string-named scope that sits above physical storage backends. From your application's perspective, you work with simple folder names like "drafts", "documents", or "archive". Behind each basket name lies a storage provider (or a list of providers) that organise data differently.

How baskets map to storage

Each basket is backed by one or more IFileStorage providers. The same basket name can be served by different physical systems, and you can combine multiple providers for redundancy or tiered storage:

Basket name Physical backend Organisation Example File ID
drafts FileSystem (fs://) Plain directory tree on disk fs://drafts/42/notes.txt
documents PostgreSQL (pg://) Relational table with date partitioning pg://documents/42/1751347200/contract.pdf
archive S3 / MinIO (s3://) Cloud bucket with flat namespace s3://archive/42/old-report.zip

Behind every basket may hide a single storage or a list of stores — the hybrid layer manages failover transparently. You never need to know which physical system holds your file; you only reference it by its File ID.

Configuring baskets to backends

Each provider binds to exactly one basket, set through its own options. Every provider registers itself with its own Add... method and is picked up by the container automatically:

// Basket "drafts" → local filesystem. Registers its own IFileStorage; the root and the file
// I/O live on a temp folder registered under the same name.
builder.Services.AddSaFileSystemFileStorage("drafts", b => b
    .Options(ob => ob.Configure(x => x.Basket = "drafts"))
    .TempFolder(tb => tb.Options(ob => ob.Configure(folder =>
        folder.RootPath = @"C:\data\drafts"))));

// Basket "documents" → PostgreSQL with auto-partitioning. The shared data source comes from
// AddNpgsqlDataSource(...) unless the storage pins its own. One key per storage, and each keyed
// storage owns its table (plus its own migration and retention schedules).
builder.Services.AddSaPostgreSqlFileStorage("documents", b => b
    .Options(ob => ob.Configure(x =>
    {
        x.Basket = "documents";
        x.TableName = "documents_files";
    })));

// Basket "archive" → S3 cloud storage. The call registers the storage's own bucket client.
builder.Services.AddSaS3FileStorage(o => o.Options(ob => ob.Configure(x =>
{
    x.Endpoint = "http://minio:9000";
    x.AccessKey = "ROOTUSER";
    x.SecretKey = "ChangeMe123";
    x.Bucket = "company-archive";
    x.Basket = "archive";
})));

// Build the hybrid container once: it picks up every IFileStorage registered above.
builder.Services.AddSaHybridFileStorage(cfg => cfg.AddLogging());

Every Add... call registers one storage, and the container collects all of them, so several baskets (and several storages in one basket) can be wired before a single AddSaHybridFileStorage call. ConfigureStorage is only needed for a custom IFileStorage implementation.

Once configured, all CRUD operations use basket names — not provider specifics:

// Upload to the "drafts" virtual folder — goes to the filesystem automatically
var result = await storage.UploadAsync("drafts", input, stream, ct);
// File ID: fs://drafts/42/my-notes.txt

// Download from "documents" — routed to PostgreSQL transparently
await storage.DownloadAsync(result.FileId, processStream, ct);

// Copy from "drafts" to "archive" — crosses FS → S3 boundary seamlessly
await storage.CopyToBasketAsync(result.FileId, "archive", ct: ct);

Supported Storage Providers

Provider Package Registration Use Case
In-Memory Sa.HybridFileStorage AddSaInMemoryFileStorage() Testing, ephemeral scenarios
File System Sa.HybridFileStorage.FileSystem AddSaFileSystemFileStorage(...) Local development, on-premise deployments
S3 Compatible Sa.HybridFileStorage.S3 AddSaS3FileStorage(...) Cloud storage (AWS S3, MinIO, etc.)
PostgreSQL Sa.HybridFileStorage.Postgres AddSaPostgreSqlFileStorage(...) Database-embedded files, transactional consistency, partitioning

Each provider's Add... method registers an IFileStorage, and the hybrid container picks every registered IFileStorage up automatically. The provider classes themselves are internal — construct them through their Add... method, not with new.


Key Features

  • ✅ Unified API — Single IHybridFileStorage interface for all storage providers
  • ✅ Basket-Tenant isolation — Multi-tenant support with scoped storage containers
  • ✅ Failover — Automatic provider switching when one backend fails
  • ✅ Streaming support — Memory-efficient file transfers via Stream
  • ✅ Native AOT ready — Full compatibility with .NET 10 Native AOT
  • ✅ Batch operations — Bulk file processing with configurable parallelism
  • ✅ Interceptors — Upload/download/delete lifecycle hooks for cross-cutting concerns
  • ✅ Read-only mode — Protect storage from accidental modifications

File ID Format

All files are identified using a unified URI-like format:

{storageType}://{basket}/{tenantId}/{path}

Each storage provider adds its own path depth:

Provider File ID Example Path Structure
In-Memory mem://share/42/document.pdf {basket}/{tenantId}/{fileName}
File System fs://documents/100/report.xlsx {basket}/{tenantId}/{fileName}
S3 s3://uploads/7/invoice.csv {basket}/{tenantId}/{fileName}
PostgreSQL pg://files/12/1751347200/photo.jpg {basket}/{tenantId}/{unixTimestamp}/{fileName}

Note: PostgreSQL includes a Unix timestamp in the path because it partitions by date. Other providers omit the timestamp.

Parsing File IDs

Use the static FileIdParser utility:

if (FileIdParser.TryParse("pg://files/42/1751347200/report.pdf", out var basket, out var tenantId, out var timestamp, out var fileName))
{
    Console.WriteLine($"Basket={basket}, Tenant={tenantId}, TS={timestamp}, Name={fileName}");
    // Basket=files, Tenant=42, TS=1751347200, Name=report.pdf
}

// The parameterless layout is guessed: a numeric second segment is read as the Postgres
// timestamp. State the provider's layout when it is known — otherwise a filesystem name with a
// numeric directory (`fs://documents/100/2024/report.pdf`) loses that directory from `fileName`.
if (FileIdParser.TryParse(
    "fs://documents/100/2024/report.pdf",
    FileIdFormat.WithoutTimestamp,
    out _, out var tenantId, out _, out var fullName))
{
    Console.WriteLine($"Tenant={tenantId}, Name={fullName}");   // Tenant=100, Name=2024/report.pdf
}

Quick Start

Registration (Generic Host)

There is no public non-DI constructor: the hybrid storage is created by AddSaHybridFileStorage, and every IFileStorage registered before it joins the failover chain in registration order.

using Microsoft.Extensions.Hosting;
using Sa.HybridFileStorage;
using Sa.HybridFileStorage.Domain;
using Sa.HybridFileStorage.FileSystem;
using Sa.HybridFileStorage.S3;
using System.Text;

var builder = Host.CreateApplicationBuilder(args);

// 1. Providers register themselves. Each call adds an IFileStorage — and, for the filesystem,
//    its own keyed temp folder. Several providers (and several baskets) can coexist.
builder.Services
    .AddSaInMemoryFileStorage()
    .AddSaFileSystemFileStorage("files", b => b
        .Options(ob => ob.Configure(x => x.Basket = "documents"))
        .TempFolder(tb => tb.Options(ob => ob.Configure(folder =>
            folder.RootPath = @"C:\data\files"))))
    .AddSaS3FileStorage(o => o.Options(ob => ob.Configure(x =>
    {
        x.Endpoint = "http://localhost:9000";
        x.AccessKey = "ROOTUSER";
        x.SecretKey = "ChangeMe123";
        x.Bucket = "mybucket";
        x.Basket = "uploads";
    })));

// 2. Build the hybrid container (one call; a second one throws). Logging interceptors are optional.
builder.Services.AddSaHybridFileStorage(cfg => cfg.AddLogging());

var host = builder.Build();
var storage = host.Services.GetRequiredService<IHybridFileStorage>();

// 3. Upload — routed to the first writable provider of the basket ("share" → in-memory).
using var stream = new MemoryStream("Hello, HybridFileStorage!"u8.ToArray());
var result = await storage.UploadAsync(
    basket: "share",
    input: new UploadFileInput { FileName = "hello.txt", TenantId = 42 },
    fileStream: stream,
    cancellationToken: ct);

Console.WriteLine(result.FileId);  // mem://share/42/hello.txt

// 4. Download — the stream is handed to your callback, never buffered by the hybrid layer.
bool wasFound = await storage.DownloadAsync(result.FileId, async (fileStream, token) =>
{
    using var reader = new StreamReader(fileStream, Encoding.UTF8);
    Console.WriteLine(await reader.ReadToEndAsync(token));   // Hello, HybridFileStorage!
}, ct);

// 5. Delete — a write, so read-only providers are skipped.
bool deleted = await storage.DeleteAsync(result.FileId, ct);

IHybridFileStorage is a singleton, so inject it into your own services instead of resolving it inside handlers. Registration stays in one place: the hybrid layer is not additive, and a second AddSaHybridFileStorage call throws.

Minimal DI registration

For quick setups, each provider has its own extension method:

// In-Memory only
builder.Services.AddSaInMemoryFileStorage();

// File System only
builder.Services.AddSaFileSystemFileStorage("files", b => b
    .Options(ob => ob.Configure(x => x.Basket = "documents"))
    .TempFolder(tb => tb.Options(ob => ob.Configure(folder =>
        folder.RootPath = @"C:\data\files"))));

// S3 only
builder.Services.AddSaS3FileStorage(o => o.Options(ob => ob.Configure(x =>
{
    x.Endpoint = "http://localhost:9000";
    x.AccessKey = "ROOTUSER";
    x.SecretKey = "ChangeMe123";
    x.Bucket = "mybucket";
    x.Basket = "uploads";
})));

// Then register the hybrid layer
builder.Services.AddSaHybridFileStorage(cfg => cfg.AddLogging());

CRUD Examples

Upload

Upload accepts a basket name (scope/container), metadata, and a Stream. The hybrid layer finds a writable provider matching the basket and uploads the file.

// Upload from a Stream
using var stream = File.OpenRead(@"C:\temp\document.pdf");
var result = await storage.UploadAsync(
    basket: "documents",
    input: new UploadFileInput { FileName = "document.pdf", TenantId = 42 },
    fileStream: stream,
    cancellationToken: ct);

Console.WriteLine($"Uploaded: {result.FileId}");
// Output: fs://documents/42/document.pdf

Copy a local file directly:

var result = await storage.CopyFromFileAsync(
    filePath: @"C:\temp\image.png",
    basket: "images",
    input: new UploadFileInput { FileName = "avatar.png", TenantId = 7 },
    ct: ct);

Download

Download delegates the file stream to a callback Func<Stream, CancellationToken, Task>. This avoids loading the entire file into memory.

// Process stream inline
bool found = await storage.DownloadAsync(result.FileId, async (stream, token) =>
{
    using var reader = new StreamReader(stream, Encoding.UTF8);
    string content = await reader.ReadToEndAsync(token);
    Console.WriteLine(content);
}, ct);

// Copy to another stream
using var destination = new FileStream(@"C:\output\copy.pdf", FileMode.Create);
await storage.DownloadAsync(result.FileId, async (source, token) =>
    await source.CopyToAsync(destination, 81920, token),
    ct);

Delete

bool deleted = await storage.DeleteAsync(result.FileId, ct);
if (deleted)
    Console.WriteLine("File removed.");
else
    Console.WriteLine("File not found.");

Get Metadata

var metadata = await storage.GetMetadataAsync(result.FileId, ct);
if (metadata != null)
{
    Console.WriteLine($"Basket: {metadata.Basket}");
    Console.WriteLine($"Tenant: {metadata.TenantId}");
    Console.WriteLine($"Name:   {metadata.FileName}");
    Console.WriteLine($"Type:   {metadata.StorageType}");
}

Cross-Basket Copying

Move or duplicate files between baskets (even across different providers):

// Copy within same basket
var copied = await storage.CopyToBasketAsync(
    fileId: "fs://documents/42/report.pdf",
    basket: "archive",
    ct: ct);

// Customise upload metadata during copy
var renamed = await storage.CopyToBasketAsync(
    fileId: "fs://documents/42/report.pdf",
    basket: "backup",
    configure: meta => new UploadFileInput
    {
        TenantId = meta.TenantId,
        FileName = $"renamed-{meta.FileName}"  // change the name
    },
    ct: ct);

Batch Operations

High-throughput parallel file operations with progress reporting and error handling:

// Batch copy with parallelism
var batchResult = await storage.CopyToScopeBatchAsync(
    fileIds:
    [
        "fs://documents/1/a.pdf",
        "fs://documents/2/b.pdf",
        "s3://uploads/3/c.pdf",
    ],
    basket: "archive",
    options: new BatchOptions
    {
        MaxDegreeOfParallelism = 8,
        ContinueOnError = true,
        OperationTimeout = TimeSpan.FromSeconds(30),
        Progress = new Progress<BatchOperationProgress>(p =>
        {
            Console.WriteLine($"{p.Completed}/{p.Total} — OK:{p.Succeeded} Fail:{p.Failed}");
        })
    },
    ct: ct);

foreach (var ok in batchResult.Succeeded)
    Console.WriteLine($"Copied: {ok.FileId}");

foreach (var err in batchResult.Failed)
    Console.WriteLine($"Failed #{err.Index}: {err.FileId} — {err.Exception.Message}");

// Or throw on any failure
batchResult.ThrowIfHasErrors();  // throws BatchOperationException<StorageResult>

BatchResult<T>

Member Type Description
Succeeded IReadOnlyList<T> Successful results
Failed IReadOnlyList<BatchError> Errors with file ID and exception
Total int Total items processed
HasErrors bool Whether any operation failed
ThrowIfHasErrors() void Throws BatchOperationException<T> if failures occurred

BatchOptions

Property Description Default
MaxDegreeOfParallelism Concurrent operations 4
ContinueOnError Keep going after individual failures true
OperationTimeout Per-operation timeout (0 = infinite) 0
Progress IProgress<BatchOperationProgress> reporter null

Interceptors

Lifecycle hooks for upload/download/delete operations. Implement one of the three interfaces:

public interface IUploadInterceptor
{
    // Return false to reject the upload
    ValueTask<bool> CanUploadAsync(IFileStorage storage, UploadFileInput input, Stream fileStream, CancellationToken ct);
    ValueTask AfterUploadAsync(IFileStorage storage, StorageResult result, CancellationToken ct);
    ValueTask OnUploadErrorAsync(IFileStorage storage, Exception exception, CancellationToken ct);
}

public interface IDownloadInterceptor { /* CanDownloadAsync / AfterDownloadAsync / OnDownloadErrorAsync */ }
public interface IDeleteInterceptor  { /* CanDeleteAsync / AfterDeleteAsync / OnDeleteErrorAsync */ }

Example — reject uploads of specific filenames:

public class DeniedExtensionInterceptor : IUploadInterceptor
{
    private static readonly HashSet<string> DeniedExtensions = ["exe", "bat", "cmd"];

    public ValueTask<bool> CanUploadAsync(IFileStorage storage, UploadFileInput input, Stream fileStream, CancellationToken ct)
    {
        var ext = Path.GetExtension(input.FileName)?.TrimStart('.').ToLowerInvariant();
        return ValueTask.FromResult(!DeniedExtensions.Contains(ext));
    }

    public ValueTask AfterUploadAsync(IFileStorage storage, StorageResult result, CancellationToken ct)
        => ValueTask.CompletedTask;

    public ValueTask OnUploadErrorAsync(IFileStorage storage, Exception exception, CancellationToken ct)
        => ValueTask.CompletedTask;
}

Register interceptors through the fluent builder:

builder.Services.AddSaHybridFileStorage(cfg => cfg
    .ConfigureInterceptors((sp, container) =>
    {
        container.AddUploadInterceptor(new DeniedExtensionInterceptor());
        container.AddDownloadInterceptor(new LoggingDownloadInterceptor());
    }));

Built-in logging interceptors are available via .AddLogging().


Read-Only Mode

Set IsReadOnly = true on any provider to prevent writes. Attempted writes throw HybridFileStorageWritableException:

builder.Services.AddSaFileSystemFileStorage("readonly", b => b
    .Options(ob => ob.Configure(x =>
    {
        x.IsReadOnly = true; // uploads/deletes will fail
    }))
    .TempFolder(tb => tb.Options(ob => ob.Configure(folder =>
        folder.RootPath = @"C:\readonly\data"))));

Settings Reference

FileSystemStorageOptions

Property Description Default
Basket Scope/container name "share"
StorageType Scheme prefix in File ID "fs"
IsReadOnly Prevent writes false

The storage root and the file-I/O policy are configured on the keyed temp folder through the mandatory TempFolder(...) channel (RootPath, MaxAge, TouchDebounce, ...). The registration is named-only: AddSaFileSystemFileStorage(name, b => b…TempFolder(…)). See Sa.HybridFileStorage.FileSystem/Readme.md for pre/post-initialisation, configuration binding, and migration from BasePath.

S3FileStorageOptions

Property Description Default
Endpoint S3 endpoint URL (required)
AccessKey S3 access key (required)
SecretKey S3 secret key (required)
Bucket Bucket name (required)
Basket Scope/container name "share"
Region Region for SigV4 signing "eu-central-1"
StorageType Scheme prefix in File ID "s3"
IsReadOnly Prevent writes false
TotalRequestTimeout Per-request timeout 180 sec
ConnectionPoolLifetime Connection pool lifetime 15 min
HandlerLifetime HttpClient handler lifetime ∞ (infinite)

Configured through the standard options pipeline — see Sa.HybridFileStorage.S3/Readme.md for pre/post-initialisation, configuration binding and validation.

PostgresFileStorageOptions

Flat options (no nested PartOptions/CleanupOptions/StorageOptions):

Property Description Default
SchemaName PostgreSQL schema (auto-detected from search_path if not set) "public"
TableName Table for file data "files"
StorageType Scheme prefix in File ID "pg"
Basket Scope/container name "share"
PgPartBy Partitioning granularity PgPartBy.Day
MigrationScheduleForwardDays Days ahead to pre-create partitions 2
ExpireDays Auto-cleanup threshold (days) 365 * 3
IsReadOnly Prevent writes false

InMemoryFileStorageOptions

Property Description Default
Basket Scope/container name "share"
MaxSizeBytes Total byte limit (0 = unlimited) 1 GB
IsReadOnly Prevent writes false

Domain Types

StorageResult

Returned by upload operations. Contains the canonical File ID and a publicly accessible URL.

public sealed record StorageResult(
    string FileId,          // e.g. "fs://documents/42/report.pdf"
    string AbsoluteUrl,     // e.g. "C:\data\files\documents\42\report.pdf"
    string StorageType,     // e.g. "fs", "s3", "pg", "mem"
    DateTimeOffset UploadedAt);

UploadFileInput

Input metadata for uploads.

public sealed record UploadFileInput
{
    public int TenantId { get; init; }              // defaults to 0
    public string FileName { get; init; } = "";     // required at validation
    public static UploadFileInput Empty { get; }    // pre-created empty instance
}

FileMetadata

Read-only metadata retrieved via GetMetadataAsync.

public sealed class FileMetadata
{
    public required string Basket { get; init; }
    public required string FileName { get; init; }
    public int TenantId { get; init; }
    public required string StorageType { get; init; }
}

Exceptions

Exception When thrown
HybridFileStorageNoAvailableException No storage is available for the operation: no provider is registered for the basket, no storage recognizes the file ID, or every candidate was vetoed by an interceptor
HybridFileStorageWritableException A write (upload or delete) was attempted while every candidate storage is read-only
HybridFileStorageAggregateException Provider errors were collected during failover and no storage produced an answer
BatchOperationException<T> Thrown by BatchResult.ThrowIfHasErrors() when at least one batch item failed

License

MIT

Product Compatible and additional computed target framework versions.
.NET net10.0 is compatible.  net10.0-android was computed.  net10.0-browser was computed.  net10.0-ios was computed.  net10.0-maccatalyst was computed.  net10.0-macos was computed.  net10.0-tvos was computed.  net10.0-windows was computed. 
Compatible target framework(s)
Included target framework(s) (in package)
Learn more about Target Frameworks and .NET Standard.

NuGet packages (3)

Showing the top 3 NuGet packages that depend on Sa.HybridFileStorage:

Package Downloads
Sa.HybridFileStorage.Postgres

File storage management in Pg

Sa.HybridFileStorage.FileSystem

File storage management

Sa.HybridFileStorage.S3

File storage management in S3

GitHub repositories

This package is not used by any popular GitHub repositories.

Version Downloads Last Updated
0.20.0 102 11/13/2026
0.18.0 124 11/3/2026
0.17.0 127 10/29/2026
0.16.0 130 10/14/2026
0.15.0 132 10/2/2026
0.12.0 171 7/28/2026
0.11.0 155 7/14/2026
0.10.0 170 7/2/2026
0.9.1 165 5/8/2026
0.9.0 153 4/29/2026
0.8.3 174 3/11/2026
0.8.1 167 3/6/2026
0.8.0 158 2/10/2026
0.7.0 161 1/27/2026
0.6.0 154 1/15/2026
0.5.0 169 12/29/2025
0.4.0 344 12/17/2025
0.3.0 234 10/17/2025
0.1.0 293 9/1/2025
0.0.3 248 6/16/2025
Loading failed