DevelopmentHelpers.Storage.Core 12.0.1

DevelopmentHelpers.Storage.Core

A small library to simplify working with Azure Blob Storage, Azure FileShare, and local filesystem directories. It provides an easy-to-use API for uploading, downloading and syncing entire directory structures and individual files. The library supports both local and Azure storage implementations and integrates with dependency injection.

Version 12.0.0 - What's New

Security Enhancements

• Input validation on all public methods using ArgumentException.ThrowIfNullOrWhiteSpace and ArgumentNullException.ThrowIfNull
• CancellationToken support on all async operations for graceful operation cancellation
• ConfigureAwait(false) throughout for better library behavior in UI/ASP.NET contexts

Performance Improvements

• ArrayPool<byte> for memory-efficient chunked uploads - reduces GC pressure for large file transfers
• Optimized async patterns with proper cancellation propagation

API Changes (Breaking)

• All async methods now accept an optional CancellationToken cancellationToken = default parameter
• Interface methods updated: IStorage, IAzureFileStorage, IContainerToContainerSyncHelper, IDirectoryToContainerSyncHelper

Key features

• Unified IStorage interface with implementations for Azure Blob Storage and local filesystem.
• Azure FileShare support via IAzureFileStorage implementation.
• Upload/download entire directories and containers.
• Upload large files in chunks (resumable/large-file friendly) with ArrayPool optimization.
• Container synchronization helpers (container-to-container and directory-to-container).
• Create ZIP archives from containers.
• Easy configuration using appsettings.json and dependency injection.
• Multiple Azure auth modes via AzureAuthMode (DevelopmentStorage, ConnectionString, ManagedIdentity, UserToken).
• Full CancellationToken support for graceful operation cancellation.

Prerequisites

• .NET 10
• Azure Storage account (when using Azure implementations)

Configuration (appsettings.json)

Configuration is bound to DevelopmentHelpers.Storage.Core.Models.AzureConfiguration from the DevelopmentHelpers:AzureConfiguration section. The AuthMode property controls how the library authenticates.

Common structure

"DevelopmentHelpers": {
  "AzureConfiguration": {
    "ConnectionString": "<connection string or UseDevelopmentStorage=true>",
    "AuthMode": "<DevelopmentStorage|ConnectionString|ManagedIdentity|UserToken>",
    "AccountName": "<storage-account-name>",
    "TenantId": "<aad-tenant-id>",
    "ClientId": "<aad-client-id>",
    "ClientSecret": "<aad-client-secret>"
  },
  "AzureFileStorageConfiguration": {
    "ConnectionString": "<file-storage-connection-string>",
    "ShareName": "<share-name>"
  }
}

Only a subset of properties is required for each AuthMode.

AuthMode: DevelopmentStorage (Azurite)

Use the Azurite/emulator connection string:

"DevelopmentHelpers": {
  "AzureConfiguration": {
    "ConnectionString": "UseDevelopmentStorage=true",
    "AuthMode": "DevelopmentStorage"
  }
}

AuthMode: ConnectionString (production account key)

Use a full Azure Storage connection string:

"DevelopmentHelpers": {
  "AzureConfiguration": {
    "ConnectionString": "DefaultEndpointsProtocol=https;AccountName=...;AccountKey=...;EndpointSuffix=core.windows.net",
    "AuthMode": "ConnectionString"
  }
}

AuthMode: ManagedIdentity

Use a managed identity (system-assigned or user-assigned). ConnectionString is not required; AccountName is.

"DevelopmentHelpers": {
  "AzureConfiguration": {
    "AccountName": "<your-storage-account-name>",
    "AuthMode": "ManagedIdentity"
  }
}

AuthMode: UserToken (client credentials)

Use an Azure AD application (client credentials) to access storage via RBAC:

"DevelopmentHelpers": {
  "AzureConfiguration": {
    "AccountName": "<your-storage-account-name>",
    "AuthMode": "UserToken",
    "TenantId": "<your-tenant-id>",
    "ClientId": "<your-client-id>",
    "ClientSecret": "<your-client-secret>"
  }
}

Note: AuthMode can also be omitted; in that case AzureConfiguration will infer DevelopmentStorage when ConnectionString contains UseDevelopmentStorage=true, otherwise it will default to ConnectionString.

Registering services (dependency injection)

The core DI extension methods are defined in DependencyInjection.cs.

1. Register using IConfiguration

This reads the DevelopmentHelpers:AzureConfiguration section and registers IStorage with AzureStorage:

// in Program.cs / Startup.cs
var builder = WebApplication.CreateBuilder(args);

builder.Services.AddAzureBlobStorage(builder.Configuration); // registers IStorage
builder.Services.AddAzureFileStorage(builder.Configuration); // registers IAzureFileStorage

var app = builder.Build();

You can also use the legacy combined registration if you want both IStorage and IAzureFileStorage from a single call:

builder.Services.AddAzureStorage(builder.Configuration);

2. Register using a connection string (blob only)

When you have a connection string at runtime (env var, secret store, etc.):

var builder = WebApplication.CreateBuilder(args);
var connectionString = builder.Configuration["StorageAccountConnectionString"]; // or any source

builder.Services.AddAzureBlobStorage(new AzureConfiguration
{
    ConnectionString = connectionString,
    AuthMode = AzureAuthMode.ConnectionString
});

var app = builder.Build();

3. Register using a strongly-typed AzureConfiguration

This is useful in tests or when you build config in code:

var azureConfig = new AzureConfiguration
{
    ConnectionString = "UseDevelopmentStorage=true",
    AuthMode = AzureAuthMode.DevelopmentStorage
};

builder.Services.AddAzureBlobStorage(azureConfig);

For managed identity:

var azureConfig = new AzureConfiguration
{
    AccountName = "mystorageaccount",
    AuthMode = AzureAuthMode.ManagedIdentity
};

builder.Services.AddAzureBlobStorage(azureConfig);

For user token:

var azureConfig = new AzureConfiguration
{
    AccountName = "mystorageaccount",
    AuthMode = AzureAuthMode.UserToken,
    TenantId = "<tenant-id>",
    ClientId = "<client-id>",
    ClientSecret = "<client-secret>"
};

builder.Services.AddAzureBlobStorage(azureConfig);

4. Register Azure File Storage

Using configuration:

builder.Services.AddAzureFileStorage(builder.Configuration);

Using explicit values:

var fileStorageConnectionString = "DefaultEndpointsProtocol=https;AccountName=...;AccountKey=...;EndpointSuffix=core.windows.net";
var shareName = "my-share";

builder.Services.AddAzureFileStorage(fileStorageConnectionString, shareName);

Basic usage examples

Create storage instances (examples)

// Using DI (recommended)
public class MyService
{
    private readonly IStorage _storage;
    private readonly IAzureFileStorage _fileStorage;

    public MyService(IStorage storage, IAzureFileStorage fileStorage)
    {
        _storage = storage;
        _fileStorage = fileStorage;
    }
}

// Or instantiate manually for quick testing
var azureConfig = new AzureConfiguration
{
    ConnectionString = "UseDevelopmentStorage=true",
    AuthMode = AzureAuthMode.DevelopmentStorage
};
var logger = loggerFactory.CreateLogger<AzureStorage>();
IStorage storage = new AzureStorage(azureConfig, logger);

Uploading and downloading with CancellationToken

// Create a cancellation token for graceful cancellation
using var cts = new CancellationTokenSource(TimeSpan.FromMinutes(30)); // 30 minute timeout

// Upload a single file with cancellation support
var fileInfo = new FileInfo("/path/to/file.txt");
var blobUrl = await _storage.UploadFileAsync(fileInfo, MimeTypes.txt, "my-container", cts.Token);

// Upload a Directory with cancellation support
await _storage.UploadDirectoryAsync(new DirectoryInfo("/path/to/folder"), "my-container", true, cts.Token);

// Download a file (by blob URL) with cancellation support
var downloaded = await _storage.DownloadToFileAsync(blobUrl, "/local/download/path", cts.Token);

// Download entire container to local folder
await _storage.DownloadContainer("my-container", "/local/download/path", cts.Token);

// Create a zip from a container
var zipPath = await _storage.CreateZipFromContainerAsync("my-container", "/tmp", "archive.zip", deleteDownloadedDirectory: true, cts.Token);

Chunked uploads (large files) with ArrayPool optimization

// Upload large file in chunks - now uses ArrayPool for better memory efficiency
using var cts = new CancellationTokenSource();
var uri = await _storage.UploadInChunks(
    fullFilePath, 
    "container-name", 
    "file-name.txt", 
    MimeTypes.txt,
    blockSize: 4 * 1024 * 1024, // 4MB chunks
    cts.Token);

Checking blobs and containers

// Check existence with cancellation support
using var cts = new CancellationTokenSource();
bool exists = await _storage.BlobExistsAsync(blobUrl, cts.Token);

// Create a container if not exists
var response = await _storage.CreateContainerAsync("container-name", publicAccess: false, cts.Token);

// List containers
var containers = await _storage.GetContainersAsync(cts.Token);

Sync helpers with CancellationToken

The library includes helpers for synchronizing between containers or from a directory to a container.

using var cts = new CancellationTokenSource();

// Container to container sync with cancellation support
var syncHelper = new ContainerToContainerSyncHelper(sourceStorage, targetStorage);
var (success, message) = await syncHelper.SyncSourceToTargetContainersAsync(
    "source", 
    "target", 
    allowPublicAccess: false, 
    workingDirectory: "/tmp",
    cts.Token);

// Directory to container sync helper
var directorySyncHelper = new DirectoryToContainerSyncHelper(storage);
var (blobUrls, _) = await directorySyncHelper.GetBlobUrlListExistInTargetButNotInSourceAsync(
    sourceDirectoryPath, 
    "target-container",
    cts.Token);

Azure File Share operations

using var cts = new CancellationTokenSource();

// Upload a file to Azure File Share
var fileInfo = new FileInfo("/path/to/file.txt");
var uri = await _fileStorage.UploadFileAsync(fileInfo, "share-directory", cts.Token);

// Upload an entire directory
await _fileStorage.UploadDirectoryAsync(new DirectoryInfo("/path/to/folder"), "share-directory", cts.Token);

// Download a file
var downloadPath = await _fileStorage.DownloadAsync("filename.txt", "share-directory", "/local/path", cts.Token);

// Check if directory/file exists
bool dirExists = await _fileStorage.DirectoryExistsAsync("share-directory", cts.Token);
bool fileExists = await _fileStorage.FileExistsAsync("share-directory", "filename.txt", cts.Token);

// Delete operations
await _fileStorage.DeleteFileAsync("share-directory", "filename.txt", cts.Token);
await _fileStorage.DeleteDirectoryAsync("share-directory", cts.Token);

Tips and best practices

• Use UploadInChunks for large files to avoid memory issues - now optimized with ArrayPool.
• Keep container names lowercase when interacting with Azure Blob Storage.
• When uploading directories with nested folders, use consistent path separators (/) for blob names.
• Clean up temporary local directories after operations to avoid consuming disk space.
• Prefer managed identities or AAD-based UserToken mode over account keys for production deployments.
• Always pass CancellationToken for long-running operations to support graceful cancellation.
• Use timeout-based CancellationTokenSource for operations that shouldn't run indefinitely.
• Handle OperationCanceledException to gracefully respond to cancellation requests.

Migration Guide from v11.x to v12.0.0

Breaking Changes

All async interface methods now have an optional CancellationToken parameter. Existing code will continue to work without modification since the parameter has a default value.

Recommended Updates

// Before (v11.x) - still works
await storage.UploadFileAsync(fileInfo, MimeTypes.txt, "container");

// After (v12.x) - recommended for production
using var cts = new CancellationTokenSource(TimeSpan.FromMinutes(10));
await storage.UploadFileAsync(fileInfo, MimeTypes.txt, "container", cts.Token);

Testing

The DevelopmentHelpers.Storage.Core.Tests project includes integration tests for all authentication modes (DevelopmentStorage, ConnectionString, ManagedIdentity, UserToken) using per-mode appsettings.*.json files in the test project. These tests exercise common flows such as uploading/downloading files, chunked uploads, container sync and zip creation.

Test Coverage

• Connection string authentication tests
• Managed identity authentication tests
• User token authentication tests
• Development storage (Azurite) tests
• File upload/download operations
• Directory operations
• Chunked upload performance tests
• Container synchronization tests
• Azure File Share operations

Contributing

Contributions and issues are welcome. Please open an issue describing the problem or the feature request and submit pull requests for fixes.

License

Include your license information here (e.g., MIT) or a note about internal use.

If you need a targeted example or the exact code snippet for a specific scenario, describe the scenario and I will add a short sample.