Wolfgang.Etl.FixedWidth 0.13.0

Wolfgang.Etl.FixedWidth

Extractor and Loader for reading and writing fixed width files and text streams

📦 Installation

dotnet add package Wolfgang.Etl.FixedWidth

NuGet Package: Wolfgang.Etl.FixedWidth on NuGet.org

📄 License

This project is licensed under the MIT License. See the LICENSE file for details.

📚 Documentation

• GitHub Repository: https://github.com/Chris-Wolfgang/ETL-FixedWidth
• API Documentation: https://Chris-Wolfgang.github.io/ETL-FixedWidth/
• Formatting Guide: README-FORMATTING.md
• Contributing Guide: CONTRIBUTING.md

🚀 Quick Start

Extraction — Reading a Fixed-Width File

Define a POCO with [FixedWidthField] attributes, then read records with await foreach:

using Wolfgang.Etl.FixedWidth;
using Wolfgang.Etl.FixedWidth.Attributes;
using Wolfgang.Etl.FixedWidth.Enums;

// 1. Define the record class.
public class PersonRecord
{
    [FixedWidthField(0, 10)]
    public string FirstName { get; set; } = string.Empty;



    [FixedWidthField(1, 10)]
    public string LastName { get; set; } = string.Empty;



    [FixedWidthField(2, 3, Alignment = FieldAlignment.Right, Pad = '0')]
    public int Age { get; set; }
}

// 2. Create the extractor (accepts any TextReader or Stream).
var reader = new StringReader
(
    "Alice     Anderson  025\n" +
    "Bob       Baker     042\n" +
    "Charlie   Clark     033"
);

var extractor = new FixedWidthExtractor<PersonRecord>(reader);

// 3. Iterate records asynchronously.
await foreach (var person in extractor.ExtractAsync(CancellationToken.None))
{
    Console.WriteLine($"{person.FirstName} {person.LastName}, Age {person.Age}");
}

// Output:
//   Alice Anderson, Age 25
//   Bob Baker, Age 42
//   Charlie Clark, Age 33

Loading — Writing a Fixed-Width File

var writer = new StringWriter();
var loader = new FixedWidthLoader<PersonRecord>(writer);

// LoadAsync accepts an IAsyncEnumerable<PersonRecord>; `sourceItems` is any
// async sequence of records (for example, the output of a FixedWidthExtractor).
await loader.LoadAsync(sourceItems, CancellationToken.None);

Console.WriteLine(writer.ToString());
// Output:
//   Alice     Anderson  025
//   Bob       Baker     042
//   Charlie   Clark     033

For file-based I/O, use the Stream constructor which creates a 64 KB buffered reader/writer for improved throughput:

// Extraction from a file
await using var readStream = File.OpenRead("people.dat");
using var extractor = new FixedWidthExtractor<PersonRecord>(readStream);

// Loading to a file
await using var writeStream = File.OpenWrite("output.dat");
using var loader = new FixedWidthLoader<PersonRecord>(writeStream);

Because the Stream constructors accept any Stream, compression works out of the box — wrap the file stream in a GZipStream or BrotliStream to read or write compressed fixed-width data (common for mainframe .gz exports) without a decompressed copy on disk:

// Extraction from a GZip-compressed file
await using var readStream = File.OpenRead("people.dat.gz");
await using var readGzip = new GZipStream(readStream, CompressionMode.Decompress);
using var extractor = new FixedWidthExtractor<PersonRecord>(readGzip);

// Loading to a GZip-compressed file
await using var writeStream = File.Create("output.dat.gz");
await using var writeGzip = new GZipStream(writeStream, CompressionLevel.Optimal);
using var loader = new FixedWidthLoader<PersonRecord>(writeGzip);

See the CompressedStreams example for a complete GZip and Brotli round trip.

Controlling line endings

FixedWidthExtractor reads any line ending automatically — \n, \r, or \r\n — so no configuration is needed for input.

For output, the loader writes each record with its TextWriter's newline. To force a specific ending regardless of the platform you run on — for example, a downstream mainframe or FTP consumer that requires Unix \n — pass a TextWriter with the NewLine you want:

// Force Unix (LF) line endings, even on Windows
await using var stream = File.Create("output.dat");
await using var writer = new StreamWriter(stream) { NewLine = "\n" };
using var loader = new FixedWidthLoader<PersonRecord>(writer);

await loader.LoadAsync(records, CancellationToken.None);

NewLine accepts any string ("\n", "\r\n", or a custom terminator). The default is Environment.NewLine.

Checkpoint and resume (large files)

Processing a multi-GB fixed-width export can take a while; if the pipeline crashes at record 5,000,000 you don't want to re-read from the top. Opt in to byte-offset tracking, persist CurrentByteOffset after each record, and resume from the saved position with StartByteOffset:

// First run — checkpoint after each record
await using var stream = File.OpenRead("huge.dat");
using var extractor = new FixedWidthExtractor<Record>(stream) { TrackByteOffset = true };
await foreach (var record in extractor.ExtractAsync(token))
{
    Process(record);
    SaveCheckpoint(extractor.CurrentByteOffset);   // byte position of the next unread line
}

// After a crash — seek straight to the next unread line, skipping everything already done
await using var stream = File.OpenRead("huge.dat");
using var extractor = new FixedWidthExtractor<Record>(stream) { StartByteOffset = LoadCheckpoint() };
await foreach (var record in extractor.ExtractAsync(token)) { /* only the remainder */ }

Terminators (\n, \r, \r\n), multi-byte UTF-8, and a leading byte-order mark are all counted exactly, so a saved offset is a precise byte position. Tracking is opt-in — it wraps the reader in a byte-counting decoder, so the default read path keeps its throughput — and requires the Stream constructor (seekable for resume). On resume, header lines are not re-skipped and SkipItemCount applies from the resumed position. CurrentLineNumber is available for diagnostics independently of checkpointing.

Inspecting the layout

FixedWidthSchema.For<T>() exposes the resolved field layout as a read-only view — useful for generating documentation, building validation tooling, or debugging a mapping. It applies the same validation as extraction, so an invalid layout (duplicate column index, a mapped field with no public setter) throws here too.

var schema = FixedWidthSchema.For<PersonRecord>();

foreach (var field in schema.Fields)   // includes skip columns (field.IsSkip)
{
    Console.WriteLine($"{field.StartPosition}-{field.EndPosition}  {field.Name}  ({field.Length})");
}

schema.ExpectedLineWidth;   // total line width, including skipped columns
schema.TotalColumnCount;    // columns including skips
schema.FieldCount;          // mapped fields only
schema.SkipCount;           // skipped columns

Each FixedWidthFieldInfo carries Name, StartPosition/EndPosition, Length, ColumnIndex, PropertyType, Alignment, Pad, Format, Header, and NumberStyles. Skipped columns have IsSkip == true and expose a SkipMessage instead of a name.

ToDiagram() renders the layout as a text table — drop it into a log line at startup or paste it into a ticket:

Console.WriteLine(FixedWidthSchema.For<EmployeeRecord>().ToDiagram());

Position  Field           Type    Length  Align  Pad  Format
--------  --------------  ------  ------  -----  ---  ------
0-9       FirstName       String  10      Left   ' '
10-17     [skip]                  8
18-23     EmployeeNumber  String  6       Left   ' '

Total width: 24  |  Columns: 3 (2 fields + 1 skip)  |  Delimiter: none

Defining the layout in code

When you can't decorate the record type — a third-party POCO, or a layout chosen at runtime — build the schema with FixedWidthSchemaBuilder<T> instead of attributes, then hand it to the extractor or loader via its Schema property:

var schema = new FixedWidthSchemaBuilder<CustomerRecord>()
    .Field(r => r.CustomerId, index: 0, length: 8)
    .Field(r => r.Name, index: 1, length: 30)
    .Skip(index: 2, length: 5)
    .Field(r => r.Balance, index: 3, length: 9, alignment: FieldAlignment.Right, format: "0000000.00")
    .Build();

using var extractor = new FixedWidthExtractor<CustomerRecord>(reader) { Schema = schema };

The builder uses lambda expressions for type-safe, refactor-proof property references — no magic strings. index is the zero-based column ordinal (the same value as [FixedWidthField(index, length)]); start positions are computed from the column lengths. A schema built this way is equivalent to one resolved from attributes: it validates the same way (duplicate index, no public setter) and is fully introspectable via Fields / ToDiagram(). Setting Schema overrides any attributes on the type.

See the SchemaBuilder example for a runnable walk-through.

Reading and writing binary / mainframe records

Mainframe (COBOL) files mix text with binary-encoded numeric fields — COMP big-endian integers and COMP-3 packed decimals — and are not newline-delimited: every record is a fixed number of bytes, written back-to-back. Because packed-decimal and binary bytes routinely contain 0x0A/0x0D, they must never be split on newlines. FixedWidthBinaryExtractor<T> / FixedWidthBinaryLoader<T> read and write these records by byte count, decoding/encoding each field per its declared type.

Declare the layout with [FixedWidthBinaryField] — widths are in bytes, and BinaryFieldType selects the decoding (Text, Binary, or PackedDecimal):

using Wolfgang.Etl.FixedWidth;
using Wolfgang.Etl.FixedWidth.Attributes;
using Wolfgang.Etl.FixedWidth.Enums;

public class AccountRecord
{
    [FixedWidthBinaryField(0, 8, BinaryFieldType.Text)]
    public string AccountId { get; set; } = string.Empty;

    [FixedWidthBinaryField(1, 4, BinaryFieldType.Binary)]              // COMP
    public int TransactionCount { get; set; }

    [FixedWidthBinaryField(2, 5, BinaryFieldType.PackedDecimal, Scale = 2)]   // PIC S9(7)V99 COMP-3
    public decimal Balance { get; set; }
}

await using var stream = File.OpenRead("accounts.dat");
using var extractor = new FixedWidthBinaryExtractor<AccountRecord>(stream);
await foreach (var account in extractor.ExtractAsync(CancellationToken.None))
{
    // account.Balance decoded from COMP-3, account.TransactionCount from COMP
}

Text fields decode with the extractor/loader's encoding — ASCII by default. For EBCDIC data, register the code-page provider and pass the encoding (the System.Text.Encoding.CodePages package is only needed by consumers who use it, so it is not bundled):

Encoding.RegisterProvider(CodePagesEncodingProvider.Instance);
using var extractor = new FixedWidthBinaryExtractor<AccountRecord>(stream, Encoding.GetEncoding("IBM037"));

Writing is symmetric — FixedWidthBinaryLoader<T> encodes each field back to its COMP/COMP-3/text bytes. See the BinaryRecords example for a runnable write → read round trip.

Transforming between layouts

To reformat a fixed-width file from one layout to another — reordering, adding/removing, or format-converting fields (a common mainframe-migration task) — FixedWidthTransformer<TSource, TDestination> is the projection stage between an extractor and a loader:

using var extractor   = new FixedWidthExtractor<LegacyRecord>(sourceReader);
using var transformer = new FixedWidthTransformer<LegacyRecord, ModernRecord>(
    legacy => new ModernRecord
    {
        Id   = legacy.OldId,
        Name = legacy.FullName.Trim(),
    });
using var loader      = new FixedWidthLoader<ModernRecord>(destinationWriter);

// Extract → transform → load in a single streaming pass.
var modern = transformer.TransformAsync(extractor.ExtractAsync(token), token);
await loader.LoadAsync(modern, token);

The projection delegate handles every reformatting case. When source and destination differ only in layout — the same property names and compatible types — use the auto-mapping factory instead of writing the copy by hand:

using var transformer = FixedWidthTransformer<LegacyRecord, ModernRecord>.ByMatchingProperties();

ByMatchingProperties() copies every source property to the destination property of the same name and an assignable type, and requires a public parameterless constructor on the destination.

Reading files with multiple record types

Mainframe and EDI batch files often interleave several record layouts on different lines — a header, many detail rows, and a trailer — distinguished by a discriminator character. FixedWidthMultiRecordExtractor routes each line to the right POCO: register one rule per type, and the first matching predicate wins.

using var extractor = new FixedWidthMultiRecordExtractor(reader)
    .When(line => line[0] == 'H', typeof(HeaderRecord))
    .When(line => line[0] == 'D', typeof(DetailRecord))
    .When(line => line[0] == 'T', typeof(TrailerRecord));

await foreach (var record in extractor.ExtractAsync(token))
{
    switch (record)
    {
        case HeaderRecord h: /* ... */ break;
        case DetailRecord d: /* ... */ break;
        case TrailerRecord t: /* ... */ break;
    }
}

From a file or other Stream, the encoding travels on the options record (the TextReader form above already knows its encoding, so it takes no record):

using var extractor = new FixedWidthMultiRecordExtractor
(
    File.OpenRead("batch.txt"),
    new FixedWidthMultiRecordExtractorOptions { Encoding = Encoding.Latin1 }
)
    .When(line => line[0] == 'H', typeof(HeaderRecord))
    .When(line => line[0] == 'D', typeof(DetailRecord))
    .When(line => line[0] == 'T', typeof(TrailerRecord));

Each record type keeps its own independent [FixedWidthField] layout. A line that matches no rule throws by default; set UnmatchedLineHandling = UnmatchedLineHandling.Skip to drop it, or register a catch-all type with .Otherwise(typeof(UnknownRecord)). Blank lines are skipped before predicates run (so a discriminator can index the line safely), and the extractor shares the family's HeaderLineCount, FieldDelimiter, ValueParser, SkipItemCount/MaximumItemCount, dead-letter OnError, and progress reporting.

Trailer record-count validation falls out of this naturally: capture the trailer as it streams past, count the details, and compare — no extra API needed.

if (trailer.RecordCount != detailCount)
    throw new InvalidDataException($"Trailer says {trailer.RecordCount}, file has {detailCount}.");

See the MultiRecordTrailer example for a runnable header/detail/trailer walk-through that checks both the record count and a control total.

Composing an ETL pipeline

Rather than wiring an extractor, transformer, and loader together by hand, the whole extract → transform → load flow can be expressed as one fluent chain on the generic EtlPipeline (introduced in Wolfgang.Etl.Abstractions 0.16.0; this package builds against the version in its NuGet dependency). FixedWidthExtractor<T> source factories hang off EtlPipeline.Create() and FixedWidthLoader<T> sink terminators hang off the pipeline, with the extractor/loader configuration exposed as inline setters:

using Wolfgang.Etl.Abstractions;
using Wolfgang.Etl.FixedWidth;

// Fixed-width in, human-readable table out — path factories own the files they open.
await EtlPipeline
    .Create()
    .FixedWidthExtractor<PersonRecord>("people.dat")
    .FixedWidthLoader<PersonRecord>("people.txt")
    .WriteHeader(true)
    .FieldSeparator('-')
    .FieldDelimiter(" | ")
    .RunAsync();

Insert transform stages with Through — an inline Func<IAsyncEnumerable<T>, IAsyncEnumerable<TOut>> stage needs no reference to the operators package:

await EtlPipeline
    .Create()
    .FixedWidthExtractor<PersonRecord>(sourceReader)
    .Through(KeepAdults)                 // a stream-to-stream transform delegate
    .FixedWidthLoader<PersonRecord>(destinationWriter)
    .RunAsync();

Every source and sink has path, Stream, and TextReader/TextWriter overloads (plus an existing-FixedWidthExtractor<T> overload). Path factories own the file stream they open and dispose it when the run finishes, on success or failure; caller-supplied streams, readers, and writers are always left open. The builder methods (HeaderLineCount, MalformedLineHandling, FieldDelimiter, Encoding, WriteHeader, ValueConverter, IsDryRun, …) map 1:1 to the FixedWidthExtractor<T> / FixedWidthLoader<T> properties.

See the PipelineExtensions example for a complete, runnable walk-through.

Metrics and observability

The extractor and loader can emit standard System.Diagnostics.Metrics instruments from the meter Wolfgang.Etl.FixedWidth, so throughput and error rates flow to OpenTelemetry, Prometheus, Grafana, Application Insights, or any MeterListener. Metrics are zero-config: they activate automatically when a listener subscribes to the Wolfgang.Etl.FixedWidth meter — there's no flag to set. When nothing is listening, the extract/load loop (sampling once per operation) runs no metric code at all, so telemetry adds zero overhead for callers that don't use it.

Every measurement is tagged etl.operation (extract or load) and etl.record_type (typeof(TRecord).Name).

// Subscribe once at startup — OpenTelemetry, zero per-call code; metrics turn on automatically:
builder.Services.AddOpenTelemetry()
    .WithMetrics(m => m.AddMeter("Wolfgang.Etl.FixedWidth"));

See the Metrics example for a runnable MeterListener walk-through.

✨ Features

Examples:

The examples/ folder contains 16 runnable console projects demonstrating each feature:

Compile-time diagnostics:

The package ships a Roslyn analyzer that catches [FixedWidthField] layout mistakes in the IDE and the build — no configuration required:

🎯 Supported Frameworks

This library targets:

• .NET Framework: 4.6.2, 4.8.1
• .NET Standard: 2.0
• .NET: 5.0, 6.0, 7.0, 8.0, 10.0

The CI test matrix additionally exercises the library on .NET Framework 4.7.x/4.8, .NET Core 3.1 and .NET 9.0 via the nearest package asset; those are tested-against runtimes, not package target frameworks.

See the NuGet package page for the authoritative per-TFM compatibility matrix.

🔍 Code Quality & Static Analysis

This project enforces strict code quality standards through 8 specialized analyzers and custom async-first rules:

Analyzers in Use

1. Microsoft.CodeAnalysis.NetAnalyzers - Built-in .NET analyzers for correctness and performance
2. Roslynator.Analyzers - Advanced refactoring and code quality rules
3. AsyncFixer - Async/await best practices and anti-pattern detection
4. Microsoft.VisualStudio.Threading.Analyzers - Thread safety and async patterns
5. Microsoft.CodeAnalysis.BannedApiAnalyzers - Prevents usage of banned synchronous APIs
6. Meziantou.Analyzer - Comprehensive code quality rules
7. SonarAnalyzer.CSharp - Industry-standard code analysis
8. Microsoft.CodeAnalysis.PublicApiAnalyzers - Tracks the shipped public surface (RS0016/RS0017)

Async-First Enforcement

This library uses BannedSymbols.txt to prohibit synchronous APIs and enforce async-first patterns:

Blocked APIs Include:

• ❌ Task.Wait(), Task.Result - Use await instead
• ❌ Thread.Sleep() - Use await Task.Delay() instead
• ❌ Synchronous file I/O (File.ReadAllText) - Use async versions
• ❌ Synchronous stream operations - Use ReadAsync(), WriteAsync()
• ❌ Parallel.For/ForEach - Use Task.WhenAll() or Parallel.ForEachAsync()
• ❌ Obsolete APIs (WebClient, BinaryFormatter)

Why? To ensure all code is truly async and non-blocking for optimal performance in async contexts.

🛠️ Building from Source

Prerequisites

• .NET 10.0 SDK or later (required for the net10.0 target framework)
• Optional: PowerShell Core for formatting scripts

Build Steps

# Clone the repository
git clone https://github.com/Chris-Wolfgang/ETL-FixedWidth.git
cd ETL-FixedWidth

# Restore dependencies
dotnet restore

# Build the solution
dotnet build --configuration Release

# Run tests
dotnet test --configuration Release

# Run code formatting (PowerShell Core)
pwsh ./scripts/format.ps1

Code Formatting

This project uses .editorconfig and dotnet format:

# Format code
dotnet format

# Verify formatting (as CI does)
dotnet format --verify-no-changes

See README-FORMATTING.md for detailed formatting guidelines.

Building Documentation

This project uses DocFX to generate API documentation:

# Install DocFX (one-time setup)
dotnet tool install -g docfx

# Generate API metadata and build documentation
cd docfx_project
docfx metadata  # Extract API metadata from source code
docfx build     # Build HTML documentation

# Documentation is generated in the docs/ folder at the repository root

The documentation is automatically built and deployed to GitHub Pages when changes are pushed to the main branch.

Local Preview:

# Serve documentation locally (with live reload)
cd docfx_project
docfx build --serve

# Open http://localhost:8080 in your browser

Documentation Structure:

• docfx_project/ - DocFX configuration and source files
• docs/ - Generated HTML documentation (published to GitHub Pages)
• docfx_project/index.md - Main landing page content
• docfx_project/docs/ - Additional documentation articles
• docfx_project/api/ - Auto-generated API reference YAML files

🔐 Verify the build

The library is built deterministically, so you can rebuild the exact assemblies from the tagged source and confirm a NuGet release was built from that source and nothing else. Every GitHub release attaches a reproducible-build-manifest.json with the expected per-framework assembly hashes and the toolchain that produced them.

# Download the manifest for a release, rebuild at the tag, and compare hashes.
gh release download v0.8.0 --repo Chris-Wolfgang/ETL-FixedWidth --pattern reproducible-build-manifest.json
git clone --depth 1 --branch v0.8.0 https://github.com/Chris-Wolfgang/ETL-FixedWidth
dotnet build ETL-FixedWidth/src/Wolfgang.Etl.FixedWidth/Wolfgang.Etl.FixedWidth.csproj -c Release -p:ContinuousIntegrationBuild=true
find ETL-FixedWidth/src/Wolfgang.Etl.FixedWidth/bin/Release -name 'Wolfgang.Etl.FixedWidth.dll' -exec sha256sum {} \;

See docs/REPRODUCIBLE-BUILD.md for the full procedure — which SDK version to use, how to file a discrepancy, and how to publish a third-party verification attestation.

🤝 Contributing

Contributions are welcome! Please see CONTRIBUTING.md for:

• Code quality standards
• Build and test instructions
• Pull request guidelines
• Analyzer configuration details

🙏 Acknowledgments

• Wolfgang.Etl.Abstractions — provides the ExtractorBase, LoaderBase, and TransformerBase base classes, progress reporting infrastructure, and the IProgressTimer contract that this library builds on.
• Microsoft.Extensions.Logging.Abstractions — provides the ILogger interface used for optional structured diagnostic logging throughout the extractor and loader.