redb.Route.GenericFile 4.1.0

redb.Route.GenericFile

Shared base library for file-based transports in the redb.Route ESB framework.
Provides abstract consumer, producer, options, and file-operations interfaces that concrete transports (File, FTP, SFTP) inherit from. Not used directly — include one of the transport-specific packages instead.

Architecture

GenericFileEndpointOptions          ← base options (polling, filtering, producer write)
 └─ RemoteFileEndpointOptions       ← + host/port/auth/reconnect

GenericFileConsumer<TOptions>       ← template-method poll loop
 └─ RemoteFileConsumer<TOptions>    ← + connect/reconnect lifecycle

GenericFileProducer<TOptions>       ← template-method write flow
 └─ RemoteFileProducer<TOptions>    ← + connect/reconnect lifecycle

IFileOperations                     ← protocol abstraction (list, read, write, move, delete)
 └─ IRemoteFileOperations           ← + connect/disconnect

Transport implementations (File, Ftp, Sftp) provide concrete IFileOperations and transport-specific headers/DSL. All polling logic, filtering, idempotency, post-processing, atomic write, and file-exist strategies are implemented in this base library.

Consumer Pipeline

The GenericFileConsumer runs a poll loop with the following steps:

1. List files in the base directory (optionally recursive)
2. Filter — exclude temp/internal files (.redb_*, temp prefix), apply glob include/exclude patterns
3. Sort — by name, date, or size (ascending or descending)
4. Limit — apply maxMessagesPerPoll
5. Eligibility checks — minAge, maxAge, done-file presence
6. Read lock — transport-specific claim on the file (local file only)
7. Idempotency — skip files already in the repository
8. Pre-move — optionally move to a staging directory before processing
9. Read body — as byte[] (default) or Stream (streamBody=true)
10. Invoke processor — pass exchange to the downstream pipeline
11. Post-process — Noop (leave in place) / Delete / MoveTo
12. Confirm — mark idempotent key, delete done file

The read lock is taken before the idempotency claim: a consumer that loses the lock race must not consume the idempotent key, or the file would never be picked up again by anyone.

On processing failure: remove idempotent key, leave the file in place, move to moveFailed directory (if configured). A file that cannot be read is treated the same way — it is never delivered as an empty message.

Producer Pipeline

The GenericFileProducer writes files with atomic temp-then-rename:

1. Resolve body — from exchange.Out (if set) or exchange.In
2. Resolve target path — from FileName option / header / auto-generated GUID
3. Validate path — jail-directory check (transport-specific)
4. Create directories — if autoCreate=true
5. Write to temp file — using tempPrefix or tempFileName
6. Handle existing — apply fileExist strategy (Override / Append / Fail / Ignore / Move / TryRename)
7. Atomic rename — move temp file to target
8. Cleanup — delete temp file on failure

Options Reference

GenericFileEndpointOptions (Base)

Consumer / Polling

Poll backoff follows Apache Camel ScheduledPollConsumer: after BackoffIdleThreshold idle polls or BackoffErrorThreshold error polls, the next BackoffMultiplier polls are skipped (no listing, no connect), then the counters reset. A skipped poll still waits Delay, so stopping the consumer during a skip resolves within Delay. Entering backoff logs one warning; resuming logs one info line. BackoffMultiplier requires at least one threshold, and a threshold requires a multiplier — otherwise the endpoint refuses the configuration.

A file that fails in the route. By default (Camel parity) a poll that created exchanges counts as success even if the route failed them, so a deterministically-bad file, or every file while the downstream is down, is retried every Delay. Two remedies:

• Poison file → OnException<T>().MaximumRedeliveries(n).RedeliveryDelay(...) and/or MoveFailed to quarantine it out of the poll directory.
• Downstream down (every file fails) → set BackoffOnFailedExchanges = true (with an error threshold + multiplier): a poll whose every created exchange failed unhandled then counts as an error and the consumer backs off instead of hammering the downstream. A poll with any success still counts as success.

Consumer / Post-Processing

Only one of Noop, Delete, MoveTo can be set.

Consumer / Idempotency

Default idempotent key: "{fullPath}|{lastModifiedUtc:O}|{length}".

Consumer / Body

Producer

RemoteFileEndpointOptions (extends Base)

Connection

Reconnection

Consumer-Specific

Enums

GenericFileExistStrategy

GenericFileSortBy

File Operations Interface

Transport implementations provide IFileOperations (or IRemoteFileOperations for network transports):

• ListFilesAsync — enumerate files in a directory
• ReadAllBytesAsync / OpenReadAsync — read file contents
• WriteAsync (byte[] / Stream) / AppendAsync / AppendTextAsync — write
• ExistsAsync / DeleteAsync / MoveAsync — file operations
• CreateDirectoryAsync / DirectoryExistsAsync — directory operations
• Path helpers: CombinePath, GetParentPath, GetFileName, GetExtension, etc.

Glob Patterns

Include/exclude patterns support comma-separated values and */? wildcards:

*.csv          — all CSV files
*.csv,*.xml    — CSV and XML files
report_*       — files starting with "report_"
data?.txt      — data1.txt, dataA.txt, etc.

Path Filters

A glob sees only the file name, so it cannot say which directory a recursive poll should take. Four options decide by path, in the order the poll applies them (Apache Camel parity):

antInclude=TYPE_A/outbox/*.csv,TYPE_B/outbox/*.csv   — Ant patterns over the path relative to the
antExclude=archive/**                                  polled directory; '*' stops at a separator,
                                                       '**' spans levels, ',' separates patterns
filterDirectory=header.redbSftp.Name != 'archive'    — asked per subdirectory BEFORE it is listed
filterFile=header.redbSftp.Length > 1024             — asked per file before it is read or moved
filter=#partnerFilter                                — an IGenericFileFilter from the registry,
                                                       for both files and directories

filterDirectory and the bean's AcceptDirectory decide while the listing is still walking, so a directory they turn down costs nothing — no listing, no round trip. That is what makes one consumer over a partner tree of two hundred directories affordable, and it is the only way to keep one connection when the server allows only one.

Everything these filters reject is left exactly as it was found: no exchange is created for it, so delete, move and preMove never touch it. Polling a parent recursively and sorting it out in the route does the opposite — the file is downloaded and post-processed first.

Done File Substitutions

Requirements

• .NET 8.0 / 9.0 / 10.0
• redb.Route (core) — no external dependencies