Nexus.Sources.StructuredFile 2.0.0-beta.41

Nexus.Sources.StructuredFile

This data source extension provides a base class for reading regularly organized structured files into Nexus. Derived data sources, such as CSV readers, implement the actual file parsing while this package handles file discovery, time-range slicing, file-source selection, offsets, and read-buffer metadata.

Configuration Shape

A structured-file data source is configured with top-level settings and one or more file source groups.

{
  "fileSourceGroupsMap": {
    "default": {
      "voltage": [
        {
          "begin": "2026-08-10T00:00:00Z",
          "pathSegments": ["'DATA'", "yyyy-MM-dd", "HH"],
          "fileTemplate": "'DAQ_5ms-'yyyy-MM-dd-HH-mm-ss'.csv'",
          "fileDateTimePreselector": null,
          "fileDateTimeSelector": null,
          "filePeriod": "00:10:00",
          "maxFileDuration": "00:10:01",
          "fileNameOffset": "00:00:00",
          "utcOffset": "00:00:00",
          "irregularTimeInterval": true,
          "additionalSettings": {}
        }
      ]
    }
  },
  "additionalSettings": {}
}

The exact additionalSettings shape is defined by the derived data source implementation.

File Discovery And Duration

The most important configuration concept is that file discovery and file coverage are separate concerns:

Short version:

filePeriod = where to look
maxFileDuration = how long one file may cover

The discovery grid should match the cadence implied by the folder and file naming scheme. For example, if folders or file names are organized every 10 minutes, use a 10-minute grid even when the row timestamps inside the files are more precise.

The duration setting is the maximum elapsed time covered by one physical file, measured from the timestamp parsed from its name. If it is omitted, the grid period is used as the duration as well.

Keeping these concepts separate is important for files whose names are less precise than their contents. A file may be discovered on a clean 10-minute grid while its rows use millisecond timestamps. The search must still follow the 10-minute layout, but the lookback window must cover the real maximum file duration.

Use Cases

Practical Rules

Glob And Wildcard Searches

maxFileDuration does not define the glob pattern. The path template and filePeriod define which candidate paths are generated. maxFileDuration only expands the time range backward before those candidate paths are generated.

Example file source:

{
  "pathSegments": ["'DATA'", "yyyy-MM-dd", "HH"],
  "fileTemplate": "'DAQ_5ms-'yyyy-MM-dd-HH-mm-ss'.csv'",
  "fileDateTimePreselector": "DAQ_5ms-(.{19})\\.csv",
  "fileDateTimeSelector": "yyyy-MM-dd-HH-mm-ss",
  "filePeriod": "00:10:00",
  "maxFileDuration": "00:10:01"
}

For a read beginning at 2026-08-10T12:32:00Z, the lookback start is approximately 12:21:59 because maxFileDuration is 00:10:01.

The algorithm then maps that enlarged range onto the filePeriod grid. With a 10-minute grid, candidate buckets include:

2026-08-10T12:20:00Z
2026-08-10T12:30:00Z

Those buckets produce paths from pathSegments, for example:

2026-08-10/12/

The file name template contains wildcard-like sections or requires a preselector, so files in that folder can be matched and then parsed:

2026-08-10/12/DAQ_5ms-2026-08-10-12-23-26.csv
2026-08-10/12/DAQ_5ms-2026-08-10-12-33-26.csv

If the read begins at 2026-08-10T13:02:00Z, the lookback start is approximately 12:51:59. Candidate buckets include:

2026-08-10T12:50:00Z
2026-08-10T13:00:00Z

Those buckets can produce two folder searches:

2026-08-10/12/
2026-08-10/13/

This is the key point: maxFileDuration caused the earlier 12:50 bucket to be considered, but the buckets are still spaced by filePeriod. The algorithm does not create a 00:10:01 search grid.

Using maxFileDuration as the grid would be wrong because a 00:10:01 grid drifts away from a 10-minute file organization:

12:00:00
12:10:01
12:20:02
12:30:03

The actual file organization may be:

12:00:00
12:10:00
12:20:00
12:30:00

StructuredFileDataSource Settings

StructuredFileDataSourceSettings<TAdditionalSettings, TAdditionalFileSourceSettings>

FileSource<TAdditionalSettings>

ReadInfo<TAdditionalSettings>

Derived data sources receive ReadInfo<TAdditionalSettings> when asked to read one candidate file. It describes both the physical file and the part of the Nexus read buffer this file should fill.

Timestamp-Aware Readers

Readers that parse timestamps from the file should write values relative to BufferBegin:

var index = (int)((timestamp - info.BufferBegin).Ticks / samplePeriod.Ticks);

Use this for files where each row has its own timestamp column.

Sequential Readers

Readers that do not parse per-row timestamps should use FileOffset and FileBlock:

ReadSamples(info.FilePath, info.FileOffset, info.FileBlock, destination);

Use this for files where the requested sample positions are derived from the file's nominal timing rather than from timestamps inside the file.

Test Case Description

The test databases cover file and folder structures that contain files with a maximum duration. Files may be shorter than the configured upper limit, but they must not exceed it. The upper limit is part of the source configuration because Nexus cannot know how long a file is before candidate files have been found and read.