SquidEyes.MarketData 2.0.0

SquidEyes.MarketData

A word of caution before you take a dependency: although this library is public, it is an idiosyncratic tool built to support SquidEyes' own trading work. It changes at the speed of that work, makes no backward-compatibility promises of any kind (the format changes in place as the work demands), and is not intended to be anything more than an educational project for others. You are welcome to read, learn, and borrow — just don't build on it expecting a general-purpose, community-driven package.

Market-data primitives for futures backtesting, built around one idea: live trading and backtesting share one code path (IMarketObserver.OnTick(in MarketEvent, IBook)), so a backtest fills the way live would have. The library is data-source neutral: any historical replay or live feed can be normalized into its canonical event stream (files carry a caller-defined source Tag in their metadata — never an enumerated vendor, and never in their names, so multi-source data merges into one dataset), and SL2 is its compact, roll-forward binary format for L1+L2 event streams, with SBD as its structural twin for windowed M1 bars. This package ships the primitives shared by backtesting and live-trading engines — the engines themselves (fill models, latency replay, calibration) live downstream. See Docs/SL2_DESIGN.md and Docs/SBD_DESIGN.md for the models.

• MarketEvent / EventKind / Side / DepthOp — the canonical event atom: Trade prints (drive all decision logic), Quote L1 best bid/ask changes (consulted at fill instants; zero-size quotes pass through verbatim — their timestamps may close bars, so interpretation is the consumer's), Depth positional L2 operations (Add/Update/Remove at positions 0..63, feeding the fill simulator), plus the Snapshot control event. UTC timestamps at 100 ns, prices in integer ticks. No order counts, no order ids, no aggressor — deliberate, documented model omissions
• IBook / Book / Level — the maintained book: a running L1 best bid/ask plus positional L2 ladders (~20 occupied levels is typical), kept separate the way feeds record them, with the fill primitives engines compose from: SweepBuyTicks/SweepSellTicks (ladder walk, honest partials) and RestingAt (displayed size at a price = resting-limit queue-ahead)
• IMarketSource / IMarketObserver — the driver/consumer contract; the .sl2 replay (Sl2Source) and any live adapter drive the identical observer
• Sl2Writer / Sl2Reader / Sl2Source / Sl2Stats / Sl2CsvEncoder — SL2 (SquidEyes L2, .sl2) binary format: single-pass streaming writes, roll-forward-only reads over CRC'd compressed blocks, a session-overview stats footer (Sl2Stats — counts, volume, OHLC, first/last stamps, readable without replay), and a one-event lookahead (NextTimeUtc) built for the engine's post-latency "book state at T + L" pattern
• SbdWriter / SbdReader / SbdBar / SbdStats / SbdCsvEncoder — SBD (SquidEyes Bar Data, .sbd) binary format: SL2's structural twin for one windowed session's M1 bars — same header/blocks/stats-footer shape, integer-tick prices, delta + varint encoded, at most 1440 bars by construction. SbdBar.FromCandle/ToCandle bridge to the Candle world, and SbdWriter.Write(Candle) is a direct sink for an IntervalCandleSet's closed 60-second bars
• Symbol / Instrument / Contract — typed futures identities with tick sizes and point values
• SessionKind — named ET trading windows (DTH, MTH, RTH, PTH) with simple expansion to (TimeOnly From, TimeOnly Until), plus ETH — the full electronic session (no intraday ET window): the session every .csv name carries, never the binaries'
• Session — SessionKind + date, validated against a US-futures holiday calendar. Supports Embargos (no-trade sub-windows) of three kinds: AdHoc (explicit times), News (anchored to a wall-clock DateTime, widened by NewsImpactDefaults), and Anchored (relative to session Start/End for a TimeSpan). session.IsTradable(when) answers "in window AND not embargoed?"
• Candle — single OHLCV bar over an arbitrary [FromET, UntilET) window. Interval-agnostic — build trade-by-trade via AddTrade/TryAdd, or load from pre-computed OHLCV. Trade-only by construction (feed it Trade prints).
• ICandleSet / CandleSet — rolling, capacity-bounded collection of candles built trade-by-trade via OnTrade. ICandleSet is the uniform contract every set implements — consumer code (strategies, indicators) depends on it and stays blind to which kind it holds. Six flavours: IntervalCandleSet (time-bucketed, wall-clock aligned), RenkoCandleSet (price-grid-driven; brick size = brickTicks * Instrument.TickSize, with or without wicks), RangeCandleSet (bar span reaches rangeTicks; grid-free, no price fabrication), and the activity bars TickCandleSet / VolumeCandleSet / ValueCandleSet (close every N prints / contracts / price*size). All raise a CandleClosed event and expose newest-first indexing (set[0] = most recent). Except for Interval (in-progress bar), Current is the most recently closed bar — an activity/range bar's end-time isn't known until the print that closes it arrives. The four activity/range sets take an optional one-time TickSkip phase offset (default: none); for a different phase, build a new set.
• EasternTime — shared DST-correct ET ↔ UTC helpers (FromUtc, ToUtc, TodayEt, WindowToUtc). The stream is UTC-canonical; session/calendar logic projects to ET at the point of use.
• Sl2File / SbdFile / CsvFile / MarketDataFile — the canonical naming convention (see File naming below). One uniform name shape, {Symbol}{Contract}_{yyyyMMdd}_{Content}_{Session}_{TZ}: Sl2File names the .sl2 binaries, SbdFile the .sbd compressed bar binaries, CsvFile the full-session .csv files; each has a BuildBlobName sibling for object storage, and MarketDataFile.TryParseFileName / TryParseBlobName recover the typed fields plus a MarketFileKind from any of them. Names carry identity only — the source Tag is file metadata, never part of a name.
• DateOnlyExtenders trade-calendar query helpers — IsTradeDate, EarliestTradeDate, LatestTradeDateBefore, EnumerateTradeDates.

Install

dotnet add package SquidEyes.MarketData

Target: net10.0.

Quickstart

The Demo project exercises every public type end-to-end with runtime assertions — events → book → .sl2 → replay → candles → .sbd → naming — and is the best place to see the library actually working (dotnet run --project Demo/SquidEyes.MarketData.Demo; CI runs it on every push). The condensed version:

using SquidEyes.MarketData;

var mes = Instrument.Create(Symbol.MES);
var info = new Sl2Info(
    mes, Contract.Create(Symbol.MES, "H26"), new DateOnly(2026, 1, 5),
    Tag.Create("PIK"), SessionKind.RTH);   // the source tag is yours to define

// Write an .sl2 file — the writer is itself an IMarketObserver, so any producer can drive it
var t = new DateTime(2026, 1, 5, 14, 30, 0, DateTimeKind.Utc);   // 09:30 ET

using (var fs = File.Create(Sl2File.BuildFileName(info)))      // MESH26_20260105_L2_RTH_ET.sl2
using (var writer = new Sl2Writer(fs, info))
{
    writer.Write(MarketEvent.Quote(t, Side.Bid, priceTicks: 27600, size: 10));    // L1
    writer.Write(MarketEvent.Quote(t, Side.Ask, priceTicks: 27601, size: 15));
    writer.Write(MarketEvent.Depth(t, Side.Bid, DepthOp.Add, position: 0, 27600, 10));  // L2
    writer.Write(MarketEvent.Depth(t, Side.Ask, DepthOp.Add, position: 0, 27601, 15));
    writer.Write(MarketEvent.Trade(t.AddMilliseconds(3), 27601, 3));
}

// Replay it through the same observer contract that runs live
sealed class MyStrategy : IMarketObserver
{
    public void OnTick(in MarketEvent e, IBook book)
    {
        if (e.Kind == EventKind.Trade && book.AskDepth > 0)
        {
            var cost = book.SweepBuyTicks(5, out var filled);       // ladder walk, honest partials
            var vwap = filled > 0 ? cost / (double)filled * book.Instrument.TickSize : 0;
            var queueAhead = book.RestingAt(Side.Bid, book.BestBidTicks);   // resting-limit model
        }
    }
}

using (var source = new Sl2Source(File.OpenRead("MESH26_20260105_L2_RTH_ET.sl2")))
{
    await source.RunAsync(new MyStrategy(), CancellationToken.None);
}

// Sessions + embargoes — is this UTC instant tradable? (project to ET at the point of use)
var session = Session.Create(new DateOnly(2026, 1, 5), SessionKind.RTH);   // 09:30–16:00 ET
session.AddAnchoredEmbargo(SessionAnchor.Start, TimeSpan.FromMinutes(1), "open-1m");
session.AddNewsEmbargo(new DateTime(2026, 1, 5, 14, 0, 0), NewsImpact.High, "FOMC");

var tradable = session.IsTradable(EasternTime.FromUtc(t));

// Candles are Trade-only consumers, fed ET times at the caller's ingest
var renko = new RenkoCandleSet(mes, brickTicks: 4, capacity: 200, withWicks: true);
renko.OnTrade(EasternTime.FromUtc(t.AddMilliseconds(3)), 6900.25, 3);
var lastBrick = renko.Current;   // most recently closed brick (Renko bricks are born closed)

// Bars → .sbd: wire an M1 IntervalCandleSet straight into an SbdWriter
var sbdInfo = new SbdInfo(mes, Contract.Create(Symbol.MES, "H26"),
    new DateOnly(2026, 1, 5), Tag.Create("PIK"), SessionKind.RTH);

using (var fs = File.Create(SbdFile.BuildFileName(sbdInfo)))   // MESH26_20260105_M1_RTH_ET.sbd
using (var sbd = new SbdWriter(fs, sbdInfo))
{
    var m1 = new IntervalCandleSet(60, capacity: 500);
    m1.CandleClosed += (_, e) => sbd.Write(e.ClosedCandle);
    // ... m1.OnTrade(...) per Trade event; closed minutes stream straight to disk
}

SL2 binary format

SL2 = SquidEyes L2 — a compact, roll-forward-only binary container for one session's L1+L2 event stream, from any source. No random access: no keyframes, no block index — one continuous delta chain, chunked into blocks purely for compression, integrity, and bounded memory.

[header]     "SL2\0" magic, version=1, symbol/contract/date, source tag, session,
             tick size, point value, base UTC ticks, codec, block cadence —
             no offsets, so writes stream in one pass (a pipe works)
[block]*     u32 payloadLen | payload (Brotli or raw) | u32 crc32c(payload)
[stats]      fixed-size session overview: event counts by kind, traded volume,
             OHLC ticks, first/last stamps + crc32c

• Records — Trade, QuoteBid/QuoteAsk, DepthBid/DepthAsk (a PP byte packs op + position), snapshot (semantic reset carrying the full book, yielded as an EventKind.Snapshot). Time deltas are 100 ns UTC ticks; prices are integer-tick zigzag deltas against five running anchors (L1 bid, L1 ask, trade, per-side L2 cursor) that run continuously across blocks and re-seed only at a snapshot.
• Stats — Sl2Reader.Stats (also on Sl2Source) exposes the footer's Sl2Stats — CRC-verified at open, so a session overview costs a header + footer read, no event replay.
• Integrity — CRC-32C per block and on the stats footer.
• Determinism — byte-identical output for identical input; the CSV projection (Sl2CsvEncoder) is deterministic too.

SBD binary format

SBD = SquidEyes Bar Data — SL2's structural twin for one windowed session's M1 OHLCV bars (see Docs/SBD_DESIGN.md): the same header / CRC'd-compressed- blocks / stats-footer layout, written single-pass by SbdWriter (which takes SbdBars directly or one-minute Candles) and read roll-forward by SbdReader.

[header]     "SBD\0" magic, version=1, symbol/contract/date, source tag, session,
             tick size, point value, codec, block cadence — no offsets, one-pass writes
[block]*     u32 payloadLen | payload (Brotli or raw) | u32 crc32c(payload)
[stats]      fixed-size session overview: bar count, volume, prints, OHLC ticks,
             first/last bar stamps + crc32c

• Records — one per traded minute (quiet minutes are simply absent — no empty bars), keyed by ET minute-of-day, strictly ascending: minute delta + OHLC as integer-tick deltas (open vs previous close, close vs open, high/low as non-negative body offsets — a record cannot express an invalid bar) + volume + print count, varint/zigzag encoded, anchors continuous across blocks.
• Bounded by construction — ascending minutes within one date cap a file at 1440 bars.
• Stats — SbdReader.Stats exposes the footer's SbdStats (bar count, volume, prints, OHLC, first/last stamps), CRC-verified at open; every field is derived from the bars in the file, never externally supplied (SL2_DESIGN §8.1 applies to both formats).
• Integrity / determinism — CRC-32C per block and on the footer; byte-identical output for identical input, SbdCsvEncoder projection included.

File naming

The public API deals only in complete filenames and blob names — the builders below produce them and MarketDataFile.TryParseFileName / TryParseBlobName recover the typed fields back out (stem construction is internal to the library). Names carry identity only: the data source is never part of a name — provenance lives in file metadata (the .sl2/.sbd headers' Source tags; blob-store metadata for CSVs) — so data from different sources merges into one dataset under one name per slot.

One uniform shape

Every name is {Symbol}{Contract}_{yyyyMMdd}_{Content}_{Session}_{TZ} plus an extension — symbol and contract fused (ESM26), content ∈ {L1, L2, M1} (ContentKind), and the timezone token derived from the session (UT for ETH, ET for the sub-sessions). The extension names the format; the stem names the content, and the two must agree (builders throw, parsers reject):

The binaries imply their content (SL2 holds L1+L2 events, labeled L2; SBD holds M1 bars) and pair with the windowed sub-sessions only; CSVs always cover the full ETH session.

Filenames (the API)

Blob names

For object storage (files never live in folders), each builder has a BuildBlobName sibling producing {bucket}/{Format}/{Symbol}/{ContractYear}/{Contract}/{Session}/{FileName} — {Format} is the extension uppercased (SL2, SBD, CSV) so a store lists one format at a time, {ContractYear} is the contract's year (an ESM26 file lives under 2026 regardless of trade date, so one contract is one prefix), and the bucket must match ^[a-z][a-z0-9]{0,31}$ (enforced by the builders and the parser):

Parsing

MarketDataFile.TryParseFileName recognizes .sl2 / .sbd / .csv, returns (Symbol, Date, Contract, ContentKind Content, SessionKind Session, MarketFileKind Kind) — nothing nullable — and rejects any name a builder could not have produced (unknown extension, unparseable stem, or a format↔content/session mismatch). TryParseBlobName additionally returns the Bucket and verifies the seven-segment shape — a valid bucket plus every path segment (format included) checked against the filename's own fields.

var info = new Sl2Info(Instrument.Create(Symbol.ES), Contract.Create(Symbol.ES, "M26"),
    new DateOnly(2026, 5, 14), Tag.Create("PIK"), SessionKind.RTH);   // Source: header metadata only

var blobName = Sl2File.BuildBlobName("md", info);
// "md/SL2/ES/2026/M26/RTH/ESM26_20260514_L2_RTH_ET.sl2"

var f = MarketDataFile.TryParseBlobName(blobName)!.Value;
// f.Bucket == "md", f.Symbol == Symbol.ES, f.Content == ContentKind.L2,
// f.Session == SessionKind.RTH, f.Kind == MarketFileKind.Sl2

Public surface

Everything ships in a single flat namespace, SquidEyes.MarketData (subfolders are physical layout only). By area:

Opinionated choices

• US futures only. Supported symbols (Symbol.cs): six full-size/micro pairs — ES/MES, NQ/MNQ, RTY/M2K, YM/MYM, GC/MGC, CL/MCL. Adding a symbol is one enum value + one row in Instrument.BuildCache(). Adding a non-futures asset class is one InstrumentKind enum value plus a corresponding row.
• Holiday-aware trade dates. Session.Create(...) rejects weekends, US market holidays (NYD, MLK, Presidents, Good Friday, Memorial, Juneteenth, Independence, Labor, Thanksgiving, Christmas), early-close days, Easter Monday, and Boxing Day. Calendar lives in HolidayExtenders.cs.
• Named session windows — DTH 08:00–16:00 ET, MTH 08:00–12:00 ET, RTH 09:30–16:00 ET (CME cash session), PTH 09:30–11:30 ET (the high-volume opening drive). ETH names the full electronic session — no intraday ET window (so ToTimes() / Session.Create reject it); it is the session every .csv name carries. Add more in SessionKind.cs — one enum value plus a switch arm in ToTimes() / ToCode() / ParseCode().
• Embargo defaults. A News-kind embargo widens by NewsImpactDefaults minutes before/after its anchor (High: 5/15, Medium: 3/10 by default; settable globally at app startup).
• Date window. Session.MinDate = 2024-01-02, Session.MaxDate = 2028-12-22. Edit if you need a wider range.
• Enum values start at 1. default(SomeEnum) is always 0 and therefore invalid — easy to spot uninitialized values in debug.

Versioning

This package follows SemVer 2.0.0. The version is derived from the latest git tag via MinVer:

• Tag v1.2.3 → package version 1.2.3
• Commits past a tag → pre-release 1.2.4-alpha.0.N+sha

The API and SL2 format carry no backward-compatibility guarantee at any version — they change in place as the work demands (the SL2 wire version stays 1, superseded shapes are replaced rather than shimmed). SemVer is honored mechanically — a breaking change still bumps the major — but that is a changelog signal, not a compatibility promise. See the caution note at the top.

Releasing

Fully automated by release-please — conventional-commit PR titles, squash-merge, then merge the bot's chore(master): release X.Y.Z PR to build, pack, and publish to nuget.org. No manual tags, no version bumps. See Docs/RELEASING.md for the full flow and one-time setup.

Local pack (no publish)

To inspect what would ship without pushing:

dotnet pack src\SquidEyes.MarketData -c Release -o artifacts

Output lands in artifacts\SquidEyes.MarketData.<version>.nupkg. Open it with NuGet Package Explorer or unzip it to verify the contents — README, LICENSE, the dll, the XML doc file, and the .snupkg companion.

Consuming locally before publishing

If you want to test the package against a real consumer before tagging, add a local feed:

mkdir C:\LocalNuGet
dotnet pack src\SquidEyes.MarketData -c Release -o C:\LocalNuGet
dotnet nuget add source C:\LocalNuGet --name LocalNuGet

Then in the consumer project: dotnet add package SquidEyes.MarketData --version <semver>. The downloader in this monorepo uses a <ProjectReference> instead, which is even simpler for live development.

Yanking a bad release

If a published version is broken, unlist it on nuget.org rather than republishing the same version — package contents are immutable. Bump the version, fix, retag.

License

MIT — see LICENSE.