Hawkynt.FileFormats.Archives 1.0.0.583

Hawkynt.FileFormats.Archives

Pure-managed compression streams + archive containers extracted from CompressionWorkbench. Sister package to Hawkynt.FileFormats.Audio / Hawkynt.FileFormats.FileSystems / Hawkynt.FileFormats.Images, all built on top of Hawkynt.Compression.Core.

The package bundles every archive-domain assembly into lib/ and is the workbench's "swiss-army knife" surface — install it and you can read or write practically any compression stream or archive container in pure managed code, with no native runtime dependency on zlib / liblzma / libarchive / libbz2.

When to use this package

• You're writing a tool that needs to enumerate / extract / write archives in a wide range of formats from a single .NET process
• You want to inspect installer payloads (.msi / .nsis / Inno Setup / .appimage / .snap / .deb / .rpm / .apk / .ipa / .nupkg / .crx / .xpi / etc.) without running the installer
• You're processing compression streams without container framing — gzip / bzip2 / xz / zstd / lz4 / snappy / brotli / lzma / lzop and the long historical tail (compress / pack / arc / lzh / quicklz / lzfse / etc.)
• You're handling ZIP-based document bundles (Office Open XML — docx / xlsx / pptx; OpenDocument — odt / ods / odp; ePub; Comic-Book .cbz; KML .kmz; .vsdx)
• You're investigating game / engine / asset packages — Bethesda BSA / BA2, Unreal pak / .upk, Unity bundles, Quake / Doom WAD, Source Engine VPK, Mass Effect SFAR, RPG Maker RGSS, Ren'Py RPA, Godot PCK, Mortal Kombat MIX, Total Annihilation HPI / TFC, Master of Orion, etc.

Skip it when:

• You only need ZIP at default settings — System.IO.Compression.ZipArchive does that with the BCL's GZip / Deflate paths
• You need format-specific RAR / 7z encoder quality at parity with the original tool — those ship as native libraries with proprietary or LGPL fallback licensing concerns; this package prioritises read-coverage and clean-room license over pixel-identical output

Quick start — extract a generic archive

using Compression.Registry;       // FormatRegistry.DetectAndOpen
using FileFormat.Tar;             // ensures TarFormatDescriptor is registered
using FileFormat.SevenZip;        // ditto for 7z
// (every IFormatDescriptor in the package self-registers via the
//  source-generator at app start; just reference the assemblies you need)

using var input = File.OpenRead("payload.tar.gz");
var ops = FormatRegistry.DetectArchiveOperations(input);
foreach (var entry in ops.List(input))
  Console.WriteLine($"{entry.Name,-40}  {entry.Size,12:N0}");

Quick start — round-trip a single compression stream

using FileFormat.Brotli;

byte[] original   = File.ReadAllBytes("page.html");
byte[] compressed = new BrotliFormatDescriptor().Compress(original);
byte[] roundTrip  = new BrotliFormatDescriptor().Decompress(compressed);
Debug.Assert(original.SequenceEqual(roundTrip));

Contents

State legend:

• R — read-only: List / Extract / Test. Cannot create new archives.
• WORM — Write-Once-Read-Many: read AND can synthesise a fresh archive from scratch (IArchiveCreatable), but cannot modify an existing archive in place.
• R/W — full read + write, including in-place add / replace / remove (IArchiveInPlaceModify). Currently no archive in this package is R/W; modifying an archive means "extract → modify → re-create from scratch".

Compression streams (single-file codecs without container framing)

Archive containers (multi-file)

Software-package containers

Office Open XML / OpenDocument / web archive zip-bundles

Game / engine / install / runtime / Amiga / vintage long-tail formats

Backup-software disk images

Whole-system / partition backups from consumer + enterprise backup suites. Several were reverse-engineered directly from vendor binaries (no published spec) — the descriptor for each one names exactly which fields are decoded versus documented-TODO. R/W formats survive their own round-trip; WORM formats can produce a fresh image that the same reader walks back.

Detailed format reference

The tables below mirror the canonical "what does each format actually support" reference from the source repo. Each row links to the upstream spec the implementation was validated against plus a one-line note covering scope / limitations. WORM = can produce a fresh archive that round-trips; - = read-only.

Archive containers (canonical)

ZIP-derived containers

All delegate to the ZIP reader/writer. WORM (Yes) means a fresh container can be produced with the correct internal layout for that flavour.

OLE2 Compound File variants

Microsoft binary-office formats built on the OLE2 / Compound File Binary (CFB) container. WORM creation produces a structurally-valid CFB envelope (round-trips through our reader and other permissive CFB tools like libgsf / Apache POI) but is not a real Word/Excel/PowerPoint/Outlook document — those require generating each application's internal binary stream layout, which is out of scope. Limitations: ~6.8 MB total file size (109 FAT sectors, no DIFAT chain), single root storage, stream names ≤ 31 UTF-16 chars.

Compression-stream formats (single-file codecs)

Compound formats

tar.gz, tar.bz2, tar.xz, tar.zst, tar.lz4, tar.lz, tar.br — auto-detected on read; matching writer composes the inner TAR with the outer compression stream.

Pseudo-archive containers

Formats whose binary structure already addresses N independent payloads, even though they're not traditionally seen as archives. Each surfaces an IArchiveFormatOperations.List with one entry per inner payload (resource / icon / font / glyph / message / segment / track …).

Streaming / video / subtitle containers

Multi-track / multi-segment containers from the streaming and broadcast world. Each demuxes into per-track or per-segment archive entries.

Scientific / ML data containers

CAD / 3D scene formats

Executable packer detection

FileFormat.ExePackers is a pseudo-archive that detects-and-surfaces evidence about executable packers / PE protectors. Each detected packer produces a metadata.ini (signature offset, version byte, packer-header fields) plus a packed_payload.bin (or in-process decompressed body for UPX). The detector handles: PKLITE / LZEXE / Petite / Shrinkler / FSG / MEW / MPRESS / Crinkler / kkrunchy / ASPack / NsPack / Yoda's Crypter / ASProtect / Themida / VMProtect.

UPX gets its own descriptor (FileFormat.Upx) with a hardened detection pipeline (BSS-style first-section + RWX flags + entry-in-last-section + payload-entropy fingerprint that catches binaries with the UPX! magic wiped) and an in-process decompressor for NRV2B / NRV2D / NRV2E (LE32 + LE16 + LE8 variants) and LZMA payloads via the BB_Nrv2{b,d,e} and BB_Lzma building blocks. PE header reconstruction (IAT / OEP) is delegated to the original upx -d.

Backup-software disk images

Whole-system / partition backups from consumer + enterprise backup suites. Closed-source formats were elevated from Stage-0 detection-only via direct binary reverse engineering of publicly-distributed vendor binaries; each descriptor's Description field names exactly which fields are decoded against the binary versus documented-TODO. R/W = our writer + reader round-trip; WORM = our writer emits a fresh image our reader walks back, but vendor byte-compat stays explicitly out of scope (no real vendor samples available for clean-room validation, or vendor tooling is restore-only).

Known limitations

Code paths that throw NotSupportedException or NotImplementedException rather than silently producing wrong output:

Modern packaging (read-only)

Versioning

Version-locked 1:1 with Hawkynt.Compression.Core. Pin both at the same version.

License

LGPL-3.0-or-later. See the source repository for the full license text.