OfficeIMO.Reader 0.1.8

OfficeIMO.Reader

OfficeIMO.Reader is an optional, read-only facade that normalizes extraction across:

• Word (.docx, .docm) → Markdown chunks
• Excel (.xlsx, .xlsm) → table chunks + optional Markdown table previews
• PowerPoint (.pptx, .pptm) → slide-aligned Markdown chunks (optionally including notes)
• Markdown (.md, .markdown) → parser-aware heading chunks with preserved fenced/table blocks
• PDF (.pdf) → page-aware text chunks

The goal is to make it easy for tools like chat bots to ingest content deterministically.

Quick Start

using OfficeIMO.Reader;

foreach (var chunk in DocumentReader.Read(@"C:\Docs\Policy.docx")) {
    Console.WriteLine(chunk.Id);
    Console.WriteLine(chunk.Location.HeadingPath);
    Console.WriteLine(chunk.Markdown ?? chunk.Text);
}

Streams / Bytes

using OfficeIMO.Reader;

// Stream (does not close the stream)
using var fs = File.OpenRead(@"C:\Docs\Policy.docx");
var chunksFromStream = DocumentReader.Read(fs, "Policy.docx").ToList();

// Bytes
var bytes = File.ReadAllBytes(@"C:\Docs\Policy.docx");
var chunksFromBytes = DocumentReader.Read(bytes, "Policy.docx").ToList();

Folders

using OfficeIMO.Reader;

var chunks = DocumentReader.ReadFolder(
    folderPath: @"C:\Docs",
    folderOptions: new ReaderFolderOptions {
        Recurse = true,
        MaxFiles = 500,
        MaxTotalBytes = 500L * 1024 * 1024,
        SkipReparsePoints = true,
        DeterministicOrder = true
    },
    options: new ReaderOptions {
        MaxChars = 8_000
    }).ToList();

Folder Progress + Detailed Summary

using OfficeIMO.Reader;

var result = DocumentReader.ReadFolderDetailed(
    folderPath: @"C:\KnowledgeBase",
    folderOptions: new ReaderFolderOptions { Recurse = true, MaxFiles = 10_000 },
    options: new ReaderOptions { ComputeHashes = true },
    includeChunks: true,
    onProgress: p => Console.WriteLine($"{p.Kind}: scanned={p.FilesScanned}, parsed={p.FilesParsed}, skipped={p.FilesSkipped}, chunks={p.ChunksProduced}"));

Console.WriteLine($"Files parsed: {result.FilesParsed}");
Console.WriteLine($"Files skipped: {result.FilesSkipped}");
Console.WriteLine($"Chunks: {result.ChunksProduced}");

Database-Ready Folder Streaming

using OfficeIMO.Reader;

foreach (var doc in DocumentReader.ReadFolderDocuments(
    folderPath: @"C:\KnowledgeBase",
    folderOptions: new ReaderFolderOptions { Recurse = true, MaxFiles = 10_000, DeterministicOrder = true },
    options: new ReaderOptions { ComputeHashes = true, MaxChars = 4_000 },
    onProgress: p => Console.WriteLine($"{p.Kind}: parsed={p.FilesParsed}, skipped={p.FilesSkipped}, chunks={p.ChunksProduced}"))) {

    if (!doc.Parsed) {
        Console.WriteLine($"SKIP {doc.Path}: {string.Join("; ", doc.Warnings ?? Array.Empty<string>())}");
        continue;
    }

    // Upsert your "sources" table keyed by doc.SourceId/doc.SourceHash,
    // then upsert chunk rows from doc.Chunks keyed by chunk.ChunkHash.
    Console.WriteLine($"{doc.Path} => {doc.ChunksProduced} chunks, ~{doc.TokenEstimateTotal} tokens");
}

AI Ingestion Pattern (With Citations)

using OfficeIMO.Reader;
using System.Text;

var chunks = DocumentReader.ReadFolder(
    folderPath: @"C:\KnowledgeBase",
    folderOptions: new ReaderFolderOptions { Recurse = true, DeterministicOrder = true },
    options: new ReaderOptions { MaxChars = 4000 }).ToList();

var context = new StringBuilder();
foreach (var chunk in chunks) {
    var source = chunk.Location.Path ?? "unknown";
    var pointer = chunk.Location.Page.HasValue
        ? $"page {chunk.Location.Page.Value}"
        : chunk.Location.HeadingPath ?? $"block {chunk.Location.BlockIndex ?? 0}";

    context.AppendLine($"[source: {source} | {pointer}]");
    context.AppendLine(chunk.Markdown ?? chunk.Text);
    context.AppendLine();
}

Options

using OfficeIMO.Reader;

var options = new ReaderOptions {
    MaxChars = 8_000,
    MaxTableRows = 200,
    IncludeWordFootnotes = true,
    IncludePowerPointNotes = true,
    ExcelHeadersInFirstRow = true,
    ExcelChunkRows = 200,
    ExcelSheetName = "Data",
    ExcelA1Range = "A1:Z500",
    MarkdownChunkByHeadings = true,
    ComputeHashes = true
};

var chunks = DocumentReader.Read(@"C:\Docs\Workbook.xlsx", options).ToList();

When MarkdownChunkByHeadings is enabled, markdown inputs are chunked from parsed OfficeIMO markdown blocks instead of raw line heuristics. That means setext headings are recognized, fenced code blocks are not split by # inside the fence, oversize markdown blocks stay intact with a warning, and markdown tables are exposed through ReaderChunk.Tables. Parser-aware markdown chunks keep SourceBlockIndex/HeadingPath for stable citations, and can also expose normalized markdown provenance through NormalizedStartLine/NormalizedEndLine, HeadingSlug, SourceBlockKind, and BlockAnchor without claiming those are exact source offsets. BlockAnchor identifies the first logical markdown block in the chunk, so hosts can point at a sub-block inside a heading section when chunk splitting occurs.

Pluggable Handlers

DocumentReader now supports extension-based handler registration for modular packages.

using OfficeIMO.Reader;

DocumentReader.RegisterHandler(new ReaderHandlerRegistration {
    Id = "sample.custom",
    DisplayName = "Sample Custom Reader",
    Kind = ReaderInputKind.Text,
    Extensions = new[] { ".sample" },
    DefaultMaxInputBytes = 5 * 1024 * 1024,
    WarningBehavior = ReaderWarningBehavior.WarningChunksOnly,
    DeterministicOutput = true,
    ReadPath = (path, opts, ct) => new[] {
        new ReaderChunk {
            Id = "sample-0001",
            Kind = ReaderInputKind.Text,
            Location = new ReaderLocation { Path = path, BlockIndex = 0 },
            Text = "custom output"
        }
    }
});

var capabilities = DocumentReader.GetCapabilities();

Use DocumentReader.UnregisterHandler("sample.custom") to remove custom handlers.

GetCapabilities() exposes a stable contract surface for hosts:

• SchemaId = officeimo.reader.capability
• SchemaVersion = 1
• stream/path support flags
• advertised warning behavior and deterministic output flag
• optional DefaultMaxInputBytes metadata for handler defaults

For one-shot host bootstrap/discovery payloads:

using OfficeIMO.Reader;

var manifest = DocumentReader.GetCapabilityManifest();
var json = DocumentReader.GetCapabilityManifestJson(indented: false);

Shared Input Guards For Adapter Authors

Use ReaderInputLimits when building modular handlers so MaxInputBytes behavior stays consistent across adapters:

using OfficeIMO.Reader;

var parseStream = ReaderInputLimits.EnsureSeekableReadStream(
    stream,
    maxInputBytes: readerOptions?.MaxInputBytes,
    cancellationToken: ct,
    ownsStream: out var ownsParseStream);

You can also call ReaderInputLimits.EnforceFileSize(path, maxBytes) and ReaderInputLimits.EnforceSeekableStreamSize(stream, maxBytes) for path/seekable prechecks.

Modular Adapter Registration (Optional Packages)

Keep dependencies split by registering only adapters you need:

using OfficeIMO.Reader.Epub;
using OfficeIMO.Reader.Html;
using OfficeIMO.Reader.Json;
using OfficeIMO.Reader.Csv;
using OfficeIMO.Reader.Text;
using OfficeIMO.Reader.Xml;
using OfficeIMO.Reader.Zip;

DocumentReaderEpubRegistrationExtensions.RegisterEpubHandler(replaceExisting: true);
DocumentReaderZipRegistrationExtensions.RegisterZipHandler(replaceExisting: true);
DocumentReaderHtmlRegistrationExtensions.RegisterHtmlHandler(replaceExisting: true);
DocumentReaderCsvRegistrationExtensions.RegisterCsvHandler(replaceExisting: true);
DocumentReaderJsonRegistrationExtensions.RegisterJsonHandler(replaceExisting: true);
DocumentReaderXmlRegistrationExtensions.RegisterXmlHandler(replaceExisting: true);

// Compatibility path for existing integrations:
DocumentReaderTextRegistrationExtensions.RegisterStructuredTextHandler(replaceExisting: true);

These adapters support both path and stream dispatch via DocumentReader.Read(...).

For host-driven auto wiring (for example IX loading only present adapter assemblies), use registrar discovery:

using OfficeIMO.Reader;
using OfficeIMO.Reader.Csv;
using OfficeIMO.Reader.Epub;
using OfficeIMO.Reader.Html;
using OfficeIMO.Reader.Json;
using OfficeIMO.Reader.Xml;
using OfficeIMO.Reader.Zip;

var registrars = DocumentReader.DiscoverHandlerRegistrars(
    typeof(DocumentReaderCsvRegistrationExtensions).Assembly,
    typeof(DocumentReaderEpubRegistrationExtensions).Assembly,
    typeof(DocumentReaderZipRegistrationExtensions).Assembly,
    typeof(DocumentReaderHtmlRegistrationExtensions).Assembly,
    typeof(DocumentReaderJsonRegistrationExtensions).Assembly,
    typeof(DocumentReaderXmlRegistrationExtensions).Assembly);

DocumentReader.RegisterHandlersFromAssemblies(
    replaceExisting: true,
    typeof(DocumentReaderCsvRegistrationExtensions).Assembly,
    typeof(DocumentReaderEpubRegistrationExtensions).Assembly,
    typeof(DocumentReaderZipRegistrationExtensions).Assembly,
    typeof(DocumentReaderHtmlRegistrationExtensions).Assembly,
    typeof(DocumentReaderJsonRegistrationExtensions).Assembly,
    typeof(DocumentReaderXmlRegistrationExtensions).Assembly);

For host bootstrap where adapter assemblies are already loaded, you can avoid hardcoded type lists:

using OfficeIMO.Reader;

var bootstrap = DocumentReader.BootstrapHostFromLoadedAssemblies(
    options: new ReaderHostBootstrapOptions {
        ReplaceExistingHandlers = true,
        IncludeBuiltInCapabilities = true,
        IncludeCustomCapabilities = true,
        IndentedManifestJson = false
    });

// Typed manifest
var manifest = bootstrap.Manifest;

// Transport-safe JSON payload for host/service boundaries
var manifestJson = bootstrap.ManifestJson;

For service hosts that prefer preset integration profiles (for example IX startup), use profile overloads:

using OfficeIMO.Reader;

var bootstrap = DocumentReader.BootstrapHostFromLoadedAssemblies(
    profile: ReaderHostBootstrapProfile.ServiceDefault,
    assemblyNamePrefix: "OfficeIMO.Reader.",
    indentedManifestJson: false);

var appliedProfile = bootstrap.Profile;

Notes

• DocumentReader.Read(...) is synchronous and streaming (returns IEnumerable<T>).
• DocumentReader.ReadFolder(...) is best-effort: unreadable/corrupt/oversized files emit warning chunks and ingestion continues.
• DocumentReader.ReadFolderDocuments(...) yields one source payload at a time (ReaderSourceDocument) for easy DB upserts.
• DocumentReader.ReadFolderDetailed(...) returns ingestion counts/file statuses and can surface progress callback events.
• Chunks include SourceId/SourceHash/ChunkHash + token estimate for incremental indexing and prompt budgeting.
• The reader is best-effort and does not attempt OCR.
• Legacy binary formats (.doc, .xls, .ppt) are not supported.