D4S.Indexer.Domain 1.0.29

D4S.Indexer

Document indexing library for Azure AI Search: extracts text, generates vector embeddings, and uploads searchable chunks.

Quick start

var indexer = IndexerBuilder.Create("my-index")
    .WithAzureAISearch(options =>
    {
        options.Endpoint = searchEndpoint;   // Uri
        options.ApiKey = searchKey;
    })
    .WithMicrosoftFoundry(options =>
    {
        options.Endpoint = foundryEndpoint;  // Uri
        options.ApiKey = foundryKey;
        options.EmbeddingOptions.Deployment = embeddingDeployment;
        options.EmbeddingOptions.Dimensions = embeddingDimensions;
    })
    .WithLocalFiles(options => options.Path = "./documents")
    .WithFileMetadataFields()
    .WithLoggerFactory(loggerFactory)
    .Build();

var result = await indexer.IndexAsync();

WithLoggerFactory is required: pass the container's ILoggerFactory in a hosted app, or one from LoggerFactory.Create(...) in a console app.

See src/samples/ for working examples (local files, SharePoint, OCR, agentic retrieval).

Architecture

D4S.Indexer.Domain          Entities, abstractions (interfaces)
D4S.Indexer.Application     Orchestration (DocumentIndexerService, DocumentExtractor)
D4S.Indexer.Infrastructure  Azure implementations, builder, processors, sources

Built-in sources: local filesystem, multi-site SharePoint (PnP Core). Built-in processors: PDF, DOCX, XLSX, PPTX, TXT/Markdown.

Indexing modes

• Full (default): all documents fetched from every source; documents missing from the source list are deleted from the index.
• Delta (.WithDeltaMode()): only changed/new/deleted documents are provided; deletion is driven by DocumentMetadata.DeletedDate (set it and pass null for GetContentAsync). No implicit cleanup.

Both modes compare LastModifiedDate against the index to skip unchanged documents.

Reindexing after a schema change

Change detection tracks document content. When the schema changes and the documents do not — a new custom field, a different BaseUrl, a newly added ConfigureMetadata transform — every document still looks untouched, so the run skips them all and reports N skipped, 0 errors while the new field stays empty on every chunk.

Use .ForceReindex() for the run that follows such a change:

IndexerBuilder.Create("my-index")
    // …
    .AddCustomField("CiteAs", CustomFieldType.String, filterable: true)
    .ConfigureMetadata(meta => meta with { CustomFields = LookupCitation(meta) })
    .ForceReindex()          // re-extract everything once, into the existing index
    .Build();

Chunks are replaced document by document, so the index stays queryable throughout and the knowledge base/knowledge source on top of it are untouched. A field that has to be removed or retyped still needs the index itself to be recreated, which Azure only allows after the knowledge base and knowledge source referencing it are deleted, in that order.

A document is only cleared from the index once its re-extraction has actually produced chunks. If the extraction yields no text (a scan with OCR disabled) or fails, the previously indexed chunks are left in place and the document is counted under No Text or Errors — stale content is a better outcome than a document silently vanishing from a corpus-wide reindex.

Driving the flag from configuration

ForceReindex is a one-off: you want it on for the run after a schema change and off afterwards. Pass the value in rather than calling .ForceReindex() unconditionally, so switching it means changing a setting and restarting instead of editing code:

// appsettings.json / environment (Indexer__ForceReindex=true)
.ForceReindex(configuration.GetValue<bool>("Indexer:ForceReindex"))

// plain console app reading the environment directly (see src/samples/LocalFiles)
.ForceReindex(bool.TryParse(Environment.GetEnvironmentVariable("FORCE_REINDEX"), out var f) && f)

false is the default, so an unset variable leaves a normal run untouched. The same shape works for any of the boolean builder toggles — WithDeltaMode(value), ContinueOnError(value). The library takes no dependency on IConfiguration: every setting arrives through the builder, so reading it is the host's job.

Note that ForceReindex re-extracts and re-embeds every document, so it costs a full indexing run in OCR and embedding calls. It is not meant to be left on.

Builder options

IndexerBuilder.Create("index-name")
    // Required
    .WithAzureAISearch(opts => { opts.Endpoint = …; opts.ApiKey = …; })
    .WithMicrosoftFoundry(opts => { opts.Endpoint = …; opts.ApiKey = …;
                                    opts.EmbeddingOptions.Deployment = …;
                                    opts.EmbeddingOptions.Dimensions = …; })
    .WithLoggerFactory(loggerFactory)
    // Sources (at least one)
    .WithLocalFiles(opts => { opts.Path = …; opts.FileExtensions = […]; opts.BaseUrl = …; })
    .WithSharePointMultiSite(spOptions, contextFactory)
    .WithCustomDocumentSource(mySource)                        // instance — no DI container needed
    .WithCustomDocumentSource(loggerFactory => new MySource(…)) // factory, like the built-in sources
    .WithCustomDocumentSource<T>(serviceProvider, serviceKey)  // keyed DI registration
    // Optional
    .WithDeltaMode()
    .ForceReindex()                                            // ignore LastModifiedDate for one run
    .WithFileMetadataFields()
    .WithChunkReferences()                                     // pageNumber / sectionName fields
    .WithChunkSize(maxSize: 1000, overlap: 200)
    .WithBatchSize(50)
    .WithVectorCompression()
    .WithKeywordExtraction(maxKeywords: 10)                    // uses MicrosoftFoundry GptOptions
    .WithAzureDocumentIntelligence(opts => { … })              // OCR
    .WithCustomOcrService(customOcrService)                    // overrides Azure OCR
    .WithCustomDocumentProcessor(myProcessor)                  // instance
    .WithCustomDocumentProcessor(loggerFactory => new MyProcessor(…))  // factory
    .WithCustomDocumentProcessor<T>(serviceProvider, serviceKey)
    .ContinueOnError(true)
    .Filter(meta => meta.Extension == ".pdf")
    .ConfigureMetadata(meta => meta with { CustomFields = … })       // after extraction
    .ConfigureMetadataBeforeFilter(meta => meta with { … })          // during the scan, before Filter
    .AddCustomField("Status", CustomFieldType.String, filterable: true)
    .AddIndexFieldsFromAttributes<MyModel>()
    .Build();

Attaching your own fields to a document

ConfigureMetadata is the hook for this — you do not need a custom document source to add fields. It runs for every document from every source (local, SharePoint, custom), and whatever it puts in CustomFields is written to the matching index fields:

IndexerBuilder.Create("my-index")
    .WithLocalFiles(opts => opts.Path = "./documents")

    // 1. declare the field in the index schema
    .AddCustomField("CiteAs", CustomFieldType.String, filterable: true)

    // 2. populate it per document
    .ConfigureMetadata(meta => meta with
    {
        CustomFields = new Dictionary<string, object>
        {
            ["CiteAs"] = LookupCitation(meta.FileMetadata?.FileName)
        }
    })
    .Build();

Declared filterable fields are usable as OData filters by anything querying the index, including the filter parameter of the search tool in D4S.KnowledgeRetrieval.MCP.

Note that added fields do not change LastModifiedDate, so pair the first run with .ForceReindex().

Filtering on the fields you added

By default the transform runs after the document's content has been extracted — late enough to read values found inside the file, but after Filter has already decided. So this does not work:

.ConfigureMetadata(meta => meta with { CustomFields = Classify(meta) })
.Filter(meta => meta.CustomFields["DocType"] == "peer-reviewed")   // ← never matches

ConfigureMetadataBeforeFilter is the same transform moved ahead of the filter:

.ConfigureMetadataBeforeFilter(meta => meta with { CustomFields = Classify(meta) })
.Filter(meta => meta.CustomFields.GetValueOrDefault("DocType")?.ToString() == "peer-reviewed")

static Dictionary<string, object> Classify(DocumentMetadata meta) => new() { ["DocType"] = Lookup(meta) };

What it costs: at that point the file has not been opened, so the metadata is exactly what the source produced — file name, path, URL, extension, size, dates, plus any custom or filter fields the source supplied. Two values are not there yet: the author read from the document's properties, and the keywords from the processor or the AI extractor. A transform that needs those belongs in the default (post-extraction) stage.

Both stages can be used at once — they are independent slots, so call one method for each. The pre-filter stage runs first and what it writes is carried through to the index and visible to the later stage.

Fields meant only for filtering, never for the index, can also come from the source itself via DocumentMetadata.FilterFields — that is how the SharePoint source exposes list columns through FilterColumns.

To replace Azure Document Intelligence, pass the custom implementation directly:

services.AddSingleton<IOcrService, MyOcrService>();
var customOcrService = serviceProvider.GetRequiredService<IOcrService>();

var indexer = IndexerBuilder.Create("index-name")
    // Required services and document sources...
    .WithCustomOcrService(customOcrService)
    .Build();

The registered IOcrService takes precedence if WithAzureDocumentIntelligence(...) is also configured.

OCR is chosen per extension, not per document. Whenever an OCR service is configured, every document whose extension it declares in SupportedExtensions goes through it — for Azure Document Intelligence that is .pdf, the image formats, and .docx / .xlsx / .pptx. A corpus of text-native documents with one scan in it therefore pays OCR on all of them. Until the choice becomes per document, narrow it by running the scanned files as a separate pass in delta mode, or supply a WithCustomOcrService implementation that declares only the extensions you want routed.