Mostlylucid.DocSummarizer 4.4.1

.NET 8.0

dotnet add package Mostlylucid.DocSummarizer --version 4.4.1

NuGet\Install-Package Mostlylucid.DocSummarizer -Version 4.4.1

This command is intended to be used within the Package Manager Console in Visual Studio, as it uses the NuGet module's version of Install-Package.

<PackageReference Include="Mostlylucid.DocSummarizer" Version="4.4.1" />

For projects that support PackageReference, copy this XML node into the project file to reference the package.

<PackageVersion Include="Mostlylucid.DocSummarizer" Version="4.4.1" />
                    

                            Directory.Packages.props

<PackageReference Include="Mostlylucid.DocSummarizer" />
                    

                            Project file

For projects that support Central Package Management (CPM), copy this XML node into the solution Directory.Packages.props file to version the package.

paket add Mostlylucid.DocSummarizer --version 4.4.1

The NuGet Team does not provide support for this client. Please contact its maintainers for support.

#r "nuget: Mostlylucid.DocSummarizer, 4.4.1"

#r directive can be used in F# Interactive and Polyglot Notebooks. Copy this into the interactive tool or source code of the script to reference the package.

#:package Mostlylucid.DocSummarizer@4.4.1

#:package directive can be used in C# file-based apps starting in .NET 10 preview 4. Copy this into a .cs file before any lines of code to reference the package.

#addin nuget:?package=Mostlylucid.DocSummarizer&version=4.4.1
                    

                            Install as a Cake Addin

#tool nuget:?package=Mostlylucid.DocSummarizer&version=4.4.1
                    

                            Install as a Cake Tool

The NuGet Team does not provide support for this client. Please contact its maintainers for support.

Mostlylucid.DocSummarizer

Local-first document summarization library using BERT embeddings, RAG retrieval, and optional LLM synthesis.

Features

Local-first: Runs entirely offline using ONNX models - no API keys required
Citation grounding: Every claim is traceable to source segments
Multiple modes: Pure BERT extraction, hybrid BERT+LLM, full RAG pipeline
Format support: Markdown, PDF, DOCX, HTML, URLs
Vector storage: In-memory, DuckDB (embedded), or Qdrant (external)
Multi-framework: .NET 8, .NET 9, and .NET 10 support
OpenTelemetry: Built-in distributed tracing and metrics
Resilience: Polly-based retry, circuit breaker, and rate limiting

Installation

dotnet add package Mostlylucid.DocSummarizer

Quick Start

// Register in DI
builder.Services.AddDocSummarizer();

// Inject and use
public class MyService(IDocumentSummarizer summarizer)
{
    public async Task<string> GetSummaryAsync(string markdown)
    {
        var result = await summarizer.SummarizeMarkdownAsync(markdown);
        return result.ExecutiveSummary;
    }
}

Configuration

Basic Configuration

builder.Services.AddDocSummarizer(options =>
{
    // Use local ONNX embeddings (default, no external services)
    options.EmbeddingBackend = EmbeddingBackend.Onnx;

    // Or use Ollama for embeddings
    options.EmbeddingBackend = EmbeddingBackend.Ollama;
    options.Ollama.BaseUrl = "http://localhost:11434";
    options.Ollama.EmbedModel = "nomic-embed-text";
});

From Configuration File

{
  "DocSummarizer": {
    "EmbeddingBackend": "Onnx",
    "BertRag": {
      "VectorStore": "DuckDB",
      "ReindexOnStartup": false,
      "CollectionName": "my-documents"
    },
    "Onnx": {
      "EmbeddingModel": "AllMiniLmL6V2"
    }
  }
}

builder.Services.AddDocSummarizer(
    builder.Configuration.GetSection("DocSummarizer"));

Embedding Models

Model	Dimensions	Max Tokens	Size	Use Case
`AllMiniLmL6V2`	384	256	~23MB	Fast general-purpose (default)
`BgeSmallEnV15`	384	512	~34MB	Best quality for size
`GteSmall`	384	512	~34MB	Good all-around
`MultiQaMiniLm`	384	512	~23MB	QA-optimized
`ParaphraseMiniLmL3`	384	128	~17MB	Smallest/fastest

Summarization Modes

Mode	LLM Required	Best For
`Bert`	No	Fast extraction, offline use
`BertHybrid`	Yes	Balance of speed and fluency
`BertRag`	Yes	Production systems, large documents
`Auto`	Varies	Automatic mode selection

// Pure BERT - no LLM needed, fastest
var summary = await summarizer.SummarizeMarkdownAsync(
    markdown,
    mode: SummarizationMode.Bert);

// BertRag - full pipeline with LLM synthesis
var summary = await summarizer.SummarizeMarkdownAsync(
    markdown,
    focusQuery: "What are the key architectural decisions?",
    mode: SummarizationMode.BertRag);

Vector Store Backends

// In-memory (no persistence, fastest)
options.BertRag.VectorStore = VectorStoreBackend.InMemory;

// DuckDB (embedded file-based, default)
options.BertRag.VectorStore = VectorStoreBackend.DuckDB;

// Qdrant (external server, best for production)
options.BertRag.VectorStore = VectorStoreBackend.Qdrant;
options.Qdrant.Host = "localhost";
options.Qdrant.Port = 6334;

Output Models

DocumentSummary

record DocumentSummary(
    string ExecutiveSummary,           // Main summary text
    List<TopicSummary> TopicSummaries, // Topic-by-topic breakdown
    List<string> OpenQuestions,        // Questions that couldn't be answered
    SummarizationTrace Trace,          // Processing metadata
    ExtractedEntities? Entities);      // Named entities (people, places, etc.)

Query Mode

Ask questions about documents with evidence-grounded answers:

var answer = await summarizer.QueryAsync(
    markdown: documentContent,
    question: "What database technology is recommended?");

Console.WriteLine(answer.Answer);
Console.WriteLine($"Confidence: {answer.Confidence}");

foreach (var evidence in answer.Evidence)
{
    Console.WriteLine($"  [{evidence.SegmentId}] {evidence.Text}");
}

Segment Extraction

Extract segments without summarizing - useful for building search indexes:

var extraction = await summarizer.ExtractSegmentsAsync(markdown);

foreach (var segment in extraction.TopBySalience)
{
    Console.WriteLine($"[{segment.Type}] {segment.Text}");
    Console.WriteLine($"  Salience: {segment.SalienceScore:F2}");
}

OpenTelemetry Observability

The library includes built-in OpenTelemetry instrumentation for distributed tracing and metrics. All instrumentation follows OpenTelemetry semantic conventions.

Activity Sources (Distributed Tracing)

Source Name	Activities	Description
`Mostlylucid.DocSummarizer`	`SummarizeMarkdown`, `SummarizeFile`, `SummarizeUrl`, `Query`	Main summarization operations
`Mostlylucid.DocSummarizer.Ollama`	`OllamaGenerate`, `OllamaEmbed`	LLM API calls
`Mostlylucid.DocSummarizer.WebFetcher`	`WebFetch`, `FetchWithSecurity`	Web content fetching

Each activity includes relevant tags (e.g., url.host, http.response.status_code, error.type) for filtering and analysis.

Metrics

DocumentSummarizerService Metrics

Metric	Type	Unit	Description
`docsummarizer.summarizations`	Counter	requests	Total summarization requests
`docsummarizer.queries`	Counter	requests	Total query requests
`docsummarizer.summarization.duration`	Histogram	ms	Summarization duration
`docsummarizer.document.size`	Histogram	bytes	Document sizes processed
`docsummarizer.errors`	Counter	errors	Total errors by type

OllamaService Metrics

Metric	Type	Unit	Description
`docsummarizer.ollama.generate.requests`	Counter	requests	LLM generation requests
`docsummarizer.ollama.embed.requests`	Counter	requests	Embedding requests
`docsummarizer.ollama.generate.duration`	Histogram	ms	Generation duration
`docsummarizer.ollama.embed.duration`	Histogram	ms	Embedding duration
`docsummarizer.ollama.prompt.tokens`	Histogram	tokens	Prompt token counts
`docsummarizer.ollama.response.tokens`	Histogram	tokens	Response token counts
`docsummarizer.ollama.errors`	Counter	errors	LLM errors by type
`docsummarizer.ollama.circuit_breaker`	Counter	transitions	Circuit breaker state changes

WebFetcher Metrics

Metric	Type	Unit	Description
`docsummarizer.webfetch.requests`	Counter	requests	Web fetch requests
`docsummarizer.webfetch.duration`	Histogram	ms	Fetch duration
`docsummarizer.webfetch.errors`	Counter	errors	Fetch errors by type
`docsummarizer.webfetch.retries`	Counter	retries	Retry attempts
`docsummarizer.webfetch.ratelimits`	Counter	responses	HTTP 429 rate limit hits
`docsummarizer.webfetch.circuit_breaker`	Counter	transitions	Circuit breaker state changes

Metric Dimensions (Tags)

Common tags available on metrics:

Tag	Metrics	Values
`mode`	summarizations, webfetch	`Bert`, `BertRag`, `MapReduce`, `Simple`, `Playwright`
`error.type`	errors	`security`, `http`, `timeout`, `operation`, `unknown`
`url.host`	webfetch	Target hostname
`model`	ollama	LLM model name
`state`	circuit_breaker	`opened`, `closed`, `half-opened`
`attempt`	retries	Retry attempt number

Example: Wire up in ASP.NET Core

builder.Services.AddOpenTelemetry()
    .WithTracing(tracing => tracing
        .AddSource("Mostlylucid.DocSummarizer")
        .AddSource("Mostlylucid.DocSummarizer.Ollama")
        .AddSource("Mostlylucid.DocSummarizer.WebFetcher")
        .AddOtlpExporter())
    .WithMetrics(metrics => metrics
        .AddMeter("Mostlylucid.DocSummarizer")
        .AddMeter("Mostlylucid.DocSummarizer.Ollama")
        .AddMeter("Mostlylucid.DocSummarizer.WebFetcher")
        .AddOtlpExporter());

Example: Console Application

using var tracerProvider = Sdk.CreateTracerProviderBuilder()
    .AddSource("Mostlylucid.DocSummarizer")
    .AddSource("Mostlylucid.DocSummarizer.Ollama")
    .AddSource("Mostlylucid.DocSummarizer.WebFetcher")
    .AddConsoleExporter()
    .Build();

using var meterProvider = Sdk.CreateMeterProviderBuilder()
    .AddMeter("Mostlylucid.DocSummarizer")
    .AddMeter("Mostlylucid.DocSummarizer.Ollama")
    .AddMeter("Mostlylucid.DocSummarizer.WebFetcher")
    .AddConsoleExporter()
    .Build();

Grafana/Jaeger Integration

Send telemetry to Jaeger or Grafana Tempo:

builder.Services.AddOpenTelemetry()
    .WithTracing(tracing => tracing
        .AddSource("Mostlylucid.DocSummarizer")
        .AddSource("Mostlylucid.DocSummarizer.Ollama")
        .AddSource("Mostlylucid.DocSummarizer.WebFetcher")
        .AddOtlpExporter(o => o.Endpoint = new Uri("http://jaeger:4317")))
    .WithMetrics(metrics => metrics
        .AddMeter("Mostlylucid.DocSummarizer")
        .AddMeter("Mostlylucid.DocSummarizer.Ollama")
        .AddMeter("Mostlylucid.DocSummarizer.WebFetcher")
        .AddPrometheusExporter());  // For Prometheus/Grafana

HTTP Resilience

The WebFetcher service includes Polly-based resilience:

Circuit Breaker: Opens after 5 failures in 30s, stays open for 60s
Retry: Exponential backoff with jitter (3 attempts, 0.5s-30s)
Rate Limiting: Respects HTTP 429 Retry-After headers
Permanent Failures: 403/404/401 fail immediately with clear messages

Dependencies

Supported: .NET 8.0+, .NET 9.0+, .NET 10.0+
Included: ONNX Runtime, Markdig, PdfPig, OpenXml, AngleSharp, Polly, OpenTelemetry.Api
Optional: Ollama (for LLM synthesis), Docling (for complex PDF conversion)

License

MIT

Product	Compatible and additional computed target framework versions.
.NET	net8.0 is compatible. net8.0-android was computed. net8.0-browser was computed. net8.0-ios was computed. net8.0-maccatalyst was computed. net8.0-macos was computed. net8.0-tvos was computed. net8.0-windows was computed. net9.0 is compatible. net9.0-android was computed. net9.0-browser was computed. net9.0-ios was computed. net9.0-maccatalyst was computed. net9.0-macos was computed. net9.0-tvos was computed. net9.0-windows was computed. net10.0 is compatible. net10.0-android was computed. net10.0-browser was computed. net10.0-ios was computed. net10.0-maccatalyst was computed. net10.0-macos was computed. net10.0-tvos was computed. net10.0-windows was computed.

Product

.NET

Compatible target framework(s)

Included target framework(s) (in package)

Learn more about Target Frameworks and .NET Standard.

net10.0
- AngleSharp (>= 1.4.0)
- DocumentFormat.OpenXml (>= 3.3.0)
- dotnet-stop-words (>= 1.1.0)
- DuckDB.NET.Data.Full (>= 1.4.3)
- Markdig (>= 0.44.0)
- Microsoft.Extensions.Configuration.Abstractions (>= 10.0.1)
- Microsoft.Extensions.DependencyInjection.Abstractions (>= 10.0.1)
- Microsoft.Extensions.Hosting.Abstractions (>= 10.0.1)
- Microsoft.Extensions.Logging.Abstractions (>= 10.0.1)
- Microsoft.Extensions.Options (>= 10.0.1)
- Microsoft.Extensions.Options.ConfigurationExtensions (>= 10.0.1)
- Microsoft.ML.OnnxRuntime (>= 1.23.0)
- Microsoft.Playwright (>= 1.57.0)
- Newtonsoft.Json (>= 13.0.4)
- OpenTelemetry.Api (>= 1.12.0)
- PdfPig (>= 0.1.13)
- Polly.Core (>= 8.6.5)
- Qdrant.Client (>= 1.16.1)
- SixLabors.ImageSharp (>= 3.1.12)
- System.IO.Hashing (>= 10.0.1)
net8.0
- AngleSharp (>= 1.4.0)
- DocumentFormat.OpenXml (>= 3.3.0)
- dotnet-stop-words (>= 1.1.0)
- DuckDB.NET.Data.Full (>= 1.4.3)
- Markdig (>= 0.44.0)
- Microsoft.Extensions.Configuration.Abstractions (>= 10.0.1)
- Microsoft.Extensions.DependencyInjection.Abstractions (>= 10.0.1)
- Microsoft.Extensions.Hosting.Abstractions (>= 10.0.1)
- Microsoft.Extensions.Logging.Abstractions (>= 10.0.1)
- Microsoft.Extensions.Options (>= 10.0.1)
- Microsoft.Extensions.Options.ConfigurationExtensions (>= 10.0.1)
- Microsoft.ML.OnnxRuntime (>= 1.23.0)
- Microsoft.Playwright (>= 1.57.0)
- Newtonsoft.Json (>= 13.0.4)
- OpenTelemetry.Api (>= 1.12.0)
- PdfPig (>= 0.1.13)
- Polly.Core (>= 8.6.5)
- Qdrant.Client (>= 1.16.1)
- SixLabors.ImageSharp (>= 3.1.12)
- System.Diagnostics.DiagnosticSource (>= 10.0.1)
- System.IO.Hashing (>= 10.0.1)
net9.0
- AngleSharp (>= 1.4.0)
- DocumentFormat.OpenXml (>= 3.3.0)
- dotnet-stop-words (>= 1.1.0)
- DuckDB.NET.Data.Full (>= 1.4.3)
- Markdig (>= 0.44.0)
- Microsoft.Extensions.Configuration.Abstractions (>= 10.0.1)
- Microsoft.Extensions.DependencyInjection.Abstractions (>= 10.0.1)
- Microsoft.Extensions.Hosting.Abstractions (>= 10.0.1)
- Microsoft.Extensions.Logging.Abstractions (>= 10.0.1)
- Microsoft.Extensions.Options (>= 10.0.1)
- Microsoft.Extensions.Options.ConfigurationExtensions (>= 10.0.1)
- Microsoft.ML.OnnxRuntime (>= 1.23.0)
- Microsoft.Playwright (>= 1.57.0)
- Newtonsoft.Json (>= 13.0.4)
- OpenTelemetry.Api (>= 1.12.0)
- PdfPig (>= 0.1.13)
- Polly.Core (>= 8.6.5)
- Qdrant.Client (>= 1.16.1)
- SixLabors.ImageSharp (>= 3.1.12)
- System.Diagnostics.DiagnosticSource (>= 10.0.1)
- System.IO.Hashing (>= 10.0.1)

NuGet packages

This package is not used by any NuGet packages.

GitHub repositories

This package is not used by any popular GitHub repositories.

Version	Downloads	Last Updated
4.4.1	89	1/3/2026
4.3.1	92	12/29/2025
4.3.0	92	12/29/2025
4.2.0	88	12/29/2025
4.2.0-rc0	90	12/29/2025
4.1.0	89	12/28/2025
4.1.0-proview0	84	12/28/2025
4.1.0-preview2	88	12/28/2025
4.0.3-preview1	85	12/28/2025
4.0.2-preview1	84	12/28/2025
4.0.1	100	12/28/2025
4.0.1-alpha0	83	12/28/2025
4.0.0	86	12/28/2025
0.0.0-alpha.0.469	43	12/28/2025
0.0.0-alpha.0.467	47	12/28/2025
0.0.0-alpha.0.455	128	12/24/2025

## v4.0.0

### New Features

#### OpenTelemetry Instrumentation
Full observability with distributed tracing and metrics for production monitoring.

**Activity Sources (Tracing):**
- `Mostlylucid.DocSummarizer` - SummarizeMarkdown, SummarizeFile, SummarizeUrl, Query
- `Mostlylucid.DocSummarizer.Ollama` - OllamaGenerate, OllamaEmbed
- `Mostlylucid.DocSummarizer.WebFetcher` - WebFetch, FetchWithSecurity

**Metrics:**
- DocumentSummarizer: `docsummarizer.summarizations`, `docsummarizer.queries`, `docsummarizer.summarization.duration`, `docsummarizer.document.size`, `docsummarizer.errors`
- OllamaService: `docsummarizer.ollama.generate.requests`, `docsummarizer.ollama.embed.requests`, `docsummarizer.ollama.generate.duration`, `docsummarizer.ollama.embed.duration`, `docsummarizer.ollama.prompt.tokens`, `docsummarizer.ollama.response.tokens`, `docsummarizer.ollama.errors`, `docsummarizer.ollama.circuit_breaker`
- WebFetcher: `docsummarizer.webfetch.requests`, `docsummarizer.webfetch.duration`, `docsummarizer.webfetch.errors`, `docsummarizer.webfetch.retries`, `docsummarizer.webfetch.ratelimits`, `docsummarizer.webfetch.circuit_breaker`

#### Enhanced HTTP Resilience
Polly-based resilience for WebFetcher with production-grade fault tolerance.

- **Circuit breaker**: Opens after 5 failures in 30s, stays open for 60s, half-open probe before recovery
- **Retry**: Exponential backoff with decorrelated jitter (3 attempts max, 0.5s-30s delay range)
- **Rate limiting**: Respects HTTP 429 Retry-After headers (both delta and date formats)
- **Permanent failures**: 400, 401, 402, 403, 404, 405, 410 fail immediately with descriptive messages (no retry)

#### New Exception Types
- `WebFetchPermanentException` - For non-retryable HTTP errors with StatusCode property
- `SecurityException` - For SSRF and security policy violations

### Dependencies
- Added `OpenTelemetry.Api` 1.12.0
- Added `System.Diagnostics.DiagnosticSource` 10.0.1
- Added `Polly.Core` 8.6.5 (for WebFetcher resilience)

---

## v3.0.0

### Improvements
- Added .NET 8 LTS support (now targets net8.0, net9.0, and net10.0)
- Fixed DI registration - all services now properly registered
- 101 unit tests covering OnnxConfig, OnnxModelRegistry, and VectorMath utilities

---

## v2.0.0

### New Features
- **.NET 10 Support** - Now targets net9.0 and net10.0 (dropped net8.0)
- **Improved Salience Scoring** - Substantive content now ranks higher than short headings
- Length-based quality scoring: segments shorter than 80 characters are penalized proportionally
- Reduced heading boosts from 4.5x to ~2x for document titles
- Short headings no longer dominate top segments

### Configuration Options
New `ExtractionConfig` properties for fine-tuning salience:
- `IdealMinLength` (default: 80) - Minimum length for full quality score
- `IdealMaxLength` (default: 500) - Maximum length before diminishing returns
- `MinLengthQualityScore` (default: 0.3) - Score floor for very short segments
- `HeadingBoost` (default: 2.0) - Boost multiplier for headings
- `DocumentTitleBoost` (default: 1.5) - Boost multiplier for document titles

### Dependencies
- Updated to latest Microsoft.Extensions.* packages (10.0.1)
- Updated System.IO.Hashing to 10.0.1
- Updated PdfPig to 0.1.13

---

## v1.0.0

Initial release of Mostlylucid.DocSummarizer.Core - a local-first document summarization library.

### Features
- BERT-based extractive summarization using local ONNX models
- RAG pipeline with semantic retrieval and LLM synthesis
- Support for Markdown, PDF, DOCX, HTML, and URLs
- Citation grounding - every claim traceable to source
- Multiple vector store backends: DuckDB (default), Qdrant, In-Memory
- Configurable summary templates and styles
- Web fetching with Playwright support for JavaScript-rendered pages

### Dependencies
- Requires .NET 9.0 or .NET 10.0
- Optional: Ollama for LLM synthesis
- Optional: Docling for advanced PDF/DOCX conversion