Angri450.Nong.MultiModal 3.0.6

Angri450.Nong.MultiModal

Multi-modal document processing library. Cloud OCR (PaddleOCR-VL-1.6), local CPU OCR (PaddleOCR), and pure .NET image structure analysis. Speech-to-text and text-to-speech planned.

Supported Platforms

.NET 8.0 and above (net8.0, net9.0, net10.0, net11.0). Windows, macOS, Linux.

Install

dotnet add package Angri450.Nong.MultiModal

Optional: local OCR

pip install paddlepaddle paddleocr

Skip if you only use cloud API or image analysis — no Python required for those.

Three Capabilities

ImageAnalyzer — Pure .NET Image Structure Analysis

Load any PNG/JPEG image and get a detailed structural report — no OCR, no Python, no cloud API. Understand image layout in code.

using MultiModalCore;

var analyzer = new ImageAnalyzer();

// From file
var layout = analyzer.Analyze("diagram.png", targetWidth: 50);

// From bytes
var layout = analyzer.Analyze(imageBytes, targetWidth: 60);

// Print an ASCII map of the image layout
Console.WriteLine(layout.AsciiMap);

// Check whitespace ratio
Console.WriteLine($"Whitespace: {layout.WhitespaceRatio:P0}");

// Get detected content regions
foreach (var region in layout.Regions)
    Console.WriteLine($"[{region.Type}] at ({region.X},{region.Y}) {region.Width}x{region.Height}");

// Get content bounding box (useful for auto-cropping)
Console.WriteLine($"Content area: {layout.ContentWidth}x{layout.ContentHeight}");

ImageLayout Properties

ContentRegion Properties

How It Works

1. Load image via SkiaSharp → decode pixels
2. Downsample to target width (default 60 chars wide)
3. Classify each sample block: white (>240 brightness), black (<40), graphic (colorful), edge (dark gray)
4. Flood-fill to find connected non-white regions
5. Return ASCII map + region list + whitespace stats

ASCII Map Characters

Use Cases

• Debug diagram/chart output — verify layout, whitespace ratio, content positioning
• Pre-process before cloud OCR — find text regions, skip blank pages
• Validate generated images — check that charts and diagrams render correctly
• Image quality checks — detect excessive whitespace, broken rendering

Cloud OCR (PaddleOCR-VL-1.6)

var client = new PaddleOcrVlClient();  // Token from PADDLEOCR_TOKEN env var

// File → Markdown
await client.ProcessAsync("scan.pdf", "output/");

// File → Word (layout-preserving)
await client.ProcessToWordAsync("scan.pdf", "output/result.docx");

// URL → Markdown
await client.ProcessAsync("https://example.com/document.png", "output/");

// Raw bytes → Markdown
await client.ProcessAsync(fileBytes, "scan.png", "output/");

Step-by-Step Control

var jobId = await client.SubmitFileAsync("scan.pdf");
var resultUrl = await client.WaitForJobAsync(jobId, TimeSpan.FromSeconds(5));
var mdFiles = await client.DownloadResultsAsync(resultUrl, "output/");

// Structured data for custom processing
var ocrResult = await client.DownloadResultsStructuredAsync(resultUrl, "output/");
foreach (var page in ocrResult.Pages)
    foreach (var block in page.Blocks)
        Console.WriteLine($"[{block.Label}] {block.Content}");

Options

var options = new OcrOptions
{
    UseDocOrientationClassify = true,   // Auto-detect orientation
    UseDocUnwarping = true,             // Document unwarping
    UseChartRecognition = true,         // Chart parsing
};
await client.ProcessAsync("scan.pdf", "output/", options);

Local CPU OCR (PaddleOCR)

var local = new LocalOcrClient(pythonExe: "python", lang: "ch");

var (ok, msg) = await local.CheckEnvironmentAsync();
if (!ok) Console.WriteLine("Install: pip install paddlepaddle paddleocr");

var blocks = await local.RecognizeAsync("crop.png");
foreach (var b in blocks)
    Console.WriteLine($"[{b.Confidence:P0}] {b.Text}");

Word Output Pipeline

ProcessToWordAsync produces layout-preserving .docx:

1. Cloud API returns parsing_res_list — each block has block_label, block_content, block_bbox
2. LayoutToWordConverter maps blocks to Docx primitives:
   • doc_title → Title, paragraph_title → Heading, text → Body
   • image → embedded image (actual download), table → OpenXML table
   • vision_footnote → Footnote
3. Multi-column pages auto-detected from block_bbox coordinates
4. ElementOrder.RectifyTree() fixes OpenXML ordering before save

Dependencies

• Angri450.Nong.Docx — Word generation for ProcessToWordAsync output
• Angri450.Nong.ThirdParty — SkiaSharp (merged, used by ImageAnalyzer)

API Reference

ImageAnalyzer

PaddleOcrVlClient (Cloud)

LocalOcrClient (CPU)

Source

https://github.com/angri450/Nong.NET — Issues and PRs welcome.

License

MIT