Shiny.DocumentIntelligence 1.0.0-beta-0021

Shiny Face & Document Intelligence

On-device intelligence for .NET (MAUI + native), built on the Shiny stack. Two independent stacks:

Face intelligence — enrollment + recognition

• Capture + detect — Shiny.Maui.Controls.Camera + Shiny.Maui.Controls.Camera.Face (live preview, on-device detection via Apple Vision / Android ML Kit). Detection gives face bounds, not embeddings.
• Embed — Shiny.FaceIntelligence.Onnx: crop the detected face → ArcFace ONNX (Microsoft.ML.OnnxRuntime) → a 512-d, L2-normalized vector.
• Store + match — Shiny.FaceIntelligence.DocumentDb.Sqlite over Shiny.DocumentDb.Sqlite vector search (NearestVectors, backed by sqlite-vec / vec0).

"Training" here is enrollment: store several embeddings per person; recognition is a nearest-neighbor lookup with a cosine-distance threshold. No model is trained on-device. The pipeline is split into a core package plus swappable embedder/store packages, composed via a registration builder — pull only what you use (native embedder without ONNX, Postgres without sqlite, etc.).

Document intelligence — native modal scanner

Shiny.DocumentIntelligence puts one IDocumentScanner.ScanAsync(...) over each platform's first-party scanner: VisionKit (iOS/Mac Catalyst), ML Kit (Android, optional PDF), and Vision document segmentation on macOS AppKit (which has no document camera — it deskews picked images). No MAUI dependency.

Quick start

dotnet build Sample/Sample.csproj -f net10.0-android
dotnet build Sample/Sample.csproj -f net10.0-ios

Compose the pipeline at startup:

services.AddFaceIntelligence(face =>
{
    face.Options.MaxDistance = 0.6f;
    face.UseOnnxEmbedder(o => o.ModelBytesProvider = () => LoadBundledModel("arcface.onnx"));
    face.UseSqliteStore(o => { o.ConnectionString = "Data Source=faces.db"; o.VectorExtensionPath = "vec0"; });
});

Add the document scanner alongside it and call it from anywhere:

services.AddDocumentIntelligence(); // registers IDocumentScanner

// later, in a ViewModel/service
var result = await scanner.ScanAsync(new DocumentScanRequest { PageLimit = 10 });
if (!result.IsCancelled)
    foreach (var page in result.Pages) { /* page.ImageData (PNG/JPEG) */ }

Two assets are not in the repo and required at runtime: an ArcFace ONNX model and the sqlite-vec native binary (vec0.dylib/vec0.so). Drop the model at Sample/Resources/Raw/arcface.onnx (a compact 112×112 ArcFace like MobileFaceNet keeps app size down) — it's loaded from the app package as bytes. The app launches without these; enroll/recognize report "model missing". See CLAUDE.md for details.

Benchmarks

Recognition latency (embed + sqlite-vec NearestVectors + threshold) as the enrolled gallery grows, at the real 512-d ArcFace width. A deterministic fake embedder isolates the vector-store cost; no ONNX model involved. Reproduce with dotnet run --project tests/Shiny.FaceIntelligence.Benchmarks -c Release -- --filter '*'.

BenchmarkDotNet v0.15.8 · Apple M5 Pro · .NET 10.0.8 (Arm64) · macOS 26.5

Brute-force exact search scales roughly linearly with gallery size (VectorIndexKind.None); ~4.8 ms at 10k enrolled shots is comfortable for on-device use. Managed allocations are constant (the result set), independent of gallery size. For much larger galleries, switch to an ANN index.

Windows

On a Windows host the face and voice MAUI controls also target net10.0-windows10.0.19041.0 (minimum 10.0.17763.0), and the face camera adapter reads WindowsCameraFrame's BGRA8 buffer through the same upright-orientation and max-width pipeline as the other platforms.

./Sample/fetch-models.ps1
dotnet build Sample/Sample.csproj -f net10.0-windows10.0.19041.0

The published NuGet packages are built on macOS and do not include the Windows target yet — Windows is build-from-source only for now, and hasn't been verified against a physical Windows camera.

Development

Architecture, build/pack details, the ONNX Runtime iOS linker fix, tuning, and the open TODOs (identity hardening, coordinate space, landmark alignment) are documented in CLAUDE.md — the canonical reference for working in this repo.

Commercial Support

Shipping this to production? ShinySoft Technologies, the company that builds and maintains the Shiny libraries, offers priority issue response, support agreements, architecture reviews and private office hours. See Commercial Support or get in touch. Community support through GitHub issues and discussions stays free.