ND.FW.DataValidation 1.1.0

ND.FW.DataValidation

A configurable data-validation and metadata-derivation nuget, built on top of ND.FW.RulesEngine. Everything is driven by the same Rule JSON shape — no C# changes needed to add a validation rule, switch rule providers, or add a metadata-derivation stage.

Install

dotnet add package ND.FW.DataValidation

The two halves

Both read rules through the same IRuleRepository and the same Rule JSON shape — a rule's RuleExpressionType decides which half consumes it. Validation types (Required/Mandatory, Regex, NumberRange, DateValidation, Conditional, CrossField, Completeness, Table, Validation (duplicate-row), Document, Lookup, Triangulation, FuzzyMatch, Enrichment, Standardization, Normalization) map to an ND.FW.RulesEngine workflow exactly as before. Metadata-derivation types (AttributeMapping, CandidateGrouping, MultiPageConsolidation, ConfidenceAggregation, CanonicalSelection, MultiValueConsolidation, AttributeDerivation, ConflictDetection, CrossFieldValidation) drive the consolidation pipeline instead. You can mix both kinds of rules in the same rule set returned by a provider — each service simply filters for the types it understands.

Quick start

builder.Services.AddDataValidation(builder.Configuration, options =>
{
    // register any product-specific operators alongside the built-in base set
    options.CustomOperatorTypes.Add(typeof(MyProduct.Operators));
});

// appsettings.json
{
  "DataValidation": {
    "RuleRepository": {
      "UseDatabaseProvider": true,
      "UseLocalJsonProvider": true,
      "Mode": "Additive"
    },
    "Database": {
      "Schema": "dbo",
      "RuleDetailsSpName": "sp_Get_Rule_Details",
      "RuleJsonColumnName": "Rule_Definition_JSON",
      "ParameterMap": {
        "FileTaxonomyId": "@p_File_Taxonomy_Id",
        "DocTaxonomyId": "@p_Doc_Taxonomy_Id",
        "FileDocTaxonomyId": "@p_File_Doc_Taxonomy_Id",
        "FieldId": "@p_File_Taxonomy_X_Doc_Taxonomy_X_Field_Id",
        "RuleEventStage": "@p_Rule_Event_Stage"
      }
    },
    "LocalJson": {
      "RootDirectory": "RuleSets",
      "FileNameTemplate": "{DocTaxonomyId}.json"
    }
  }
}

Validation

var criteria = RuleQueryCriteria.FromKeys(
    ("FileTaxonomyId", 12),
    ("DocTaxonomyId", 4),
    ("FieldId", 101));

var input = new Dictionary<string, object?> { ["phone"] = "+919876543210", ["pan"] = "ABCDE1234F" };

var result = await dataValidationService.ValidateAsync(criteria, input);

foreach (var r in result.RuleResults)
    Console.WriteLine($"{r.RuleName}: {(r.IsSuccess ? "PASS" : $"FAIL - {r.ErrorMessage}")}");

Metadata derivation

var candidates = new List<AttributeCandidate>
{
    new() { AttributeName = "PT_NAME_1", Value = "John A Smith", Page = 1, Position = 0, ValueStrength = 0.81, Source = "ocr" },
    new() { AttributeName = "patient full name", Value = "John Smith", Page = 3, Position = 2, ValueStrength = 0.93, Source = "model-x" },
    new() { AttributeName = "PhoneNumber", Value = "555-0100", Page = 1, ValueStrength = 0.7 },
    new() { AttributeName = "PhoneNumber", Value = "555-0199", Page = 2, ValueStrength = 0.6 },
};

var result = await metadataDerivationService.DeriveAsync(criteria, candidates);

var patientName = result.Attributes["PatientName"]; // canonical value + evidence + confidence
foreach (var conflict in result.Conflicts)
    Console.WriteLine($"Conflict on {conflict.AttributeName}: {conflict.ConflictingValues.Count} candidate values");

Rule providers — multiple, at the same time

IRuleRepository has two built-in implementations:

• DatabaseRuleRepository — calls a stored procedure. SP name, schema, and the mapping from RuleQueryCriteria.Keys to SP parameter names are all configuration (DatabaseRuleProviderOptions), not hardcoded — so this provider works for any product's own key shape, not just taxonomy IDs.
• LocalJsonRuleRepository — reads rule JSON from local files (or inline configuration). Accepts a single Rule object, a Rule array, or a { "Rules": [...] } wrapper per file.

When both are enabled, CompositeRuleRepository merges their results:

• Additive (default) — rules from every enabled provider are combined, de-duplicated by RuleName (earliest-registered provider wins on a collision).
• Fallback — providers are tried in order; the first one that returns any rules wins and the rest are skipped.

Add a third provider (e.g. a remote config service) by implementing IRuleRepository and registering it — CompositeRuleRepository accepts any IEnumerable<IRuleRepository>.

Metadata-derivation pipeline

Stages run in a fixed logical order, but only the stages whose RuleExpressionType is present in the resolved rule set actually run — a rule set with just CandidateGrouping + CanonicalSelection rules skips multi-page consolidation and cross-field validation entirely.

AttributeMapping → CandidateGrouping → MultiPageConsolidation →
ConfidenceAggregation → CanonicalSelection → MultiValueConsolidation →
AttributeDerivation → ConflictDetection → CrossFieldValidation

New in 1.1.0 — additional validation RuleExpressionTypes

Added to support rule sheets authored with Mandatory/CrossField/Completeness/ Table/Validation/Document types and Lookup rules using LookupCode/MatchType/MinimumConfidence/ReturnTop/SuggestionOnly naming. All additions are purely additive on RuleExpression — no existing field was renamed, retyped, or removed, so previously authored rule rows/JSON keep deserializing unchanged.

Important — plugin implementations are not included. Exactly like the existing Lookup/Triangulation/FuzzyMatch/Enrichment types, which only attach a "ReferenceLookupPlugin" reference and rely on something else (the consuming service or NdRulesEngine itself) to actually implement that plugin, the new Standardization-with-Operations path attaches a "StandardizationPlugin" reference by name — this nuget does not ship a StandardizationPlugin implementation. Operators.ApplyStandardization(...) is provided as the transform logic you'd call from inside that plugin, but someone still has to register a plugin named StandardizationPlugin with NdRulesEngine (the same way ReferenceLookupPlugin is registered today) before a Standardization rule with Operations will do anything beyond attach the reference.

Also note:

• CrossField's pair-comparison operators (SAME_MASTER_RECORD, DESCRIPTION_MATCH) reduce to a presence-consistency check (Operators.CrossFieldPresenceConsistent) — both fields present or both blank. Verifying that two codes truly resolve to the same master record or matching description requires reference-data access, which belongs in a Lookup-style plugin, not this stateless expression operator. Treat this as a first-pass presence check, not full semantic equivalence, until/unless a dedicated plugin is wired in.
• Table and Validation (duplicate-row) rules validate one row's data at a time (or one pre-built key list, for duplicates) per ValidateAsync call — this nuget has no concept of iterating a repeating table's rows on its own. The calling service is responsible for looping over rows and calling ValidateAsync per row (same pattern DataPostProcessingService already uses to loop over reviewItem.values).

Deriving a new attribute from other attributes

AttributeDerivation is the piece that answers "from one attribute, derive another" — as opposed to CanonicalSelection, which reconciles multiple observations of the same attribute. It reads whichever canonical attributes have already been produced (by InputFields, in order) and writes a new one (OutputField), using a built-in DerivationOperator:

{
  "RuleName": "DerivePatientFullName",
  "ErrorMessage": "",
  "RuleExpressionType": "AttributeDerivation",
  "Expression": {
    "InputFields": ["FirstName", "LastName"],
    "OutputField": "PatientFullName",
    "DerivationOperator": "Concat",
    "DerivationArgs": { "Separator": " ", "SkipBlanks": true }
  }
}

{
  "RuleName": "DerivePatientAge",
  "ErrorMessage": "",
  "RuleExpressionType": "AttributeDerivation",
  "Expression": {
    "InputFields": ["DateOfBirth"],
    "OutputField": "PatientAge",
    "DerivationOperator": "DateDiff",
    "DerivationArgs": { "Unit": "Years" }
  }
}

The derived attribute lands in MetadataDerivationResult.Attributes["PatientFullName"] just like any other canonical attribute — same Evidence/AggregatedConfidence shape (evidence here points back at the source attributes it was computed from, Source: "derived-from:FirstName" etc., confidence fixed at 1.0 since it's a deterministic computation, not a scoring decision) — and flows into ToCanonicalDictionary() for CrossFieldValidation or downstream consumers the same way. Derivation rules can chain within one rule set (a later AttributeDerivation rule can reference an earlier rule's OutputField), since rules of the same stage run in the order they appear.

Because InputFields are read from context.Result.Attributes — which is filled in by CanonicalSelection/MultiValueConsolidation, which run before AttributeDerivation in the fixed stage order — a derivation rule can only see attributes those stages (or an earlier AttributeDerivation rule) have already resolved. SourcePriorityCanonicalSelection currently runs after CrossFieldValidation (i.e. after AttributeDerivation too), so an InputFields entry that's only ever produced by that stage won't be visible yet; keep that in mind when combining rule types.

If a built-in operator doesn't cover what you need (e.g. a currency conversion against a live rate table, or fuzzy name matching), the fallback is the same one the nuget already documents for any new stage type — implement IMetadataDerivationStage under a new RuleExpressionType and register it via options.MetadataDerivationStageTypes.Add(typeof(YourStage)). No existing stage needs to change.

Note: ToCanonicalDictionary() exposes multi-value attributes (IsMultiValue: true) as a List<object?>, not a scalar. A CrossFieldValidation expression referencing such an attribute must treat it as a list (e.g. ((List<object?>)input1["PhoneNumber"]).Count > 0) rather than comparing it directly — ND.FW.RulesEngine passes the value through unchanged and does not coerce it to a string or number.

Every canonical attribute retains full evidence (EvidenceRef: value, page, position, source location, source, and computed weight) for every candidate that contributed to it, whether it won or lost — satisfying audit/traceability needs without any extra configuration.

Extending with a custom stage

Metadata-derivation stages are just another named plugin point, exactly like ND.FW.RulesEngine's INdContextPlugin:

public sealed class MyCustomStage : IMetadataDerivationStage
{
    public string RuleExpressionType => "MyCustomStageType";
    public Task ExecuteAsync(Rule rule, MetadataDerivationContext context, CancellationToken ct) { ... }
}

services.AddDataValidation(configuration, options =>
{
    options.MetadataDerivationStageTypes.Add(typeof(MyCustomStage));
});

Rule JSON compatibility

Existing rule rows/files are not restructured. Every new capability is added by:

1. New RuleExpressionType string values (data, not schema).
2. A small number of new, optional fields on RuleExpression — AttributeNameMap, GroupByKeys, ConfidenceWeights, ConflictResolutionStrategy, ConflictThreshold, IsMultiValue, MaxValues, PageOrderStrategy, CanonicalFields, DerivationOperator, DerivationArgs. All nullable; absent in existing rows and simply unused by validation rule types.
3. Reuse of one existing field, Condition, as the raw cross-field expression string for CrossFieldValidation rules (this field already existed on RuleExpression and was previously unused by any mapped type).

No existing field was renamed, retyped, or removed.

Packaging note

ND.FW.DataValidation references ND.FW.RDBM / Microsoft.Data.SqlClient directly so DatabaseRuleRepository is available out of the box, matching how DVS already depends on ND.FW.RDBM. If a consumer only ever wants the local-JSON provider (no database dependency at all — e.g. an edge/offline scenario), the cleaner long-term shape is to split this into ND.FW.DataValidation.Core (abstractions, local JSON, metadata derivation) + ND.FW.DataValidation.SqlServer (the database provider) so the SQL dependency is opt-in per package reference rather than always-transitive. That split is a bigger packaging change than this version takes on; call it out if a consumer without SQL Server ever needs this nuget.

License

Internal use.