Tokenizer 3.0.0-beta.1

Tokenizer

A .NET library that extracts structured data from text using pattern matching and maps it onto typed objects.

Installation

dotnet add package Tokenizer --version 3.0.0-beta.1

Or add a PackageReference:

<PackageReference Include="Tokenizer" Version="3.0.0-beta.1" />

Quick Start

Define a pattern with {TokenName} placeholders. Tokenizer matches the surrounding text and extracts values into your object's properties.

var tokenizer = new Tokenizer();

var pattern = "Name: {Name}, Age: {Age:ToInt()}";
var input = "Name: Alice, Age: 30";

var template = tokenizer.Compile(pattern).Template;
var person = tokenizer.Tokenize<Person>(template, input);

Assert.Equal("Alice", person.Name);
Assert.Equal(30, person.Age);

Tokens work by matching the preceding text (the "preamble") in the input. When a match is found, the text after the preamble is extracted as the token value. Extraction continues until the next token's preamble is found or the input ends.

Features

In-Order vs Out-of-Order Processing

By default, tokens must appear in the order defined by the template. Set OutOfOrder: true in front matter (or OutOfOrderTokens = true on TokenizerOptions) to allow tokens to match in any order. In-order mode supports the ? suffix to mark tokens as optional.

var pattern =
@"---
OutOfOrder: false
---
First Name: {FirstName}
Middle Name: {MiddleName?}
Last Name: {LastName}";

var input =
@"First Name: Alice
Last Name: Smith";

var template = tokenizer.Compile(pattern).Template;
var student = tokenizer.Tokenize<Student>(template, input);

Assert.Equal("Alice", student.FirstName);
Assert.Null(student.MiddleName);
Assert.Equal("Smith", student.LastName);

Multiline Tokens

Tokens can span multiple lines. A token's value extends until the next token's preamble is found.

var pattern =
@"Comments:
{Comment:Trim()}

Name:
{Name}";

var input =
@"Comments:
10/10
Would parse text again.

Name:
Bob";

var template = tokenizer.Compile(pattern).Template;
var review = tokenizer.Tokenize<Review>(template, input);

Assert.Equal("10/10\nWould parse text again.", review.Comment);
Assert.Equal("Bob", review.Name);

Newline Termination

Append $ to a token name to terminate extraction at the end of the current line.

var pattern = @"Name: {Name$}
Age: {Age:IsNumeric()}";

var input = @"Name: Bob
Surname: Jones
Age: 31";

var template = tokenizer.Compile(pattern).Template;
var person = tokenizer.Tokenize<Person>(template, input);

Assert.Equal("Bob", person.Name);  // Not "Bob\nSurname: Jones"
Assert.Equal(31, person.Age);

Repeating Tokens

Append * to extract multiple values into a List<> or IList<> property. Repeating tokens are implicitly optional.

var pattern =
@"Name: {Manager.Name}
Employee: {Manager.Manages*}
Number: {Manager.Number}";

var input =
@"Name: Sue
Employee: Alice
Employee: Bob
Employee: Charles
Number: 1234";

var template = tokenizer.Compile(pattern).Template;
var manager = tokenizer.Tokenize<Manager>(template, input);

Assert.Equal("Sue", manager.Name);
Assert.Equal(3, manager.Manages.Count);
Assert.Equal("Alice", manager.Manages[0]);
Assert.Equal(1234, manager.Number);

Required and Optional Fields

Mark tokens as required with !. If a required token is missing, Tokenize<T> returns null (because TokenizeResult.Success is false).

var pattern = @"First Name: {FirstName!}, Last Name: {LastName!}";
var input = "First Name: Alice";

var template = tokenizer.Compile(pattern).Template;

// Tokenize<T> returns null when required tokens are missing
var student = tokenizer.Tokenize<Student>(template, input);
Assert.Null(student);

// Use the raw result to access partial matches
var result = tokenizer.Tokenize(template, input);
Assert.False(result.Success);

Configuration

Options can be set per-instance or per-template via YAML front matter.

// Per-instance
var tokenizer = new Tokenizer(new TokenizerOptions
{
    TrimTrailingWhiteSpace = false,
    OutOfOrderTokens = true
});

// Per-template front matter (overrides instance settings)
var pattern = @"---
TrimTrailingWhitespace: true
CaseSensitive: false
---
First Name: {FirstName}
Last Name: {LastName}";

Front matter is placed between --- markers at the start of a template. Lines starting with # are comments.

Data Transformers

Extracted values can be transformed before assignment. Chain multiple transformers with commas.

var pattern = "Name: {Name:Trim(),ToLower()}";
var input = "Name:      Alice      ";

var template = tokenizer.Compile(pattern).Template;
var person = tokenizer.Tokenize<Person>(template, input);

Assert.Equal("alice", person.Name);

Data Validators

Validators test extracted values. If validation fails, the engine skips the current match and searches for the next one.

var pattern = "Age: {Age:IsNumeric}";
var input = "Age: Ten, Age: 11";

var template = tokenizer.Compile(pattern).Template;
var person = tokenizer.Tokenize<Person>(template, input);

Assert.Equal(11, person.Age);

In this example, "Ten" fails the IsNumeric check, so the engine continues scanning and finds "11".

Template Compilation and Caching

Compile templates once and reuse them across multiple tokenization calls.

var compiled = tokenizer.Compile("Name: {Name}, Age: {Age:ToInt()}");

// Check for compilation errors
if (compiled.Errors.Any())
{
    // Handle errors
}

// Reuse the compiled template
var template = compiled.Template;
var person1 = tokenizer.Tokenize<Person>(template, input1);
var person2 = tokenizer.Tokenize<Person>(template, input2);

Multi-Template Matching

Use TemplateMatcher to match input against multiple templates and select the best result.

var matcher = new TemplateMatcher(tokenizer);
matcher.RegisterTemplate("Name: {Name}", "person");
matcher.RegisterTemplate("Order: {OrderId}", "order");

var person = matcher.Tokenize<Person>("Name: Alice");
Assert.Equal("Alice", person.Name);

Async and Streaming

Compile and tokenize from streams or readers for large inputs.

using var reader = new StreamReader(stream);
var template = (await tokenizer.CompileAsync(reader)).Template;
var person = await tokenizer.TokenizeAsync<Person>(template, reader);

Dependency Injection

Register Tokenizer services with the built-in DI container.

// Default options
services.AddTokenizer();

// With explicit options
services.AddTokenizer(new TokenizerOptions { OutOfOrderTokens = true });

// From configuration section
services.AddTokenizer(configuration.GetSection("Tokenizer"));

This registers ITokenizer, Tokenizer, and ITemplateMatcher as singletons.

Built-in Transformers

Built-in Validators

Custom Transformers and Validators

Implement ITokenTransformer or ITokenValidator and register them via options.

using Tokens.Transformers;

public sealed class ReverseTransformer : ITokenTransformer
{
    public bool TryTransform(object value, string[] args, out object transformed)
    {
        if (value is string s)
        {
            transformed = new string(s.Reverse().ToArray());
            return true;
        }

        transformed = value;
        return false;
    }
}

// Register it
var options = new TokenizerOptions()
    .WithTransformer<ReverseTransformer>();
var tokenizer = new Tokenizer(options);

Validators follow the same pattern with ITokenValidator.IsValid(object value, params string[] args).

Configuration Reference

These options can be set on TokenizerOptions (constructor) or in template front matter (YAML between --- markers).

Architecture

See ARCHITECTURE.md for a detailed overview of the compilation pipeline and tokenization engine.

Contributing

See CONTRIBUTING.md for guidelines on building, testing, and submitting changes.

License

MIT. See LICENSE.txt.