TinyTokenizer 0.1.0

TinyTokenizer

A high-performance, zero-allocation tokenizer library for .NET that parses text into abstract tokens using ReadOnlySpan<char> for maximum efficiency.

Features

• Zero-allocation parsing — Uses ReadOnlySpan<char> internally for fast, allocation-free text traversal
• Recursive declaration blocks — Automatically parses nested {}, [], and () blocks with child tokens
• Configurable symbols — Define which characters are recognized as symbol tokens
• Immutable tokens — All token types are immutable record classes
• Error recovery — Gracefully handles malformed input with ErrorToken and continues parsing

Installation

Add a reference to the TinyTokenizer project or include the source files in your solution.

Quick Start

using TinyTokenizer;

// Create a tokenizer with source text
var tokenizer = new Tokenizer("func(a, b)".AsMemory());
var tokens = tokenizer.Tokenize();

// tokens contains:
// - TextToken("func")
// - BlockToken("(a, b)") with children:
//   - TextToken("a")
//   - SymbolToken(",")
//   - WhitespaceToken(" ")
//   - TextToken("b")

Token Types

BlockToken Properties

var tokenizer = new Tokenizer("{inner content}".AsMemory());
var tokens = tokenizer.Tokenize();
var block = (BlockToken)tokens[0];

block.FullContent;      // "{inner content}" (includes delimiters)
block.InnerContent;     // "inner content" (excludes delimiters)
block.Children;         // ImmutableArray<Token> of parsed inner tokens
block.OpeningDelimiter; // '{'
block.ClosingDelimiter; // '}'
block.Type;             // TokenType.BraceBlock

Configuration

Customize the tokenizer with TokenizerOptions:

// Default symbols: / : , ; = + - * < > ! & | . @ # ? % ^ ~ \
var options = TokenizerOptions.Default;

// Add custom symbols
options = TokenizerOptions.Default.WithAdditionalSymbols('$', '_');

// Remove symbols (they become part of text tokens)
options = TokenizerOptions.Default.WithoutSymbols('/');

// Replace entire symbol set
options = TokenizerOptions.Default.WithSymbols(':', ',', ';');

// Use with tokenizer
var tokenizer = new Tokenizer(source.AsMemory(), options);

Nested Blocks

Declaration blocks are parsed recursively:

var tokenizer = new Tokenizer("{outer [inner (deepest)]}".AsMemory());
var tokens = tokenizer.Tokenize();

var braceBlock = (BlockToken)tokens[0];           // {outer [inner (deepest)]}
var bracketBlock = (BlockToken)braceBlock.Children[2];  // [inner (deepest)]
var parenBlock = (BlockToken)bracketBlock.Children[2];  // (deepest)

Error Handling

The tokenizer produces ErrorToken for malformed input and continues parsing:

var tokenizer = new Tokenizer("}hello{".AsMemory());
var tokens = tokenizer.Tokenize();

// tokens contains:
// - ErrorToken("}", "Unexpected closing delimiter '}'", position: 0)
// - TextToken("hello")
// - ErrorToken("{", "Unclosed block starting with '{'", position: 6)

// Check for errors
if (tokens.HasErrors())
{
    foreach (var error in tokens.GetErrors())
    {
        Console.WriteLine($"Error at {error.Position}: {error.ErrorMessage}");
    }
}

Utility Extensions

Extensions on ImmutableArray<Token> for common operations:

// Check if any errors exist (including nested)
bool hasErrors = tokens.HasErrors();

// Get all errors (including nested)
IEnumerable<ErrorToken> errors = tokens.GetErrors();

// Get all tokens of a specific type (including nested)
IEnumerable<TextToken> textTokens = tokens.OfTokenType<TextToken>();
IEnumerable<BlockToken> blocks = tokens.OfTokenType<BlockToken>();

API Reference

Tokenizer (ref struct)

// Constructor
public Tokenizer(ReadOnlyMemory<char> source, TokenizerOptions? options = null)

// Tokenize the source
public ImmutableArray<Token> Tokenize()

Token (abstract record)

public abstract record Token(ReadOnlyMemory<char> Content, TokenType Type)
{
    public ReadOnlySpan<char> ContentSpan { get; }
}

TokenType (enum)

public enum TokenType
{
    BraceBlock,       // { }
    BracketBlock,     // [ ]
    ParenthesisBlock, // ( )
    Symbol,           // configurable characters
    Text,             // plain text
    Whitespace,       // spaces, tabs, newlines
    Error             // parsing errors
}

Requirements

• .NET 8.0 or later

License

MIT