CDTk 7.0.0

CDTk - Compiler Description Toolkit

CDTk (Compiler Description Toolkit) is a framework for defining compilers. It unifies lexing, parsing, and semantic analysis with regex tokens, PEG-style grammar rules, keyword enforcement, and flexible semantic mapping. This repository is implemented in C# (100% of the codebase).

Why CDTk?

Traditional compiler construction requires managing separate lexers, parsers, and code generators. CDTk unifies these components into a single, coherent workflow:

1. Define tokens — Describe what words your language recognizes
2. Define rules — Specify how those words combine into valid programs
3. Define mappings — Transform parsed programs into output code
4. Compile — One simple method call to process everything

Hello, Compiler! (New Fully-Typed API)

CDTk 7.0 introduces a fully-typed, sovereign API surface. All string-based APIs are deprecated. The new API uses type parameters for compile-time safety and better tooling support.

Here's a complete compiler using the new type-safe API:

using CDTk;

// Define token types (marker classes)
class Number { }
class Identifier { }
class Equals { }
class WS { }

// Define rule types
class Assignment { }

// Define AST node types
class AssignmentNode { }

// Define attribute types for dynamic code generation
class TypeAnnotation { }

// Define tokens with type parameters - NO string names!
var tokens = new TokenSet
{
    new Token<Number>(@"\d+"),
    new Token<Identifier>(@"[A-Za-z_][A-Za-z0-9_]*"),
    new Token<Equals>("="),
    new Token<WS>(@"\s+").Ignore()
};

// Define grammar rules with type parameters
var rules = new RuleSet
{
    new Rule<Assignment>("variable:@Identifier '=' value:@Number")
        .Returns<AssignmentNode>("variable", "value")  // Type-safe node creation
};

// Define code generation with type parameters
var target = new MapSet
{
    new Map<AssignmentNode>("let {variable}: {TypeAnnotation} = {value};")
        .Define<TypeAnnotation>(node => {
            // Dynamic attribute computation
            var value = node["value"] as string;
            return int.TryParse(value, out _) ? "number" : "any";
        })
};

// Build the compiler
var compiler = new Compiler()
    .WithTokens(tokens)
    .WithRules(rules)
    .WithTarget(target)  // Pass MapSet directly
    .Build();

// Compile code
var result = compiler.Compile("x = 42");

if (result.Success)
{
    Console.WriteLine(result.Output);  // Output: let x: number = 42;
}

Key benefits of the new API:

• Type safety: Token and rule names are types, not strings
• No duplicate names: The type system prevents naming conflicts
• Better IDE support: IntelliSense, go-to-definition, and refactoring work perfectly
• Sovereign API: No string-based lookups or magic strings
• Dynamic attributes: .Define<TAttr>() for computed template values

Multi-target example with the fully-typed API:

using CDTk;

// Define types (same as before)
class Number { }
class Identifier { }
class Equals { }
class WS { }
class Assignment { }
class AssignmentNode { }

// Tokens
var tokens = new TokenSet
{
    new Token<Number>(@"\d+"),
    new Token<Identifier>(@"[A-Za-z_][A-Za-z0-9_]*"),
    new Token<Equals>("="),
    new Token<WS>(@"\s+").Ignore()
};

// Rules
var rules = new RuleSet
{
    new Rule<Assignment>("name:@Identifier '=' value:@Number")
        .Returns<AssignmentNode>("name", "value")
};

// Two targets: JavaScript and Python
var js = new MapSet
{
    new Map<AssignmentNode>("let {name} = {value};")
};

var py = new MapSet
{
    new Map<AssignmentNode>("{name} = {value}")
};

// Build compiler with multi-target code generation
var compiler = new Compiler()
    .WithTokens(tokens)
    .WithRules(rules)
    .WithTargets(js, py)  // Pass MapSets directly
    .Build();

var result = compiler.Compile("x = 42");

if (result.Success)
{
    Console.WriteLine("JavaScript: " + result.Outputs[0]); // JS: let x = 42;
    Console.WriteLine("Python: " + result.Outputs[1]);     // Py: x = 42
}

How It Works

TokenSet → Define Your Vocabulary

Use Token<T> where T is the token type:

// Define token types
class Keyword { }
class Number { }
class WS { }

var tokens = new TokenSet
{
    new Token<Keyword>(@"if|while|return"),
    new Token<Number>(@"\d+"),
    new Token<WS>(@"\s+").Ignore()  // Whitespace is ignored
};

Use .Ignore() on tokens (such as whitespace or comments) that should not appear in the parse tree.

RuleSet → Define Your Grammar

Use Rule<T> where T is the rule type and .Returns<TNode>() for type-safe AST node specification:

// Define rule and node types
class Statement { }
class StatementNode { }
class Expression { }

var rules = new RuleSet
{
    new Rule<Statement>("keyword:@Keyword '(' expr:Expression ')'")
        .Returns<StatementNode>("keyword", "expr")  // Type-safe!
};

Rules reference tokens with @TokenName. You can label parts of a rule with label:@TokenOrRule, and those labels become named fields in the returned node.

MapSet → Generate Output

Use Map<T> where T is the AST node type. Use .Define<TAttr>() for dynamic attributes:

// Define node and attribute types
class StatementNode { }
class TypeInfo { }

var C = new MapSet
{
    new Map<StatementNode>("{keyword}({expr}); // {TypeInfo}")
        .Define<TypeInfo>(node => {
            // Compute type information dynamically
            return $"type_{node["expr"]}";
        })
};

var Ruby = new MapSet
{
    new Map<StatementNode>("{keyword}({expr})")
};

Map templates use {field} placeholders where field corresponds to the names returned by a rule's .Returns(...) declaration. Dynamic placeholders like {TypeInfo} are computed using .Define<TAttr>().

Naming recommendation: name your MapSet variable after the target language it generates for better code readability.

Compiler → One Entry Point

The Compiler orchestrates everything. Use builder-style methods to register tokens, rules, and either a single target (WithTarget(MapSet)) or multiple targets (WithTargets(MapSet, ...)), then call .Build() to produce a compiler instance.

Naming recommendation: name your compiler instance after your source language:

var Python = new Compiler()
    .WithTokens(tokens)
    .WithRules(rules)
    .WithTarget(C)  // Pass MapSet directly
    .Build();

var result = Python.Compile(sourceCode);

This reads naturally: "Python compiles to C"

Multi-Target Compilation

Compile to multiple targets in a single pass using the fully-typed API:

var Python = new Compiler()
    .WithTokens(tokens)
    .WithRules(rules)
    .WithTargets(C, Ruby, Wasm)  // Pass MapSets directly - variadic parameters
    .Build();

var result = Python.Compile(sourceCode);

// Access each output by index (order matches WithTargets parameter order)
Console.WriteLine(result.Outputs[0]);  // C output
Console.WriteLine(result.Outputs[1]);  // Ruby output
Console.WriteLine(result.Outputs[2]);  // Wasm output

The compiler parses input once and applies each MapSet to generate multiple outputs.

Getting Started

• Install — Install the CDTk NuGet package from https://www.nuget.org/packages/CDTk
• Learn — Read the complete guide in Learn.md
• Explore — Check out examples in Examples.md
• Ask — Common questions answered in FAQ.md

Documentation

• Learn.md — Complete guided walkthrough
• Examples.md — Sample compilers and languages
• FAQ.md — Common questions and solutions
• Contributing.md — How to report issues and contribute
• API Reference — See the code and XML docs in the repository for up-to-date API details

Quick Links

• Token Definition Guide
• Grammar Rules Guide
• Code Generation Guide
• Compiler API Reference
• Architecture Overview

License

See LICENSE.md for details.