CDTk 6.5.0

CDTk - Compiler Description Toolkit

CDTk (Compiler Description Toolkit) is a framework for defining compilers. It unifies lexing, parsing, and semantic analysis with regex tokens, PEG-style grammar rules, keyword enforcement, and flexible semantic mapping. This repository is implemented in C# (100% of the codebase).

Why CDTk?

Traditional compiler construction requires managing separate lexers, parsers, and code generators. CDTk unifies these components into a single, coherent workflow:

1. Define tokens — Describe what words your language recognizes
2. Define rules — Specify how those words combine into valid programs
3. Define mappings — Transform parsed programs into output code
4. Compile — One simple method call to process everything

Hello, Compiler!

Here's a complete compiler in just a few lines using the current public API. It demonstrates token definitions, a labeled grammar rule, and both single-target and multi-target code generation.

Single-target example:

using CDTk;

// Define tokens
var tokens = new TokenSet
{
    new Token("Number", @"\d+"),
    new Token("Identifier", @"\w+"),
    new Token("Equals", "="),
    new Token("WS", @"\s+").Ignore()
};

// Define grammar rules (labels in the rule become fields in the returned node)
var rules = new RuleSet
{
    new Rule("Assignment", "variable:@Identifier '=' value:@Number")
        .Returns("AssignmentNode", "variable", "value")
};

// Define code generation
var mapping = new MapSet
{
    new Map("AssignmentNode", "let {variable} = {value};")
};

// Build the compiler (single target)
var compiler = new Compiler()
    .WithTokens(tokens)
    .WithRules(rules)
    .WithTarget(mapping)   // single-target builder
    .Build();

// Compile code
var result = compiler.Compile("x = 42");

if (result.Success)
{
    Console.WriteLine(result.Output);  // Output: let x = 42;
}

Multi-target example (current API supports named targets and multi-target compilation in one pass):

using CDTk;

// Lexer: identifiers, numbers, '=', and ignored whitespace
var tokens = new TokenSet {
    new Token("Number",     @"\d+"),
    new Token("Identifier", @"\w+"),
    new Token("Equals",     "="),
    new Token("WS",         @"\s+").Ignore()
};

// Grammar: a single assignment statement using labeled parts
var rules = new RuleSet {
    new Rule("Assignment", "name:@Identifier '=' value:@Number")
        .Returns("AssignmentNode", "name", "value")
};

// Two targets: JavaScript-like and Python-like output
var js = new MapSet {
    new Map("AssignmentNode", "let {name} = {value};")
};
var py = new MapSet {
    new Map("AssignmentNode", "{name} = {value}")
};

// Build compiler with multi-target code generation
var compiler = new Compiler()
    .WithTokens(tokens)
    .WithRules(rules)
    .WithTargets(("JS", js), ("Py", py)) // multi-target builder (named targets)
    .Build();

var result = compiler.Compile("x = 42"); // Input

// Access generated outputs per target
Console.WriteLine("JavaScript: " + result.Outputs["JS"]); // JS Output: let x = 42;
Console.WriteLine("Python: " + result.Outputs["Py"]);    // PY Output: x = 42

How It Works

TokenSet → Define Your Vocabulary

TokenSet defines what text patterns your language recognizes. Each Token has a name and a regex pattern:

var tokens = new TokenSet
{
    new Token("Keyword", @"if|while|return"),
    new Token("Number", @"\d+"),
    new Token("WS", @"\s+").Ignore()  // Whitespace is ignored
};

Use .Ignore() on tokens (such as whitespace or comments) that should not appear in the parse tree.

RuleSet → Define Your Grammar

RuleSet specifies how tokens combine to form valid programs. Rules reference tokens with @TokenName. You can label parts of a rule with label:@TokenOrRule, and those labels become named fields in the returned node.

var rules = new RuleSet
{
    new Rule("Statement", "keyword:@Keyword '(' expr:Expression ')'")
        .Returns("StatementNode", "keyword", "expr")
};

.Returns(nodeName, field1, field2, ...) declares the named node type and the field names the rule produces.

MapSet → Generate Output

MapSet transforms your parsed program into the target language. Map templates use {field} placeholders where field corresponds to the names returned by a rule's .Returns(...) declaration.

Naming recommendation: name your MapSet variable after the target language it generates:

var C = new MapSet
{
    new Map("StatementNode", "{keyword}({expr});")
};

var Ruby = new MapSet
{
    new Map("StatementNode", "{keyword}({expr})")
};

This naming convention makes your code more readable and enables natural multi-target compilation.

Compiler → One Entry Point

The Compiler orchestrates everything. Use builder-style methods to register tokens, rules, and either a single target (WithTarget) or multiple named targets (WithTargets), then call .Build() to produce a compiler instance.

Naming recommendation: name your compiler instance after your source language:

var Python = new Compiler()
    .WithTokens(tokens)
    .WithRules(rules)
    .WithTarget(C)
    .Build();

var result = Python.Compile(sourceCode);

This reads naturally: "Python compiles to C"

Multi-Target Compilation

Compile to multiple targets in a single pass:

var Python = new Compiler()
    .WithTokens(tokens)
    .WithRules(rules)
    .WithTargets(("C", C), ("Ruby", Ruby), ("Wasm", Wasm))
    .Build();

var result = Python.Compile(sourceCode);

// Access each output
Console.WriteLine(result.Outputs["C"]);
Console.WriteLine(result.Outputs["Ruby"]);
Console.WriteLine(result.Outputs["Wasm"]);

The compiler parses input once and applies each MapSet to generate multiple outputs.

Getting Started

• Install — Add CDTk to your .NET project (NuGet packaging details may be in Packaging.md or in the project build files)
• Learn — Read the complete guide in Learn.md
• Explore — Check out examples in Examples.md
• Ask — Common questions answered in FAQ.md

Documentation

• Learn.md — Complete guided walkthrough
• Examples.md — Sample compilers and languages
• FAQ.md — Common questions and solutions
• Contributing.md — How to report issues and contribute
• API Reference — See the code and XML docs in the repository for up-to-date API details

Quick Links

• Token Definition Guide
• Grammar Rules Guide
• Code Generation Guide
• Compiler API Reference
• Architecture Overview

License

See LICENSE.md for details.