CDTk 8.0.0

CDTk – The Compiler Description Toolkit

CDTk is the modern .NET library for building high-performance compilers, transpilers, and code generators. It makes compiler creation intuitive, combining tokenization, grammar rules, semantic models, and output generation into a unified framework.

This tutorial provides a concise overview of how to define the essential components—TokenSet, RuleSet, and MapSet—and explains their capabilities.

How CDTk Organizes Compiler Design

A CDTk compiler is defined as a streamlined pipeline:

1. TokenSet: Defines lexical patterns (tokens).
2. RuleSet: Maps tokens into higher-level grammar constructs.
3. MapSet: Transforms parsed data into your desired output.

1. TokenSet

The TokenSet class is used to define the tokens that represent the building blocks of your language, such as numbers, operators, and keywords.

Example: Defining Tokens

class Tokens : TokenSet
{
    public Token Number = @"\d+";            // Defines numbers
    public Token Plus = @"\+";               // Defines the '+' operator
    public Token Whitespace = new Token(@"\s+").Ignore(); // Ignores whitespace
}

Capabilities of TokenSet

• Ignored Tokens: Tokens like Whitespace or comments can be ignored using .Ignore().

public Token Comment = new Token(@"//.*").Ignore();

• Token Ordering: More specific patterns (e.g., keywords) should appear before generic patterns (e.g., identifiers).

public Token Keyword = @"if|else";
public Token Identifier = @"[A-Za-z_][A-Za-z0-9_]*";

2. RuleSet

The RuleSet class is used to specify the grammar of your language. Rules describe how tokens combine to form expressions, statements, or other constructs.

Example: Defining Rules

class Rules : RuleSet
{
    public Rule Expression = new Rule("left:@Number '+' right:@Number")
        .Returns("left", "right");
}

Capabilities of RuleSet

• Nested Rules: Rules can reference other rules.

public Rule Factor = new Rule("@Number | '(' inner:Expression ')'");

• Optional Components: Use ? to define optional elements.

public Rule VariableDeclaration = new Rule("@Identifier ('=' Expression)?;");

• Repetitions: Use * for zero or more repetitions, and + for one or more.

public Rule ArgumentList = new Rule("@Identifier (',' @Identifier)*");

• Validation: Attach semantic checks to ensure constraints are satisfied.

public Rule Assignment = new Rule("variable:@Identifier '=' value:@Number")
    .Validate((ctx, node, token) =>
    {
        if (!ctx.IsDeclared(node.GetString("variable")))
            ctx.Report(DiagnosticLevel.Error, "Variable not declared");
    });

3. MapSet

The MapSet class is used to transform the Abstract Syntax Tree (AST) into target output, such as code, JSON, or other formats.

Example: Defining Mappings

class Maps : MapSet
{
    public Map Expression = "{left} {op} {right}";
}

Capabilities of MapSet

• Simple Placeholders: Use {fieldName} to reference AST fields in templates.

public Map Assignment = "{variable} = {value}";

• Multi-Target Support: Define multiple mapping sets for different languages or platforms.

class JavaScriptMap : MapSet
{
    public Map Expression = "let result = {left} {op} {right};";
}

class PythonMap : MapSet
{
    public Map Expression = "result = {left} {op} {right}";
}

• Fallback Templates: Define a default mapping for unmapped nodes.

public Map Fallback = "// TODO: Unsupported node type {type}";

Bringing It Together

Once the TokenSet, RuleSet, and MapSet are defined, you can build the compilation pipeline:

var compiler = new Compiler()
    .WithTokens(new Tokens())
    .WithRules(new Rules())
    .WithTarget(new Maps())
    .Build();

var result = compiler.Compile("3 + 5");
Console.WriteLine(result.Output);  // Outputs: "3 + 5"

Key Takeaways

• TokenSet organizes the lexical structure by defining patterns like numbers, operators, and keywords.
• RuleSet defines how tokens combine into higher-level constructs, supporting repetition, nesting, and validation.
• MapSet transforms parsed structures into the desired output (e.g., code, configurations).

Explore More

CDTk is a powerful framework built for developers who value clarity and efficiency in compiler design. Learn more:

• CDTk Wiki
• Examples
• Contributing to CDTk