Tokensharp 4.5.3

.NET 8.0 This package targets .NET 8.0. The package is compatible with this framework or higher. 
dotnet add package Tokensharp --version 4.5.3
                    


                    

                
NuGet\Install-Package Tokensharp -Version 4.5.3
This command is intended to be used within the Package Manager Console in Visual Studio, as it uses the NuGet module's version of Install-Package. 
<PackageReference Include="Tokensharp" Version="4.5.3" />
For projects that support PackageReference, copy this XML node into the project file to reference the package. 
<PackageVersion Include="Tokensharp" Version="4.5.3" />
                    


                            Directory.Packages.props
                        

                    

                
<PackageReference Include="Tokensharp" />
                    


                            Project file
For projects that support Central Package Management (CPM), copy this XML node into the solution Directory.Packages.props file to version the package. 
paket add Tokensharp --version 4.5.3
                    


                    

                
#r "nuget: Tokensharp, 4.5.3"
#r directive can be used in F# Interactive and Polyglot Notebooks. Copy this into the interactive tool or source code of the script to reference the package. 
#:package Tokensharp@4.5.3
#:package directive can be used in C# file-based apps starting in .NET 10 preview 4. Copy this into a .cs file before any lines of code to reference the package. 
#addin nuget:?package=Tokensharp&version=4.5.3
                    


                            Install as a Cake Addin
                        

                    

                
#tool nuget:?package=Tokensharp&version=4.5.3
                    


                            Install as a Cake Tool

Tokensharp

NuGet

Tokensharp is a high-performance, lightweight tokenization library for .NET. It provides a flexible and efficient way to break text into tokens based on a defined set of token types.

Features

  • High Performance: Optimized for speed and low memory allocation using ReadOnlySpan<char> and ReadOnlyMemory<char>.
  • Streaming Support: Easily tokenize large files or network streams using IAsyncEnumerable.
  • Extensible: Define your own token types and lexemes.
  • Source Generation: Works seamlessly with Tokensharp.TokenTypeGenerator to automatically generate token type classes from JSON definitions.

Installation

Install the package via NuGet:

dotnet add package Tokensharp

Quick Start

  1. Define your token types (manually or using the generator):
public record MyTokenType(string Lexeme) : TokenType<MyTokenType>(Lexeme), ITokenType<MyTokenType>
{
    public static readonly MyTokenType Var = new("var");
    public static readonly MyTokenType Assign = new("=");
    public static readonly MyTokenType EndStatement = new(";");
    
    public static MyTokenType Create(string lexeme) => new(lexeme);
    public static TokenConfiguration<MyTokenType> Configuration => new TokenConfigurationBuilder<MyTokenType>
    {
        Var,
        Assign,
        EndStatement
    }.Build();
}
  1. Use the Tokenizer to parse tokens:
string input = "var x = 42;";
foreach (var token in Tokenizer.EnumerateTokens<MyTokenType>(input))
{
    Console.WriteLine($"Token: {token.Type}, Lexeme: {token.Lexeme}");
}
Product Compatible and additional computed target framework versions.
.NET net8.0 is compatible.  net8.0-android was computed.  net8.0-browser was computed.  net8.0-ios was computed.  net8.0-maccatalyst was computed.  net8.0-macos was computed.  net8.0-tvos was computed.  net8.0-windows was computed.  net9.0 is compatible.  net9.0-android was computed.  net9.0-browser was computed.  net9.0-ios was computed.  net9.0-maccatalyst was computed.  net9.0-macos was computed.  net9.0-tvos was computed.  net9.0-windows was computed.  net10.0 is compatible.  net10.0-android was computed.  net10.0-browser was computed.  net10.0-ios was computed.  net10.0-maccatalyst was computed.  net10.0-macos was computed.  net10.0-tvos was computed.  net10.0-windows was computed. 
Compatible target framework(s)
Included target framework(s) (in package)
Learn more about Target Frameworks and .NET Standard.

  • net10.0

    • No dependencies.

  • net8.0

    • No dependencies.

  • net9.0

    • No dependencies.

NuGet packages

This package is not used by any NuGet packages.

GitHub repositories

This package is not used by any popular GitHub repositories.

Version Downloads Last Updated
4.5.3 91 1/27/2026
4.5.2 92 1/27/2026
4.5.1 91 1/26/2026
4.5.0 99 1/26/2026
4.4.0 92 1/23/2026
4.3.0 86 1/23/2026
4.2.0 91 1/23/2026
4.1.0 88 1/23/2026
4.0.1 91 1/22/2026
4.0.0 87 1/20/2026
3.1.2 90 1/14/2026
3.1.1 91 1/13/2026
3.1.0 91 1/13/2026
3.0.3 93 1/12/2026
3.0.2 90 1/12/2026
3.0.1 96 1/12/2026
3.0.0 93 1/12/2026
2.0.0 95 1/10/2026
1.3.3 94 1/8/2026
1.3.2 90 1/8/2026
Loading failed