SentencePieceTokenizer 0.1.3
dotnet add package SentencePieceTokenizer --version 0.1.3
NuGet\Install-Package SentencePieceTokenizer -Version 0.1.3
This command is intended to be used within the Package Manager Console in Visual Studio, as it uses the NuGet module's version of Install-Package.
<PackageReference Include="SentencePieceTokenizer" Version="0.1.3" />
For projects that support PackageReference, copy this XML node into the project file to reference the package.
<PackageVersion Include="SentencePieceTokenizer" Version="0.1.3" />
<PackageReference Include="SentencePieceTokenizer" />
For projects that support Central Package Management (CPM), copy this XML node into the solution Directory.Packages.props file to version the package.
paket add SentencePieceTokenizer --version 0.1.3
The NuGet Team does not provide support for this client. Please contact its maintainers for support.
#r "nuget: SentencePieceTokenizer, 0.1.3"
#r directive can be used in F# Interactive and Polyglot Notebooks. Copy this into the interactive tool or source code of the script to reference the package.
#addin nuget:?package=SentencePieceTokenizer&version=0.1.3
#tool nuget:?package=SentencePieceTokenizer&version=0.1.3
The NuGet Team does not provide support for this client. Please contact its maintainers for support.
SentencePieceTokenizer
Usage
The tokenizers should be thread-safe, as the underlying sentencepiece processor is thread-safe. This has not been extensively tested however!
MarianTokenizer
Uses the tokenizer of a sentencepiece model, but a different vocabulary for the ids.
References
- Using SentencePiece v0.2.0 from 2024-02-19
- For BERT-style embeddings it is recommended to use FastBertTokenizer
- Inspired by SIL.Machine.Tokenization.SentencePiece
| Product | Versions Compatible and additional computed target framework versions. |
|---|---|
| .NET | net9.0 is compatible. net9.0-android was computed. net9.0-browser was computed. net9.0-ios was computed. net9.0-maccatalyst was computed. net9.0-macos was computed. net9.0-tvos was computed. net9.0-windows was computed. |
Compatible target framework(s)
Included target framework(s) (in package)
Learn more about Target Frameworks and .NET Standard.
-
net9.0
- protobuf-net (>= 3.2.45)
- System.Numerics.Tensors (>= 9.0.0)
NuGet packages (1)
Showing the top 1 NuGet packages that depend on SentencePieceTokenizer:
| Package | Downloads |
|---|---|
|
Darcara.TextAnalysis
TextAnalysis, sentence spliting, named entity recognition, translation and more |
GitHub repositories
This package is not used by any popular GitHub repositories.