siew24
-
- 189 total downloads
- last updated 11/24/2023
- Latest version: 0.4.6-beta (prerelease)
Fast and memory-efficient WordPiece tokenizer as it is used by BERT and others. Tokenizes text for further processing using NLP/language models. This fork only supports .NET Standard 2.1