Darcara.TextAnalysis 0.1.2

Sentence splitting, named entity recognition, translation and more

• Sentence splitting with SaT / WtP
  • Configuration
  • Evaluation
  • Implementation notes
• Language Identification
• Compiling onnx runtime on Windows
  • Prerequisites

Sentence splitting with SaT / WtP

Segment Any Text (June 2024) is the successor to Where's the Point (July 2023). The code from both papers is available on GitHub.
SaT supports 85 languages. The detailed list is available in their GitHub readme.
Models for SaT come in 3 flavors:

• Base models with 1, 3, 6, 9 or 12 layers available on HuggingFace
  More layers means higher accuracy, but longer inference time
• Low-Rank Adaptation (LoRA) modules are available for 3 and 12 layer base models in their respective repositories
  The LoRA modules enable the base models to be adapted to specific domains and styles
• Supervised Mixture (sm) models with 1, 3, 6, 9, 12 layers available on HuggingFace
  SM models have been trained with a "supervised mixture" of diverse styles and corruptions. They score higher both on english and multilingual text.

This project supports the *-sm model family in onnx format.

Configuration

The SaT-Models benefit greatly from the GPU. For running on GPU setting the SessionConfiguration.Batching to batch=4 is best. For running on CPU setting the SessionConfiguration.Batching to batch=1 with IterOperationThreads=1 and IntraOperationThreads=2 will . Higher values for IntraOperationThreads will slightly decrease computing time, but use a lot more processing power. It is preferable to sentencize multiple text in parallel.

A consuming project must nce a proper ONNX-runtime. For Windows deployments Microsoft.ML.OnnxRuntime.DirectML with Microsoft.AI.DirectML will yield the best performance. Setting the RuntimeIdentifier in the project csproj to win-x64 is required.

Evaluation

The corpora scores are from the original SaT Github
This benchmark used the novel "The Adventures of Tom Sawyer, Complete by Mark Twain" from Project Gutenberg
The -model columns give the speed of only the model runtime, whereas -complete includes all pre and post data preparations, including the word tokenization.

¹ From the original SaT-Github

Implementation notes

Word tokenization is done by sentencepiece using the xlm-roberta-base (Alt1, Alt2) model. It is used in C# with the help of the SentencePieceTokenizer library.

See:

• https://www.kaggle.com/code/samuellongenbach/xlm-roberta-tokenizers-issue/notebook
• https://github.com/google/sentencepiece/issues/1042#issuecomment-2295028056
• Seems to have no resolution, other than re-writing the model, since I don't know how to "modify the indexing scheme to start from 1" ?

Language Identification

With this library you can identify languages with ILanguageDetector detector = new LinguaLanguageDetector(useLowAccuracy: false);

The LinguaLanguageDetector is based on Panlingo.Lingua, but optimized to use less memory and provide additional detection methods. The overloads for ReadOnlySpan<Char> (instead of String.SubString) reduce memory traffic and overloads for ReadOnlySpan<Byte> means you can use already utf8-ized strings directly.

The following table contains the evaluated libraries:

¹ Reliability is the sum of accurate predictions and the 'unknown' predictions of a library. It is usually better to know that the library can not ascertain the language properly instead of giving out a random wrong language.
² Average for 1000 word, bi-words and sentence predictions for each language. Actual timings will depend on your CPU.
³ Approximate memory requirements for one predictor. Includes native memory for libraries, rules and ML-models.

Compiling onnx runtime on Windows

Prerequisites

Reference https://onnxruntime.ai/docs/build/inferencing.html

• Python 3.12
• CMake
• Visual Studio 2022 (with MSVC v143 C++ x64/x86 BuildTools(v14.41-17.11))
• Make sure the build folder is empty or missing before starting

git clone https://github.com/microsoft/onnxruntime
-- or --
git fetch
git checkout v1.20.1

onnxruntime> PATH=%PATH%;C:\Program Files\Python312
onnxruntime> build.bat --cmake_path "C:\Program Files\CMake\bin\cmake.exe" --ctest_path "C:\Program Files\CMake\bin\ctest.exe" --config Release --build_shared_lib --parallel --compile_no_warning_as_error --skip_tests --use_mimalloc --use_dml

Currently with problems:
--build_nuget and --use_extensions

The result will be in build\Windows\Release\Release