FastTextProcess 1.0.2

FastTextProcess

Natural Language Processing feature extraction multilingual toolset. (in the development phase now)

• Axaprj.FastTextProcess Text to embedded vectors conversion high-performance framework (Facebook FastText engine and dictionaries based).
• Axaprj.Textc.Vect Natural Language features extraction framework (Takenet.Textc based)
• Axaprj.WordToVecDB DB access layer of embedded vectors processing tools.

code

VS 2017, .NET Core, SQLite, XUnit

toolchain and data (currently used)

• FastText pre-trained word vectors model extract into DataArcDir
• FastText pre-trained word vectors dictionary extract into DataArcDir
• Large Movie Review Dataset v1.0 extract into DataArcDir
• own FastText VS 2017 CMake compatibility fix
• own cnn-text-classification-tf preprocessed data loader
• own Takenet.Textc .Net Core port

start of usage

• setup FastTextProcess.Tests.json

{
  "DataArcDir": "c:/data.arc/",
  "FastTextBin": "c:/data.arc/fasttext.exe"
}

• Create pretrained vectors DB FastTextProcess.Tests.FastTextRoutines

ProcCreateDbEn()

• Append Train and Tests AclImdb Data FastTextProcess.Tests.FastTextRoutines

ProcAclImdbTrain()
ProcAclImdbTest()

• Looking for result into $DataArcDir/AclImdb_proc.db

advanced and experimental

FastTextRoutinesCyr - multilingual processing: languages detection and Ru, Uk, En texts processor. Aligned word vectors used.

reference

[1] T. Mikolov, E. Grave, P. Bojanowski, C. Puhrsch, A. Joulin. Advances in Pre-Training Distributed Word Representations @inproceedings{mikolov2018advances, title={Advances in Pre-Training Distributed Word Representations}, author={Mikolov, Tomas and Grave, Edouard and Bojanowski, Piotr and Puhrsch, Christian and Joulin, Armand}, booktitle={Proceedings of the International Conference on Language Resources and Evaluation (LREC 2018)}, year={2018}}

[2] A. Joulin, P. Bojanowski, T. Mikolov, H. Jegou, E. Grave, Loss in Translation: Learning Bilingual Word Mapping with a Retrieval Criterion @InProceedings{joulin2018loss, title={Loss in Translation: Learning Bilingual Word Mapping with a Retrieval Criterion}, author={Joulin, Armand and Bojanowski, Piotr and Mikolov, Tomas and J\'egou, Herv\'e and Grave, Edouard}, booktitle={Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing}, year={2018}}

[3] P. Bojanowski*, E. Grave*, A. Joulin, T. Mikolov, Enriching Word Vectors with Subword Information @article{bojanowski2017enriching, title={Enriching Word Vectors with Subword Information}, author={Bojanowski, Piotr and Grave, Edouard and Joulin, Armand and Mikolov, Tomas}, journal={Transactions of the Association for Computational Linguistics}, volume={5}, year={2017}, issn={2307-387X}, pages={135--146}}

author

Axaprj, Igor Alexeev for Property Indicators Lab