Text processing using natural language methods
Keywords:
natural language processing, machine learning, tokenization, artificial intelligence, neural networksAbstract
The paper examines text processing methods using natural language (NLP), which plays a key role in the modern world of information technology. The article covers basic NLP concepts and techniques such as tokenization, stemming, lemmatization, stop word removal, regular expressions, and text representation methods including Bag of Words and TF-IDF. Particular attention is paid to sentiment analysis, machine translation, automatic summarization and chatbots, which are important areas in the field of NLP.
References
Обработка естественного языка – [Электронный ресурс] / Интернет-ресурс. - Режим доступа: https://neerc.ifmo.ru/wiki/index.php?title=Обработка_естественного_языка - Загл. с экрана
Правильный NLP: как работают и что умеют системы обработки естественного языка – [Электронный ресурс] / Интернет-ресурс. - Режим доступа: https://tproger.ru/articles/natural-language-processing/ - Загл. с экрана
Jurafsky, D., & Martin, J. H.. Speech and Language Processing (3rd ed.). - Pearson, 2021.
Thomas Landauer, Peter W. Foltz, & Darrell Laham. Introduction to Latent Semantic Analysis (англ.) // Discourse Processes (англ.) русск.: journal. – 1998. – Vol. 25. – Pp. 259–284. – DOI: 10.1080/01638539809545028.
Pennington, J., Socher, R., & Manning, C. D. Glove: Global Vectors for Word Representation // Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). – РР. 1532-1543.
Mark Needham. Scikit-learn: TF/IDF and cosine similarity for computer science papers. – 2017. – [Электронный ресурс] / Интернет-ресурс. - Режим доступа: https://markhneedham.com/blog/2016/07/27/scitkit-learn-tfidf-and-cosine-similarity-for-computer-science-papers/ - Загл. с экрана
Pang, B., & Lee, L. Opinion Mining and Sentiment Analysis // Foundations and Trends in Information Retrieval, 2008. – Т. 2(1-2). – Р. 1-135.
Koehn, P. Statistical Machine Translation // Cambridge University Press, 2010.
