Abstract

In recent years, the applications of NLP have seen a boom with the emergence of Large Language Models. Computer Science, Artificial Intelligence, and Machine Learning are essentially powered by mathematics, and to make it practical, the real-world objects and concepts are quantified so the math can be applied to them practically. This works well in the cases where the data can be quantified without losing any information. The same cannot be said about the textual data as there are no direct and straight forward methods of converting text into numbers meaningfully. To make the language processing possible using mathematics techniques effectively, several studies have been carried out over time on the matter of text representation numerically. Some rely on using vectors containing word frequencies while some techniques go for an even simpler approach like one hot encoding. This manuscript is the extract of a comprehensive comparative study of various text representation techniques that have been in use for the majority of NLP history.