Normalising Macronic text into a uniform language

Abstract

SMS are short-length text documents written in a informal style. SMS text processing is challenging because of multi-varied text composition in terms of language, vocabulary, style and quality. In this project, with the help of RapidMiner software tool we have tried to standardize SMS texts. We have worked on American English messages only. With the help of a slang dictionary, we corrected most of the word. In order to improve the efficiency of the system, we created a database to perform next word prediction from. We performed bigram on our corrected dataset, retrieved the previous values to the error, and from our prediction dataset predicted what possible words could be used. Our system gives an accuracy of about 96% and can be further improved.