Building SMS SPAM Detector and Generating a WordCloud with Kaggle Dataset in JupyterLab

Background problem

At least 97% of American use text messages over mobile phones every day. In 2016, according to the research conducted by Portio research, 8.3 trillion messages exchanged over the mobile phones. The rising flood of big data shows an exchange of 23 billion messages per day and 16 million messages per minute. There are around 6.4 billion mobile subscribers around the world by the end of 2012. According to Portio Research, there will be a CAGR growth of 4.8% of growth in mobile subscriber base from 2014 to 2017. By the end of 2017, the mobile subscriber reached to 7.4 billion mobile subscribers. The proliferation of smart devices powered by exponential computing has shown a significant rise in the global smartphone system-on-chip market lead by Qualcomm, Apple, MediaTrek, Samsung, HiSilicon, Spreadtrum, and a vast number of other smartphone chip manufacturers in the market. Powering the chips with artificial intelligence technology paves the path to 5G for higher performance and signal processing. Regardless of the multi-functional and advanced capabilities of smartphones, simple text messaging continued to soar in the worldwide markets. The exponential growth of computing processing power gave rise to generating such massive big data over the text messages. The timeline of mobile handset industry from 1983 (when the first mobile handset launched) to 2002 (first mobile phone with touchscreen) shows the significant increase in the computing and SoC (System-on-Chip) architecture for such tsunami of big data over the SMS messages. The first SMS communication service launched in 1992. 3G mobile services launched in 2002. In 2010, 4G networks launched. The speed of the delivery gave rise to the communication increase through SMS messages for businesses and individuals to manage a significant part of their lives. The mobile messaging service industry generated revenues of $212b in 2012 alone.

Figure 1. Adapted from Portio Research.

According to Portio Research, SMS traffic jumped to 100 billion messages from just 0.5 billion messages between 1996 and 1999. By the end of 2003, in another four years, SMS traffic quadrupled to 450 billion messages. In 2005, the SMS traffic reached over to a trillion messages mark only in two years between 2003 to 2005. By 2009, the world has seen traffic of five trillion messages. In 2015, the traffic peaked to 8.3 trillion messages. The SMS text traffic went by leaps and bounds with application-to-peer messaging and person-to-application SMS messaging with banking, mobile health, and mobile payments sector. This gave room to abundant SPAM from many telemarketers sending SMS texts. Nowadays, many recruiting agencies send SMS with job positions with subscription and sometimes without a subscription. I receive heavy SMS messages from recruiting agencies without subscription. This has been coordinated by some of the people who hacked my account on Facebook and Twitter. I still have those messages with me.

According to two Forrester Research publications Forrester Research Mobile Media Application Spending Forecast 2012–2017 EU-7 and Forrester Research Mobile Media Application Spending Forecast 2012–2017 US six billion SMS messages sent in the US alone every day. Majority of texts, i.e., 80% of the texts generated from American adults. The rise of the SPAM can be attributed to the SMS success open rate of 45% as opposed to 20% success open rate through emails. The response rate goes higher for text messages with 45% and 6% for emails. Americans exchange twice as many text messages as phone calls.

Naïve Bayes classifier

Considering exponential growth in big data and SMS traffic, there’s significant growth in SMS spam as a medium to commit fraud and advertise their job opportunities. The spam filtering can be applied through Naïve Bayes classifier by classifying SMS whether SPAM or HAM. In essence, Naïve Bayes classifier can work as anti-spam software with higher accuracy rates. In this Python implementation, it has shown an accuracy rate of 99.38% training and 98.15% of accuracy rate of testing. Kaggle dataset has been utilized to perform the SPAM detection through Naïve Bayes classifier. Kaggle dataset file has two columns with the label v1 and v2. V1 contains label either spam or ham text data, while the v2 column contains the actual SMS message. Approximately, in US users receive 1.1 billion SPAM SMS messages and Chinese mobile users receive 8.29 billion SMS spams every week by various advertising media and fraudulent corporations. Many classifiers can be applied to filter the SMS SPAM problem such as rule induction, neural networks, decision trees, Naïve Bayes, k-nearest neighbors, and support vector machines. One has to consider the fact that, classifying email is entirely different from classifying SMS text, as the length of the text is limited to 160 characters. Therefore, the featurization has to be adequate to identify between ham and spam. Historically, Naïve Bayes classification algorithm has proven to be highly effective in identifying SPAM.

Figure 2. Kaggle SPAM dataset.

Figure 3.Adapted from The 2016 IEEE Asia Pacific Conference on Wireless and Mobile (APWiMob) Research Paper

TF-IDF Vectorizer vs. Common vectorizer strategies

As with other problems, the process involves at first loading the dataset by reading in Python with ISO-8859–1 encoding and applying Naïve Bayes machine learning algorithm by training and testing stages of building a machine learning model. Any irrelevant column names in the file need to be dropped. The feature extraction can be performed either by count vectorizer or through TF-IDF vectorizer. The countvectorizer applies tokenization and occurrence counting through a single class. By applying the common vectorizer, the words can be tokenized through natural language processing and count the word occurrences through a minimalistic corpus of text files or documents. Alternatively, TF-IDF vectorizer can be applied as well as in the case of large text corpus; there will be the repetitive occurrence of words such as the, a, or is in the English language. The TFIDFTransformer and TFIDVectorizer in scikit learn will perform the count of the word occurrences.

Data Visualization

Generating a word cloud through wordcloud library shows the most frequently repeating SPAM words such as call, free, now, UK, ringtone, customer service, chat, landline, text, etc. with a combination of blue and green. The wordcloud generated from the program attached below:

Figure 4. Python program output from Jupyter console for SPAM text.

The data visualization for ham shows the following word cloud from the program.Figure 5. Python program output from Jupyter console for HAM text.

Results

I have shared the program on Github at GPSingularity.

Figure 6. Jupyter Lab Python Notebook .

References

Arifin, D. D., Shaufiah, & Bijaksana, M. A. (2017, Janury 2017). Enhancing spam detection on mobile phone Short Message Service (SMS) performance using FP-growth and Naive Bayes Classifier. IEEE Explore Wireless and Mobile (APWiMob), 2016 IEEE Asia Pacific Conference. http://dx.doi.org/10.1109/APWiMob.2016.7811442

O’Grady, M. (2012). SMS Usage Remains Strong In The US: 6 Billion SMS Messages Are Sent Each Day. Retrieved May 13, 2015, from https://go.forrester.com/blogs/12-06-19-sms_usage_remains_strong_in_the_us_6_billion_sms_messages_are_sent_each_day/

Portio Research (2017). WorldWide SMS Markets 2014–2017. Retrieved May 13, 2018, from http://www.xconnect.net/wp-content/uploads/worldwide-sms-markets-portio-strikeiron.pdf

Smith, A. (2015). U.S. Smartphone Use in 2015. Retrieved May 13, 2018, from http://www.pewinternet.org/2015/04/01/us-smartphone-use-in-2015/

Srivastava, S. (2017). Global Smartphone SoC Market Crossed US$8 Billion in Q3 2017, A Third Quarter Record. Retrieved May 13, 2018, from https://www.counterpointresearch.com/global-smartphone-soc-market-crossed-us8-billion-q3-2017-third-quarter-record/

28 thoughts on “Building SMS SPAM Detector and Generating a WordCloud with Kaggle Dataset in JupyterLab

  1. adidas ultra boost 3.0 Posted On

    I intended to put you a little note so as to say thanks a lot over again for your personal magnificent opinions you have discussed here. This is surprisingly open-handed with people like you to convey freely all most people would’ve sold as an e-book to earn some bucks on their own, notably given that you could have done it if you ever decided. These tactics as well acted to be a easy way to know that other people online have the identical desire just like my personal own to grasp much more in terms of this matter. Certainly there are a lot more enjoyable times up front for folks who go through your blog.

    Reply
  2. yeezy shoes Posted On

    I together with my pals ended up checking out the good guidelines on the website while before long I got a horrible suspicion I never thanked the web blog owner for those techniques. Most of the people were as a result stimulated to read them and now have honestly been taking pleasure in those things. Thanks for being well considerate as well as for utilizing this kind of terrific guides millions of individuals are really needing to understand about. Our sincere regret for not expressing gratitude to you earlier.

    Reply
  3. adidas yeezy boost Posted On

    Thank you a lot for providing individuals with an extraordinarily spectacular chance to read in detail from this website. It is often so enjoyable and also packed with a lot of fun for me personally and my office friends to search your web site at least thrice in one week to read through the latest tips you will have. And indeed, I am at all times contented with your astounding solutions you serve. Certain 2 ideas on this page are unquestionably the finest we’ve ever had.

    Reply
  4. air max Posted On

    I’m also writing to make you know of the incredible discovery our girl undergone checking yuor web blog. She noticed such a lot of pieces, not to mention what it’s like to possess a wonderful coaching mood to let many others without hassle gain knowledge of specified grueling things. You really surpassed our own expected results. Thanks for displaying these good, trustworthy, explanatory and even easy tips about the topic to Janet.

    Reply
  5. golden goose outlet Posted On

    I am only commenting to let you understand what a useful experience my cousin’s princess obtained reading your site. She came to understand lots of issues, which include what it is like to possess a wonderful teaching nature to have men and women really easily know just exactly various complicated subject matter. You actually did more than our own expected results. Thank you for coming up with those beneficial, healthy, explanatory and fun tips on the topic to Tanya.

    Reply
  6. hermes belt Posted On

    I precisely desired to appreciate you again. I am not sure the things that I could possibly have sorted out without the type of tips and hints shared by you over this area. It was before a very horrifying problem in my opinion, however , finding out the very skilled way you resolved the issue took me to weep over fulfillment. Extremely happier for your help and in addition believe you comprehend what an amazing job you were getting into teaching the others using your website. Most likely you have never encountered any of us.

    Reply
  7. send bulk sms india Posted On

    I simply send bulk sms india would not go away your internet site prior to advising i actually loved the common data somebody supply as part of your readers? Will likely be yet again routinely to check completely new discussions

    Reply
  8. michael kors purses Posted On

    There are actually numerous details like that to take into consideration. That may be a great point to bring up. I supply the thoughts above as general inspiration however clearly there are questions like the one you deliver up the place the most important thing might be working in sincere good faith. I don?t know if best practices have emerged around things like that, but I am sure that your job is clearly recognized as a fair game. Both girls and boys feel the affect of only a moment抯 pleasure, for the rest of their lives.

    Reply
  9. air jordan 13 Posted On

    Aw, this was a very nice post. In idea I would like to put in writing like this moreover ?taking time and precise effort to make a very good article?but what can I say?I procrastinate alot and certainly not seem to get one thing done.

    Reply
  10. michael kors outlet Posted On

    Can I simply say what a aid to search out someone who truly knows what theyre talking about on the internet. You definitely know the best way to carry a difficulty to mild and make it important. More folks must read this and understand this facet of the story. I cant imagine youre not more in style since you positively have the gift.

    Reply
  11. yeezy shoes Posted On

    My husband and i felt now fulfilled Albert could carry out his researching with the precious recommendations he acquired while using the web pages. It is now and again perplexing just to happen to be freely giving strategies which other people have been selling. And now we understand we have got the blog owner to give thanks to for that. The most important explanations you have made, the easy web site menu, the relationships you can help create – it is all powerful, and it is leading our son and us consider that this concept is amusing, which is certainly pretty serious. Many thanks for all!

    Reply
  12. supreme clothing Posted On

    I抦 impressed, I must say. Actually not often do I encounter a weblog that抯 both educative and entertaining, and let me inform you, you’ve hit the nail on the head. Your idea is excellent; the issue is one thing that not enough individuals are speaking intelligently about. I am very pleased that I stumbled across this in my search for one thing referring to this.

    Reply
  13. Adidas NMD New Geometry Purple Grey Posted On

    My wife and i have been very delighted Albert managed to finish up his researching by way of the precious recommendations he got out of the site. It is now and again perplexing to simply find yourself giving freely information that people today could have been making money from. We acknowledge we now have the blog owner to give thanks to for this. The main illustrations you have made, the straightforward website menu, the friendships you help instill – it’s all unbelievable, and it’s really assisting our son and the family consider that the matter is exciting, which is certainly exceptionally fundamental. Many thanks for all!

    Reply
  14. nike roshe run Posted On

    I found your weblog site on google and test a number of of your early posts. Proceed to maintain up the very good operate. I simply further up your RSS feed to my MSN Information Reader. In search of forward to studying extra from you later on!?

    Reply
  15. cheap oakleys Posted On

    There are some interesting cut-off dates in this article however I don抰 know if I see all of them middle to heart. There is some validity but I’ll take maintain opinion until I look into it further. Good article , thanks and we want more! Added to FeedBurner as properly

    Reply
  16. adidas ultra boost Posted On

    My wife and i felt cheerful that Michael managed to complete his preliminary research via the precious recommendations he came across while using the web pages. It’s not at all simplistic to just continually be releasing tips and hints which often people today might have been trying to sell. And we do understand we’ve got the website owner to appreciate for that. The entire explanations you made, the easy blog menu, the friendships you can give support to foster – it’s everything great, and it is leading our son and the family believe that this concept is awesome, which is unbelievably mandatory. Thanks for all!

    Reply
  17. nmd Posted On

    I抎 have to check with you here. Which isn’t one thing I often do! I take pleasure in studying a post that can make individuals think. Also, thanks for allowing me to remark!

    Reply
  18. tory burch outlet Posted On

    There are certainly plenty of details like that to take into consideration. That could be a nice level to carry up. I supply the ideas above as basic inspiration but clearly there are questions just like the one you deliver up the place a very powerful thing will probably be working in trustworthy good faith. I don?t know if finest practices have emerged round issues like that, however I’m sure that your job is clearly recognized as a good game. Both boys and girls really feel the impact of just a moment抯 pleasure, for the remainder of their lives.

    Reply
  19. yeezy boost 350 v2 Posted On

    I found your weblog website on google and test a few of your early posts. Continue to maintain up the excellent operate. I just additional up your RSS feed to my MSN Information Reader. In search of forward to reading more from you later on!?

    Reply
  20. yeezy boost 350 Posted On

    My husband and i were so more than happy Edward managed to complete his basic research by way of the ideas he discovered in your web site. It is now and again perplexing to simply possibly be handing out tricks which people today may have been making money from. So we fully grasp we need you to appreciate for this. The most important explanations you have made, the straightforward website menu, the friendships you help to foster – it is most awesome, and it’s really assisting our son and us reason why that subject matter is pleasurable, and that’s particularly essential. Thanks for all the pieces!

    Reply
  21. cheap basketball shoes Posted On

    I抦 impressed, I need to say. Actually rarely do I encounter a weblog that抯 each educative and entertaining, and let me tell you, you have hit the nail on the head. Your concept is excellent; the issue is something that not enough individuals are talking intelligently about. I am very comfortable that I stumbled throughout this in my seek for something regarding this.

    Reply
  22. adidas nmd Posted On

    I wanted to send you a very little remark to be able to thank you yet again for the nice opinions you have provided in this case. This is really open-handed of people like you to present easily what exactly a number of people could have supplied for an ebook to get some cash for themselves, and in particular considering the fact that you might well have tried it if you ever decided. Those good tips additionally worked to become easy way to realize that someone else have the identical keenness the same as my own to figure out much more when considering this matter. I believe there are some more pleasurable sessions ahead for folks who start reading your website.

    Reply
  23. michael kors outlet online Posted On

    Oh my goodness! an amazing article dude. Thanks However I’m experiencing issue with ur rss . Don抰 know why Unable to subscribe to it. Is there anybody getting equivalent rss downside? Anybody who knows kindly respond. Thnkx

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *