Skip to main content

What it takes to create a news classification app?


Fooyo recently received a request to make a news classification app. It sounds quite challenging and interesting. We feel excited to solve hard problems. The request is a bit too urgent, but we managed to deliver it with good feedbacks from the client.

The requests is to crawl news data from various news sources, delete the duplications, classify the news into different categories and then display them on a mobile app.

It sounds quite straightforward, however, each module actually involves quite hard components that need deep knowledge/understanding to solve them.

In summary there are four components:
1. A web news crawler
2. A duplication deletion mechanism
3. A news classification mechanism
4. A mobile app which interacts with the server

A web crawler is used to crawl news list data and the corresponding page links. The web contents of the page links are then saved into the server. There are quite a few good open sourced crawler out there, e.g., scrapy.  Some sites may have anti-crawler feature which will block IPs when they crawl too much. Thus we may need different proxy servers to crawl the data when some are blocked by the news site.

A duplication deletion mechanism firstly need a good text parser which involves the following tasks:

1. Texts need to be tokenised into list of words(This step would be important for languages like Chinese).
2. Each word needs to be be stemmed(say ran, running and run are all stemmed into run).
3. Remove the stop words(a, to, the, and, etc)[may not be a necessary step], punctuations, lower case, lemmatising, etc.

After parsing the text data, we need to calculate the text similarity to remove the duplications: cosine similarity, Jaccard similarity coefficient, etc.

To categorize the news into different groups, we need to adopt suitable text classification algorithms, e.g., Naive Bayes , Tf-idf, etc. Some recommended python libraries includes: nltk, scikit-learn, etc.

To transfer the data into the client app, we also created a data storage mechanism which saves the crawled data into database and then gets retrieved according to category&page number.

The final app is like the following, with a news list tableview page and and a news detail page. The category tabs are displayed at the top scrollview, which makes the experience more user friendly.

There are a lot of challenges. One of the challenge occurs when we use our news duplication deletion algorithm, the complexity is high when we need to compare one article with every others(n^2). If the similarity comparison itself is not efficient enough, the whole thing would be very inefficient. Imagine there are 1,000 articles, then we'll need 1,000,000 comparisons. If every second can process 1000 comparisons, then it can take 17 minutes to process the whole thing. When it's less efficient, that would be a disaster. Initially, we compare the whole text body. It took so long time. We switched to compare title which is much more efficient. The result turns out to be OK.  A better strategy would be to process the text body to abstract the top N keywords and then compare the keywords instead of the whole article.

The urgent task was not confirmed until 3 days before the deadline. It looked like an impossible task to finish in such a short time at this complicity level. Thankfully, three out of four developers in our team learned NLP/Machine learning and we confirmed it doable. Our team worked really hard staying overnight and together we made it happen. Thanks for the help from many open source communities for providing very good NLP/Machine learning libraries. Also a special thank to a friend from Tencent who gave us nice guidances in the very early stage.

This article is a personal sharing and there may be some parts which are not academically rigorous, please feel free to point out the parts which you find anything improper. Thanks!




Comments

Popular posts from this blog

InnovFest 2015

I attended the innovFest 2015 event. It was quite eye opening. Besides the booth, some topics in the forums also interested me. The first topic I joined was the Kopi Chat with Yossi Vardi, a famous Israeli entrepreneur and investor. He is straightforward and humorous. When talking about the most important reason why people wake up with a great idea but ended up sleeping without executing anything, he collected answers from the audiences. One answer pretty much fitted his appetite-- "People fear about losing faces". He shared his opinion with the quotes from Theodore Roosevelt, “It is not the critic who counts; not the man who points out how the strong man stumbles, or where the doer of deeds could have done them better. The credit belongs to the man who is actually in the arena, whose face is marred by dust and sweat and blood; who strives valiantly; who errs, who comes short again and again, because there is no effort without error and shortcoming; but who does actually st

Thoughts

I've been working as a project manager on an interesting project named " Dadafish " for quite some time. The intention is to create a sharing economy platform for lifestyle teaching/learning. Put in a more straightforward way, an Airbnb for classes. There is one assumption:  People are willing to pay to learn skills from amateur or semi-pros at a lower price. Quite some ongoing arts classes in meetups are taught in cafe shops at 15-50SGD/session. It's an indirect validation of our assumption. There are also many free language classes in meetups and facebook groups. We did an experiment to validate the assumption by introducing a Japanese friend who's organizing a free meetup to teach Japanese. She doesn't have a certificate and she's currently teaching Japanese in meetups for free. The experiment class was charged at 15 SGD/h per person for a 3-4 pax group class. We posted the class and teacher info in a NUS student/alumni SNS group and three peopl

Consistency Matters

I didn't post anything last week, which means consistency has been broken. There's a need to reflect the task management skills, otherwise I'm very likely to driving the wrong road Priority changes. A good exercise would be to list the priorities down everyday and assign reasonable time to the tasks. Then never second-doubt. 100% focusing on the task when doing it. Priority changes from time to time.  Human minds are single-threaded, thus, we'll need to keep focused when doing one particular task. Be Grateful. I received the confirmation from NOC Israel that I got admitted by the programme and I'm heading to Israel next Jan:-) Thanks a lot for Prof.Ben and Karl's help. Ultimately, it might be a plan from God. Thanks every one. The additional interview from Google went well, however, I didn't get the winter intern opportunity this time. Though it's a bit too greedy to ask for too much, I still feel a bit sad after informed that I didn't get th