Project information

• Link: Project description and code on Github.

In this project real Twitter messages are analysed. The messages stem from disasters. The goal is to allow disaster relief agencies to automatically scan and classify Twitter messages to act immediately.

Automating the collection of Twitter messages is done in another project of mine. Providing this data to a web application yields the classification using Natural Language Processing (NLP). The web application simplifies allowing to enter individual messages and seeing the classification result immediately.

The user can indepently retrain the model. A dataset with the correct classification has to be provided which means it's supervised learning. Then an ETL pipeline prepares the data and stores it in a database while a Machine Learning (ML) pipeline finds the new best model which is used by the web application.

Challenges: coping with imbalanced data, e.g. category size in classification; limitations to model optimisation can be overcome with distributed computing -- Big Data processing is part of another project of mine.
Technology: web app with Flask and Bootstrap, Python package nltk for NLP and sqlalchemy for the database used.