An Algorithm for Exploratory Analysis and Normalization of Big Data with Pandas

Abstract

The digitization of business processes and the extraction of answers to user requests for big data are modern problems that are of great interest to scientists and researchers. The data generated so far, located in various corpora, is much more than can be analyzed. Therefore, they are collected, identified, cleaned and normalized to be used most adequately. Segmentation, assumptions and hypotheses contribute to the degree of satisfaction with the returned result. The research proposed a general method for collecting, cleaning and normalizing data from various sources, structurally modelling it into appropriate models, then testing hypotheses and analyzing the obtained results to conclude large academic data that will benefit the business in making management decisions. This is possible with the means of computational linguistics and with the help of Python data manipulation libraries.