Machine learning algorithms work with data, ML algorithms without data are like a BMW car without gasoline . It is very important that you feed the algorithm the right data in a suitable format ,scale and features.
In this post I try to elaborate the data preprocessing steps for preparing raw data for machine learning algorithms in Python. This steps make data suitable for learning and predicting steps.
Before the First step, I review the packages that we need in our practice. A package is a collection of modules , and every module contains functions, classes and other staffs related to one subject. Here we need pandas, matplotlib, seaborn, sklearn packages. Before we dig into preprocessing steps, I will simply describe these packages here :
Pandas : Pandas is a library of functions to read, modify and rewrite the dataset and to get some useful information such as minimum , maximum, mean and median of dataset’s columns.
Matplotlib , Seaborn : matplotlib is a library that consists of classes and functions for graphical representation of data and of mathematical functions. Diagrams play a key role in data preprocessing, because we use diagrams to have better understanding of dataset, and we make decision (to add/remove a feature) based on them. Seaborn is a visualization python package based on matplotlib. It provides a easy-to-use functions for drawing informative and useful statistical diagrams.
Sklearn: Is a fullstack machine learning python package that not only includes classification, regression and clustering algorithms, but also it has very helpful modules for preprocessing and feature extraction. It’s based on and compatible with NumPy, SciPy and Matplotlib. After finishing dataset preprocessing, We will use one of ML algorithms from this package to see how much our preprocessing is effective on final result.
Step 1 : Reading the dataset and retrieve its information
Thanks to panadas library, it is very easy to read the whole dataset at a once with a single line of python code using
read_cvs() function by passing the CVS file local adress to it. However, There are many other parameters one can pass to function that changes the its behavior . For example, we can specify which columns we need to read so that we don’t have to edit the data frame later.
import os import matplotlib.pyplot as plt import pandas as pd import numpy as np import seaborn as sns import sklearn as sk df_train = pd.read_csv("task1/data.csv") // read the given dataset df_train.head() // shows the top 5 records of dataset
describe() is useful method that shows statistical information of a data frame such as minimum, maximum, percentile, mean, std. So we call it on our data frame to have some overview of it.
print(df_train.describe()) //show statistical details of dataset
The post will be updated soon…